On Wednesday, the US Copyright Office began seeking public comment on issues surrounding generative AI systems and copyright. The public comment period, which starts on August 30, aims to explore the complex intersection of AI technology with copyright laws, and it closes on November 15. The comments could inform how the agency decides to grant copyrights in the future.

Outlined in a 24-page document published as a PDF file by the Federal Register, the “Notice of inquiry and request for comments” asks questions with far-reaching consequences for intellectual property in America.

Specifically, the Office is interested in four main areas: the use of copyrighted materials to train AI models and whether this constitutes infringement; the extent to which AI-generated content should or could be copyrighted, particularly when a human has exercised some degree of control over the AI model’s operations; how liability should be applied if AI-generated content infringes on existing copyrights; and the impact of AI mimicking voices or styles of human artists, which, while not strictly copyright issues, may engage state laws related to publicity rights and unfair competition.

Comments are sought to address these concerns because, in a relatively short period, we’ve seen the arrival of generative AI tools with surprising capabilities that can render images and video, imitate voices, answer questions, and compose text on demand. AI models have spawned hype and backlash, heavy corporate investment, and lawsuits. Those instances have put the Copyright Office on high alert as the frequency of AI-related copyright incidents increases:

Over the past several years, the Office has begun to receive applications to register works containing AI-generated material, some of which name AI systems as an author or co-author. At the same time, copyright owners have brought infringement claims against AI companies based on the training process for, and outputs derived from, generative AI systems.

As concerns and uncertainties mount, Congress and the Copyright Office have been contacted by many stakeholders with diverse views. The Office has publicly announced a broad initiative earlier this year to explore these issues. This Notice is part of that initiative and builds on the Office’s research, expertise, and prior work, as well as information that stakeholders have provided to the Office.

Listed among the incidents noted in the Copyright Office document are stories we’ve reported on in the past, including the first registration of latent diffusion artwork to Kris Kashtanova, which was granted in September and conditionally revoked in February; an attempt to register an AI-created artwork solely to a machine by Stephen Thaler (who has attempted similar things with patents), which a judge declined this month; and a copyright lawsuit against OpenAI on behalf of Sarah Silverman regarding the company’s use of her copyrighted works in ChatGPT training data. Additionally, artists sued Stability AI, Midjourney, and DeviantArt as a class action in January for alleged copyright violations.

In response to similar copyright concerns, several news organizations (including Ars Technica’s parent company, Conde Nast) recently took steps to block OpenAI’s web crawler, hoping it will prevent OpenAI from scraping data used to train future AI models. Most large language models, such as the ones that power ChatGPT, learned about the world by absorbing billions of documents found openly online—typically without consulting copyright holders.

The flexible arm of the law

While US copyright law is fixed, its interpretation of emerging technology is nuanced and can change depending on how regulators (and judges) feel about the topic, which depends partially on public sentiment.

As the interpretation stands, AI artwork may be copyrighted in the US as part of a larger, human-authored work, but not individually. And we’ve seen that copyright does not extend to creative works without any human authorship. However, there are still unresolved questions about what degree of human involvement would trigger the threshold for copyright registration, similar to questions that arose upon the invention of the camera in the 19th century.

In 1884, the defendant in Burrow-Giles Lithographic Co. v. Sarony claimed that photographs could not be subject to copyright because a photo is “a reproduction on paper of the exact features of some natural object or of some person.” They argued that a photo is the work of a machine and not a creative expression. But the court ruled that photos can be copyrighted because they are “representatives of original intellectual conceptions of [an] author.”

The question today is: What action in using a machine learning tool counts as the “original intellectual conception of an author?” Is it the selection of the particular AI tool? The input of a prompt? The curation or editing of the resulting machine-generated output? While current US copyright guidelines seem to sidestep these questions, the Copyright Office is open to hearing the public’s opinions.

Also, the Copyright Office would like to hear opinions about the unsettled issue over the inclusion of copyrighted material in training data sets, notably writing, “the Office is aware that there is disagreement about whether or when the use of copyrighted works to develop datasets for training AI models (in both generative and non-generative systems) is infringing.”

But that’s not all. The original 24-page request document is a fascinating look into the minds of current US copyright regulators, and it includes 34 multi-part questions of interest for the Copyright Office that would take another 24-page document to straighten out in detail.

The deadline for written comments is October 18, and they can be submitted online through a Regulations.gov form. Reply comments must be submitted to the Copyright Office by November 15.

https://arstechnica.com/?p=1964248

This week, OpenAI finally responded to a pair of nearly identical class-action lawsuits from book authors—including Sarah Silverman, Paul Tremblay, Mona Awad, Chris Golden, and Richard Kadrey—who earlier this summer alleged that ChatGPT was illegally trained on pirated copies of their books.

In OpenAI’s motion to dismiss (filed in both lawsuits), the company asked a US district court in California to toss all but one claim alleging direct copyright infringement, which OpenAI hopes to defeat at “a later stage of the case.”

The authors’ other claims—alleging vicarious copyright infringement, violation of the Digital Millennium Copyright Act (DMCA), unfair competition, negligence, and unjust enrichment—need to be “trimmed” from the lawsuits “so that these cases do not proceed to discovery and beyond with legally infirm theories of liability,” OpenAI argued.

OpenAI claimed that the authors “misconceive the scope of copyright, failing to take into account the limitations and exceptions (including fair use) that properly leave room for innovations like the large language models now at the forefront of artificial intelligence.”

According to OpenAI, even if the authors’ books were a “tiny part” of ChatGPT’s massive data set, “the use of copyrighted materials by innovators in transformative ways does not violate copyright.” Unlike plagiarists who seek to directly profit off distributing copyrighted materials, OpenAI argued that its goal was “to teach its models to derive the rules underlying human language” to do things like help people “save time at work,” “make daily life easier,” or simply entertain themselves by typing prompts into ChatGPT.

The purpose of copyright law, OpenAI argued, is “to promote the Progress of Science and useful Arts” by protecting the way authors express ideas, but “not the underlying idea itself, facts embodied within the author’s articulated message, or other building blocks of creative,” which are arguably the elements of authors’ works that would be useful to ChatGPT’s training model. Citing a notable copyright case involving Google Books, OpenAI reminded the court that “while an author may register a copyright in her book, the ‘statistical information’ pertaining to ‘word frequencies, syntactic patterns, and thematic markers’ in that book are beyond the scope of copyright protection.”

“Under the resulting judicial precedent, it is not an infringement to create ‘wholesale cop[ies] of [a work] as a preliminary step’ to develop a new, non-infringing product, even if the new product competes with the original,” OpenAI wrote.

In particular, OpenAI hopes to convince the court that the authors’ vicarious copyright infringement claim—which alleges that every ChatGPT output represents a derivative work, “regardless of whether there are any similarities between the output and the training works”— is an “erroneous legal conclusion.”

The company’s motion to dismiss cited “a simple response to a question (e.g., ‘Yes’),” or responding with “the name of the President of the United States” or with “a paragraph describing the plot, themes, and significance of Homer’s The Iliad” as examples of why every single ChatGPT output cannot seriously be considered a derivative work under authors’ “legally infirm” theory.

“That is not how copyright law works,” OpenAI argued, while claiming that any ChatGPT outputs that do connect to authors’ works are similar to “book reports or reviews.”

Further, OpenAI argued that the authors have failed to show that the company has a “direct financial interest” in allegedly infringing the copyrights of their works.

“It is not enough that the challenged activity is carried out by users of tools offered for profit by a technology company: rather, to satisfy the ‘direct financial interest’ prong” of copyright infringement, the material that infringes the plaintiff’s works must ‘act as a draw for [defendant’s] customers’ such that there is a direct ‘causal link between the infringement of the plaintiff’s own copyrighted works and any profit to the [defendant],’” OpenAI wrote.

Neither OpenAI nor lawyers representing the authors suing from the Joseph Saveri Law Firm immediately responded to Ars’ request to comment.

https://arstechnica.com/?p=1964348

On Tuesday, Google announced the launch of its Duet AI assistant across its Workspace apps, including Docs, Gmail, Drive, Slides, and more. First announced in May at Google I/O, Duet has been in testing for some time, but it is now available to paid Google Workspace business users (what Google calls its suite of cloud productivity apps) for $30 a month in addition to regular Workspace fees.

Duet is not just one thing—instead, it’s a blanket brand name for a multitude of different AI capabilities and probably should have been called “Google Kitchen Sink.” It likely represents several distinct AI systems behind the scenes. For example, in Gmail, Duet can summarize a conversation thread across emails, use the content of an email to write a brief or draft an email based on a topic. In Docs, it can write content such as a customer proposal or a story. In Slides, it can generate custom visuals using an image synthesis model. In Sheets, it can help format existing spreadsheets or create a custom spreadsheet structure suited to a particular task, such as a project tracker.

Some of Duet’s applications feel like confusion in branding. In Google Meet, Google says that Duet AI can “ensure you look and sound your best with studio look, studio lighting, and studio sound,” including “dynamic tiles” and “face detection”—functions that feel far removed from typical generative AI capabilities—as well as automatically translated captions. It can also reportedly capture notes and video, sending a summary to attendees in the meeting. In fact, using Duet’s “attend for me” feature, Google says that “Duet AI will be able to join the meeting on your behalf” and send you a recap later.

In Google Chat, Duet reads everything that’s going on in your conversations so that you can “ask questions about your content, get a summary of documents shared in a space, and catch up on missed conversations.”

Those are the marketing promises. So far, as spotted on social media, Duet in practice seems fairly mundane, like a mix of what we’ve seen with Google Bard and more complex versions of Google’s existing autocomplete features. An author named Charlie Guo ran through Duet features in a helpful X thread, noting the AI model’s awkward email compositions. “The writing is almost painfully formal,” he says.

In Slides, a Google-sponsored teacher named Alice Keeler asked Google Duet to make a robot teacher in front of a chalkboard and posted it on X. The results are awkward and arguably unusable, full of telltale glitches found in image synthesis artwork from 2022. Sure, it’s neat as a tech demo, but this is what a trillion-dollar company says is a production-ready tool today.

Of course, these capabilities can (and will) change over time as Google refines its offerings. Eventually, Duet may be absorbed into daily usage as if it weren’t even there, much like Google’s myriad other machine-learning features in its products.

https://arstechnica.com/?p=1964020

Meta is adding another Llama to its herd—and this one knows how to code. On Thursday, Meta unveiled “Code Llama,” a new large language model (LLM) based on Llama 2 that is designed to assist programmers by generating and debugging code. It aims to make software development more efficient and accessible, and it’s free for commercial and research use.

Much like ChatGPT and GitHub Copilot Chat, you can ask Code Llama to write code using high-level instructions, such as “Write me a function that outputs the Fibonacci sequence.” Or it can assist with debugging if you provide a sample of problematic code and ask for corrections.

As an extension of Llama 2 (released in July), Code Llama builds off of weights-available LLMs Meta has been developing since February. Code Llama has been specifically trained on source code data sets and can operate on various programming languages, including Python, Java, C++,  PHP, TypeScript, C#, Bash scripting, and more.

Notably, Code Llama can handle up to 100,000 tokens (word fragments) of context, which means it can evaluate long programs. To compare, ChatGPT typically only works with around 4,000-8,000 tokens, though longer context models are available through OpenAI’s API. As Meta explains in its more technical write-up:

Aside from being a prerequisite for generating longer programs, having longer input sequences unlocks exciting new use cases for a code LLM. For example, users can provide the model with more context from their codebase to make the generations more relevant. It also helps in debugging scenarios in larger codebases, where staying on top of all code related to a concrete issue can be challenging for developers. When developers are faced with debugging a large chunk of code they can pass the entire length of the code into the model.

Meta’s Code Llama comes in three sizes: 7, 13, and 34 billion parameter versions. Parameters are numerical elements of the neural network that get adjusted during the training process (before release). More parameters generally mean greater complexity and higher capability for nuanced tasks, but they also require more computational power to operate.

The different parameter sizes offer trade-offs between speed and performance. While the 34B model is expected to provide more accurate coding assistance, it is slower and requires more memory and GPU power to run. In contrast, the 7B and 13B models are faster and more suitable for tasks requiring low latency, like real-time code completion, and can run on a single consumer-level GPU.

Meta has also released two specialized variations: Code Llama – Python and Code Llama – Instruct. The Python variant is optimized specifically for Python programming (“fine-tuned on 100B tokens of Python code”), which is an important language in the AI community. Code Llama – Instruct, on the other hand, is tailored to better interpret user intent when provided with natural language prompts.

Additionally, Meta says the 7B and 13B base and instruct models have also been trained with “fill-in-the-middle” (FIM) capability, which allows them to insert code into existing code, which helps with code completion.

License and data set

Code Llama is available with the same license as Llama 2, which provides weights (the trained neural network files required to run the model on your machine) and allows research and commercial use, but with some restrictions laid out in an acceptable use policy.

Meta has repeatedly stated its preference for an open approach to AI, although its approach has received criticism for not being fully “open source” in compliance with the Open Source Initiative. Still, what Meta provides and allows with its license is far more open than OpenAI, which does not make the weights or code for its state-of-the-art language models available.

Meta has not revealed the exact source of its training data for Code Llama (saying it’s based largely on a “near-deduplicated dataset of publicly available code”), but some suspect that content scraped from the StackOverflow website may be one source. On X, Hugging Face data scientist Leandro von Werra shared a potentially hallucinated discussion about a programming function that included two real StackOverflow user names.

In the Code Llama research paper, Meta says, “We also source 8% of our samples data from natural language datasets related to code. This dataset contains many discussions about code and code snippets included in natural language questions or answers.”

Still, von Werra would like to see specifics cited in the future. “It would be great for reproducibility and sharing knowledge with the research community to disclose what data sources were used during training,” von Werra wrote. “Even more importantly it would be great to acknowledge that these communities contributed to the success of the resulting models.”

https://arstechnica.com/?p=1963185

Paralysis had robbed the two women of their ability to speak. For one, the cause was amyotrophic lateral sclerosis, or ALS, a disease that affects the motor neurons. The other had suffered a stroke in her brain stem. Though they can’t enunciate clearly, they remember how to formulate words.

Now, after volunteering to receive brain implants, both are able to communicate through a computer at a speed approaching the tempo of normal conversation. By parsing the neural activity associated with the facial movements involved in talking, the devices decode their intended speech at a rate of 62 and 78 words per minute, respectively—several times faster than the previous record. Their cases are detailed in two papers published Wednesday by separate teams in the journal Nature.

“It is now possible to imagine a future where we can restore fluid conversation to someone with paralysis, enabling them to freely say whatever they want to say with an accuracy high enough to be understood reliably,” said Frank Willett, a research scientist at Stanford University’s Neural Prosthetics Translational Laboratory, during a media briefing on Tuesday. Willett is an author on a paper produced by Stanford researchers; the other was published by a team at UC San Francisco.

While slower than the roughly 160-word-per-minute rate of natural conversation among English speakers, scientists say it’s an exciting step toward restoring real-time speech using a brain-computer interface, or BCI. “It is getting close to being used in everyday life,” says Marc Slutzky, a neurologist at Northwestern University who wasn’t involved in the new studies.

A BCI collects and analyzes brain signals, then translates them into commands to be carried out by an external device. Such systems have allowed paralyzed people to control robotic arms, play video games, and send emails with their minds. Previous research by the two groups showed it was possible to translate a paralyzed person’s intended speech into text on a screen, but with limited speed, accuracy, and vocabulary.

In the Stanford study, researchers developed a BCI that uses the Utah array, a tiny square sensor that looks like a hairbrush with 64 needle-like bristles. Each is tipped with an electrode, and together they collect the activity of individual neurons. Researchers then trained an artificial neural network to decode brain activity and translate it into words displayed on a screen.

They tested the system on volunteer Pat Bennett, the ALS patient, who is now 68 years old. In March 2022, a surgeon inserted four of these tiny sensors into Bennett’s cerebral cortex—the outermost layer of the brain. Thin wires connect the arrays to pedestals atop her head, which can be hooked up to a computer via cables.

Over the course of four months, scientists trained the software by asking Bennett to try to say sentences out loud. (Bennett can still produce sounds, but her speech is unintelligible.) Eventually, the software taught itself to recognize the distinct neural signals associated with the movements of the lips, jaw, and tongue that she was making to produce different sounds. From there, it learned the neural activity that corresponds to the motions used to create the sounds that make up words. It was then able to predict sequences of those words and string together sentences on a computer screen.

With the help of the device, Bennett was able to communicate at an average rate of 62 words per minute. The BCI made mistakes 23.8 percent of the time on a 125,000-word vocabulary. The previous record was only 18 words per minute—a record established in 2021, when members of the Stanford team published a paper describing a BCI that converted a paralyzed person’s imagined handwriting into text on a screen.

https://arstechnica.com/?p=1963217

California’s main firefighting agency, Cal Fire, is training AI models to detect visual signs of wildfires using a network of 1,039 high-definition cameras, reports The New York Times and the Los Angeles Times. When it sees signs of smoke, it quickly warns firefighters of emerging threats. During the pilot program, the system has already detected 77 wildfires before dispatch centers received 911 calls—about a 40 percent success rate, according to the NYT.

Traditionally, Cal Fire detects emerging wildfires by relying on the same network of over 1,000 mountaintop cameras monitored by humans, who look out for signs of smoke. But it’s tedious and tiring work. Phillip SeLegue, the staff chief of intelligence for Cal Fire, told the NYT that the new AI system has not only improved response times but also identified some fires early, allowing those blazes to be tackled while still being a manageable size.

However, the technology is not without limitations. It can only detect fires that are visible to its network of cameras, and human intervention is still required to confirm the AI model’s alerts. Engineers from DigitalPath, the California-based company responsible for creating the software, have been manually vetting each fire the AI identifies. The process has been challenging, with many false positives from fog, haze, dust kicked up from tractors, and steam from geothermal plants, according to Ethan Higgins, a chief architect of the software. “You wouldn’t believe how many things look like smoke,” Higgins told the NYT.

Some experienced fire operators like Andrew Emerick, the duty chief for Cal Fire’s northern region, remain skeptical about the AI’s ability to understand the nuanced context in which certain fires occur, such as those deliberately set for agricultural purposes. “I don’t think this robot is ever going to take my job,” he said.

The AI system processes “billions of megapixels” of images every minute from cameras that cover approximately 90 percent of California’s fire-prone areas. The AI program started in June and was initially deployed in six of Cal Fire’s command centers but will expand to all of Cal Fire’s 21 command centers in September.

Cal Fire also utilizes other fire detection methods, including 911 calls from residents and a partnership with the United States military, Fireguard, which uses classified spy satellites, drones, and other aircraft to detect fires. Even with early drawbacks like false positives, Neal Driscoll, a leader of the Cal Fire AI project, told the NYT that the AI program’s ultimate success will be measured by the fires that the public will never hear about, as they will be quickly extinguished while still small.

This year has been relatively less destructive in terms of wildfires, compared to previous years: Cal Fire reported 4,792 wildfires so far, which is lower than the five-year average of 5,422 for the summer months. Even so, that doesn’t make any wildfire less devastating when it spirals out of control.

https://arstechnica.com/?p=1962998

On Tuesday, OpenAI announced fine-tuning for GPT-3.5 Turbo—the AI model that powers the free version of ChatGPT—through its API. It allows training the model with custom data, such as company documents or project documentation. OpenAI claims that a fine-tuned model can perform as well as GPT-4 with lower cost in certain scenarios.

In AI, fine-tuning refers to the process of taking a pretrained neural network (like GPT-3.5 Turbo) and further training it on a different dataset (like your custom data), which is typically smaller and possibly related to a specific task. This process builds off of knowledge the model gained during its initial training phase and refines it for a specific application.

So basically, fine-tuning teaches GPT-3.5 Turbo about custom content, such as project documentation or any other written reference. That can come in handy if you want to build an AI assistant based on GPT-3.5 that is intimately familiar with your product or service but lacks knowledge of it in its training data (which, as a reminder, was scraped off the web before September 2021).

“Since the release of GPT-3.5 Turbo, developers and businesses have asked for the ability to customize the model to create unique and differentiated experiences for their users,” writes OpenAI on its promotional blog. “With this launch, developers can now run supervised fine-tuning to make this model perform better for their use cases.”

While GPT-4, the more powerful cousin of GPT-3.5, is well-known as a generalist that is adaptable to many subjects, it is slower and more expensive to run. OpenAI is pitching 3.5 fine-tuning as a way to get GPT-4-like performance in a specific knowledge domain at a lower cost and faster execution time. “Early tests have shown a fine-tuned version of GPT-3.5 Turbo can match, or even outperform, base GPT-4-level capabilities on certain narrow tasks,” they write.

Also, OpenAI says that fine-tuned models provide “improved steerability,” which means following instructions better; “reliable output formatting,” which improves the model’s ability to consistently output text in a format such as API calls or JSON; and “custom tone,” which can bake-in a custom flavor or personality to a chatbot.

OpenAI says that fine-tuning allows users to shorten their prompts and can save money in OpenAI API calls, which are billed per token. “Early testers have reduced prompt size by up to 90% by fine-tuning instructions into the model itself,” says OpenAI. Right now, the context length for fine-tuning is set at 4,000 tokens, but OpenAI says that fine-tuning will extend to the 16,000-token model “later this fall.”

Using your own data comes at a cost

By now, you might be wondering how using your own data to train GPT-3.5 works—and what it costs. OpenAI lays out a simplified process on its blog that shows setting up a system prompt with the API, uploading files to OpenAI for training, and creating a fine-tuning job using the command-line tool curl to query an API web address. Once the fine-tuning process is complete, OpenAI says the customized model is available for use immediately with the same rate limits as the base model. More details can be found in OpenAI’s official documentation.

All of this comes at a price, of course, and it’s split into training costs and usage costs. To train GPT-3.5 costs $0.008 per 1,000 tokens. During the usage phase, API access costs $0.012 per 1,000 tokens for text input and $0.016 per 1,000 tokens for text output.

By comparison, the base 4k GPT-3.5 Turbo model costs $0.0015 per 1,000 tokens input and $0.002 per 1,000 tokens output, so the fine-tuned model is about eight times more expensive to run. And while GPT-4’s 8K context model is also cheaper at $0.03 per 1,000 tokens input and $0.06 per 1,000-token output, OpenAI still claims that money can be saved due to the reduced need for prompting in the fine-tuned model. It’s a stretch, but in narrow cases, it may apply.

Even at a higher cost, teaching GPT-3.5 about custom documents may be well worth the price for some folks—if you can keep the model from making stuff up about it. Customizing is one thing, but trusting the accuracy and reliability of GPT-3.5 Turbo outputs in a production environment is another matter entirely. GPT-3.5 is well-known for its tendency to confabulate information.

Regarding data privacy, OpenAI notes that, as with all of its APIs, data sent in and out of the fine-tuning API is not used by OpenAI (or anyone else) to train AI models. Interestingly, OpenAI will send all customer fine-tuning training data through GPT-4 for moderation purposes using its recently announced moderation API. That may account for some of the cost of using the fine-tuning service.

And if 3.5 isn’t good enough for you, OpenAI says that fine-tuning for GPT-4 is coming this fall. From our experience, that GPT-4 doesn’t make things up as much, but fine-tuning that model (or the rumored 8 models working together under the hood) will likely be far more expensive.

https://arstechnica.com/?p=1962565

Large language models, the AI tech behind things like Chat GPT, are just what their name implies: big. They often have billions of individual computational nodes and huge numbers of connections among them. All of that means lots of trips back and forth to memory and a whole lot of power use to make that happen. And the problem is likely to get worse.

One way to potentially avoid this is to mix memory and processing. Both IBM and Intel have made chips that equip individual neurons with all the memory they need to perform their functions. An alternative is to perform operations in memory, an approach that has been demonstrated with phase-change memory.

Now, IBM has followed up on its earlier demonstration by building a phase-change chip that’s much closer to a functional AI processor. In a paper released on Wednesday by Nature, the company shows that its hardware can perform speech recognition with reasonable accuracy and a much lower energy footprint.

In phase

Phase-change memory has been under development for a while. It offers the persistence of flash memory but with performance that’s much closer to existing volatile RAM. It operates by heating a small patch of material and then controlling how quickly it cools. Cool it slowly, and the material forms an orderly crystal that conducts electricity reasonably well. Cool it quickly, and it forms a disordered mess that has much higher resistance. The difference between these two states can store a bit that will remain stored until enough voltage is applied to melt the material again.

This behavior also turns out to be a great match for neural networks. In neural networks, each node receives an input and, based on its state, determines how much of that signal to forward to further nodes. Typically, this is viewed as representing the strength of the connections between individual neurons in the network. Thanks to the behavior of phase-change memory, that strength can also be represented by an individual bit of memory operating in an analog mode.

When storing digital bits, the difference between the on and off states of phase-change memory is maximized to limit errors. But it’s entirely possible to set the resistance of a bit to values anywhere in between its on and off states, allowing analog behavior. This smooth gradient of potential values can be used to represent the strength of connections between nodes—you can get the equivalent of a neural network node’s behavior simply by passing current through a bit of phase-change memory.

As mentioned above, IBM has already shown this can work. The chip described today, however, is much closer to a functional processor, containing all the hardware needed to connect individual nodes. And it has done so at a scale much closer to that needed to handle large language models.

https://arstechnica.com/?p=1962669

Microsoft’s efforts to put new AI-powered features into Windows 11 will pick up steam this fall when Windows Copilot is officially released, but the company isn’t stopping there. According to a report from Windows Central, Microsoft is in the early stages of experimenting with new features for built-in Windows apps like Photos, Snipping Tool, and even Paint, which all fall under the broad heading of “AI.”

The report claims that Photos, Camera, and Snipping Tool—all apps that work with either photos or screenshots—could soon include optical character recognition (OCR) features that would allow users to copy and paste text from images into word processors and text editors. The Photos app could also gain the ability to recognize people and objects in photos and make it easier to separate them from their backgrounds.

The venerable MS Paint app, on the other hand, could gain some generative AI functions that would allow it to create images based on text prompts, similar to features currently supported by more high-end image editors like Adobe Photoshop. Microsoft’s Bing Image Creator already uses a DALL-E-based model to create AI-generated images.

The report says that some of these features may rely on a neural processing unit (NPU) built into your PC’s processor. These have been included in some of Qualcomm’s Arm processors for a while now, but more traditional x86 PCs with AMD and Intel processors don’t include them. AMD’s recent 7040-series chips and Intel’s upcoming “Meteor Lake” refresh both add NPUs.

Currently, only a small handful of NPU-exclusive features are included in Windows 11, mostly related to image enhancement and background replacement for video calls. A local NPU makes it possible to do more AI-accelerated stuff locally on your computer rather than relying on the cloud, which can address some of the privacy and model-training concerns that arise when using AI-powered products.

Many of these features sound like the kinds of inoffensive, broadly useful features that we might have put under the umbrella of “machine learning” in a different era, and not merely another app with ChatGPT bolted onto it. Apple supports similar character-recognition features in the Photos apps for macOS and iOS, powered by the “neural engine” that the company has been building into its A- and M-series processors for years now.

https://arstechnica.com/?p=1962570

On Tuesday, Meta announced SeamlessM4T, a multimodal AI model for speech and text translations. As a neural network that can process both text and audio, it can perform text-to-speech, speech-to-text, speech-to-speech, and text-to-text translations for “up to 100 languages,” according to Meta. Its goal is to help people who speak different languages communicate with each other more effectively.

Continuing Meta’s relatively open approach to AI, Meta is releasing SeamlessM4T under a research license (CC BY-NC 4.0) that allows developers to build on the work. They’re also releasing SeamlessAlign, which Meta calls “the biggest open multimodal translation dataset to date, totaling 270,000 hours of mined speech and text alignments.” That will likely kick-start the training of future translation AI models from other researchers.

Among the features of SeamlessM4T touted on Meta’s promotional blog, the company says that the model can perform speech recognition (you give it audio of speech, and it converts it to text), speech-to-text translation (it translates spoken audio to a different language in text), speech-to-speech translation (you feed it speech audio, and it outputs translated speech audio), text-to-text translation (similar to how Google Translate functions), and text-to-speech translation (feed it text and it will translate and speak it out in another language). Each of the text translation functions supports nearly 100 languages, and the speech output functions support about 36 output languages.

In the SeamlessM4T announcement, Meta references the Babel Fish, a fictional fish from Douglas Adams’ classic sci-fi series that, when placed in one’s ear, can instantly translate any spoken language:

Building a universal language translator, like the fictional Babel Fish in The Hitchhiker’s Guide to the Galaxy, is challenging because existing speech-to-speech and speech-to-text systems only cover a small fraction of the world’s languages. But we believe the work we’re announcing today is a significant step forward in this journey.

How did they train it? According to the Seamless4MT research paper, Meta’s researchers “created a multimodal corpus of automatically aligned speech translations of more than 470,000 hours, dubbed SeamlessAlign” (previously mentioned above). They then “filtered a subset of this corpus with human-labeled and pseudo-labeled data, totaling 406,000 hours.”

As usual, Meta is being a little vague about where it got its training data. The text data came from “the same dataset deployed in NLLB,” (sets of sentences pulled from Wikipedia, news sources, scripted speeches, and other sources and translated by professional human translators). And SeamlessM4T’s speech data came from “4 million hours of raw audio originating from a publicly available repository of crawled web data,” of which 1 million hours were in English, according to the research paper. Meta did not specify which repository or the provenance of the audio clips used.

Meta is far from the first AI company to offer machine-learning translation tools. Google Translate has used machine-learning techniques since 2006, and large language models (such as GPT-4) are well known for their ability to translate between languages. But more recently, the tech has heated up on the audio processing front. In September, OpenAI released its own open source speech-to-text translation model, called Whisper, that can recognize speech in audio and translate it to text with a high level of accuracy.

SeamlessM4T builds from that trend by expanding multimodal translation to many more languages. In addition, Meta says that SeamlessM4T’s “single system approach”—a monolithic AI model instead of multiple models combined in a chain (like some of Meta’s previous audio-processing techniques)—reduces errors and increases the efficiency of the translation process.

More technical details on how SeamlessM4T works are available on Meta’s website, and its code and weights (the actual trained neural network files) can be found on Hugging Face.

https://arstechnica.com/?p=1962192