Huawei in difficoltà con i chip AI, le sanzioni USA funzionano
Huawei a corto di chip AI Ascend
La migliore alternativa ‘made in China’ ai chip AI di Nvidia sono gli Ascend 910B di Huawei, ma secondo quanto riportato dalla testata The Information, in un articolo firmato da Qianer Liu e Amir Efrati, le sanzioni imposte dall’amministrazione Biden al gigante tecnologico cinese sembrano funzionare.
D’altronde, come spiegato dal portavoce del Dipartimento del Commercio degli Stati Uniti agli autori dell’articolo, “Huawei rappresenta una minaccia continua alla nostra sicurezza nazionale, dobbiamo rimanere concentrati nel rintracciare qualsiasi tentativo di evasione o elusione dei nostri controlli”.
I funzionari americani, giustamente, non si fidano dei grandi giri che le merci possono compiere nei transiti globali (il caso del gas e del petrolio russi sotto sanzioni che ancora arrivano in Europa è un esempio eclatante).
Necessario l’aiuto dei produttori di chip AI americani
La produzione di chip a cui si appoggia il gigante di Shenzhen è di generazione passata, le fabbriche asiatiche su cui può fare affidamento non hanno raggiunto i livelli di qualità e innovazione delle taiwanesi TSMC e UMC o della coreana Samsung.
Questo stato di cose potrebbe spingere Huawei ad utilizzare tecnologia americana per non mettere a repentaglio gli ordini.
L’obiettivo di Washington era proprio questo: rendere la vita difficile alle aziende cinesi e favorire quelle americane, in ogni comparto produttivo, soprattutto quelli più strategici, tra cui i semiconduttori.
Questo accadeva ad agosto 2023. Huawei è sulla lista nera commerciale stilata dal DoC degli Stati Uniti dal 2019, ma ha potuto ricevere strumenti avanzati per la produzione di chip da aziende statunitensi, grazie ad una licenza speciale per questo tipo di transazioni.
Researchers upend AI status quo by eliminating matrix multiplication in LLMs
Enlarge/ Illustration of a brain inside of a light bulb.
Researchers claim to have developed a new way to run AI language models more efficiently by eliminating matrix multiplication from the process. This fundamentally redesigns neural network operations that are currently accelerated by GPU chips. The findings, detailed in a recent preprint paper from researchers at the University of California Santa Cruz, UC Davis, LuxiTech, and Soochow University, could have deep implications for the environmental impact and operational costs of AI systems.
Matrix multiplication (often abbreviated to “MatMul”) is at the center of most neural network computational tasks today, and GPUs are particularly good at executing the math quickly because they can perform large numbers of multiplication operations in parallel. That ability momentarily made Nvidia the most valuable company in the world last week; the company currently holds an estimated 98 percent market share for data center GPUs, which are commonly used to power AI systems like ChatGPT and Google Gemini.
In the new paper, titled “Scalable MatMul-free Language Modeling,” the researchers describe creating a custom 2.7 billion parameter model without using MatMul that features similar performance to conventional large language models (LLMs). They also demonstrate running a 1.3 billion parameter model at 23.8 tokens per second on a GPU that was accelerated by a custom-programmed FPGA chip that uses about 13 watts of power (not counting the GPU’s power draw). The implication is that a more efficient FPGA “paves the way for the development of more efficient and hardware-friendly architectures,” they write.
The paper doesn’t provide power estimates for conventional LLMs, but this post from UC Santa Cruz estimates about 700 watts for a conventional model. However, in our experience, you can run a 2.7B parameter version of Llama 2 competently on a home PC with an RTX 3060 (that uses about 200 watts peak) powered by a 500-watt power supply. So, if you could theoretically completely run an LLM in only 13 watts on an FPGA (without a GPU), that would be a 38-fold decrease in power usage.
The technique has not yet been peer-reviewed, but the researchers—Rui-Jie Zhu, Yu Zhang, Ethan Sifferman, Tyler Sheaves, Yiqiao Wang, Dustin Richmond, Peng Zhou, and Jason Eshraghian—claim that their work challenges the prevailing paradigm that matrix multiplication operations are indispensable for building high-performing language models. They argue that their approach could make large language models more accessible, efficient, and sustainable, particularly for deployment on resource-constrained hardware like smartphones.
Doing away with matrix math
In the paper, the researchers mention BitNet (the so-called “1-bit” transformer technique that made the rounds as a preprint in October) as an important precursor to their work. According to the authors, BitNet demonstrated the viability of using binary and ternary weights in language models, successfully scaling up to 3 billion parameters while maintaining competitive performance.
However, they note that BitNet still relied on matrix multiplications in its self-attention mechanism. Limitations of BitNet served as a motivation for the current study, pushing them to develop a completely “MatMul-free” architecture that could maintain performance while eliminating matrix multiplications even in the attention mechanism.
https://arstechnica.com/?p=2033314
OpenAI’s ChatGPT for Mac is now available to all users
Enlarge/ The app lets you invoke ChatGPT from anywhere in the system with a keyboard shortcut, Spotlight-style.
Samuel Axon
OpenAI’s official ChatGPT app for macOS is now available to all users for the first time, provided they’re running macOS Sonoma or later.
It was previously being rolled out gradually to paid subscribers to ChatGPT’s Plus premium plan.
The ChatGPT Mac app mostly acts as a desktop window version of the web app, allowing you to carry on back-and-forth prompt-and-response conversations. You can select between the GPT-3.5, GPT-4, and GPT-4o models. It also supports the more specialized GPTs available in the web version, including the DALL-E image generator and custom GPTs.
There is one important omission in this desktop app: It doesn’t support using the API. For that, those wanting a desktop app will still need to use a third-party one like Jordi Bruin’s MacGPT.
OpenAI’s app lets you enable a system-wide keyboard shortcut (option + space by default) to type in a prompt any time; it works a bit like opening a Spotlight search in macOS.
It was announced alongside the GPT-4o model, which is faster and cheaper to use than GPT-4 with similar (albeit not exactly the same) accuracy and quality, and which expands the model’s ability to interact with images and videos.
At the same time it unveiled GPT-4o and the Mac app, OpenAI demonstrated a new, conversational approach to voice chatting with ChatGPT. That’s not yet widely available, but it’s said to be coming soon. For now, the Mac app supports the old style of back-and-forth voice chats with the chatbot.
The Mac app is unavailable in the Mac App Store, but you can download it directly from OpenAI’s website.
Note that, like the iPhone or Android ChatGPT app, this is distinct from the ChatGPT integration coming to Apple’s operating systems this fall. That integration will see Siri referring users to ChatGPT (and possibly alternative models in the future) to answer queries that are outside Siri’s usual scope, and it will be baked into the operating system.
There’s still no Windows app. Why not? At least one report claimed that OpenAI prioritized a Mac app over a Windows app “because that’s where most of its users are.” Of course, Windows users aren’t hurting for AI chatbot options, as Microsoft has been throwing ChatGPT-powered Copilot into everything it can of late.
https://arstechnica.com/?p=2033436
Taking a closer look at AI’s supposed energy apocalypse
Enlarge/ Someone just asked what it would look like if their girlfriend was a Smurf. Better add another rack of servers!
Getty Images
Late last week, both Bloomberg and The Washington Post published stories focused on the ostensibly disastrous impact artificial intelligence is having on the power grid and on efforts to collectively reduce our use of fossil fuels. The high-profile pieces lean heavily on recent projections from Goldman Sachs and the International Energy Agency (IEA) to cast AI’s “insatiable” demand for energy as an almost apocalyptic threat to our power infrastructure. The Post piece even cites anonymous “some [people]” in reporting that “some worry whether there will be enough electricity to meet [the power demands] from any source.”
Digging into the best available numbers and projections available, though, it’s hard to see AI’s current and near-future environmental impact in such a dire light. While generative AI models and tools can and will use a significant amount of energy, we shouldn’t conflate AI energy usage with the larger and largely pre-existing energy usage of “data centers” as a whole. And just like any technology, whether that AI energy use is worthwhile depends largely on your wider opinion of the value of generative AI in the first place.
Not all data centers
While the headline focus of both Bloomberg and The Washington Post’s recent pieces is on artificial intelligence, the actual numbers and projections cited in both pieces overwhelmingly focus on the energy used by Internet “data centers” as a whole. Long before generative AI became the current Silicon Valley buzzword, those data centers were already growing immensely in size and energy usage, powering everything from Amazon Web Services servers to online gaming services, Zoom video calls, and cloud storage and retrieval for billions of documents and photos, to name just a few of the more common uses.
The Post story acknowledges that these “nondescript warehouses packed with racks of servers that power the modern Internet have been around for decades.” But in the very next sentence, the Post asserts that, today, data center energy use “is soaring because of AI.” Bloomberg asks one source directly “why data centers were suddenly sucking up so much power” and gets back a blunt answer: “It’s AI… It’s 10 to 15 times the amount of electricity.”
Enlarge/ The massive growth in data center power usage mostly predates the current mania for generative AI (red 2022 line added by Ars).
Unfortunately for Bloomberg, that quote is followed almost immediately by a chart that heavily undercuts the AI alarmism. That chart shows worldwide data center energy usage growing at a remarkably steady pace from about 100 TWh in 2012 to around 350 TWh in 2024. The vast majority of that energy usage growth came before 2022, when the launch of tools like Dall-E and ChatGPT largely set off the industry’s current mania for generative AI. If you squint at Bloomberg’s graph, you can almost see the growth in energy usage slowing down a bit since that momentous year for generative AI.
Determining precisely how much of that data center energy use is taken up specifically by generative AI is a difficult task, but Dutch researcher Alex de Vries found a clever way to get an estimate. In his study “The growing energy footprint of artificial intelligence,” de Vries starts with estimates that Nvidia’s specialized chips are responsible for about 95 percent of the market for generative AI calculations. He then uses Nvidia’s projected production of 1.5 million AI servers in 2027—and the projected power usage for those servers—to estimate that the AI sector as a whole could use up anywhere from 85 to 134 TWh of power in just a few years.
To be sure, that is an immense amount of power, representing about 0.5 percent of projected electricity demand for the entire world (and an even greater ratio in the local energy mix for some common data center locations). But measured against other common worldwide uses of electricity, it’s not representative of a mind-boggling energy hog. A 2018 study estimated that PC gaming as a whole accounted for 75 TWh of electricity use per year, to pick just one common human activity that’s on the same general energy scale (and that’s without console or mobile gamers included).
Enlarge/ Worldwide projections for AI energy use in 2027 are on the same scale as the energy used by PC gamers.
More to the point, de Vries’ AI energy estimates are only a small fraction of the 620 to 1,050 TWh that data centers as a whole are projected to use by 2026, according to the IEA’s recent report. The vast majority of all that data center power will still be going to more mundane Internet infrastructure that we all take for granted (and which is not nearly as sexy of a headline bogeyman as “AI”).
https://arstechnica.com/?p=2033273
Music industry giants allege mass copyright violation by AI firms
Enlarge/ Michael Jackson in concert, 1986. Sony Music owns a large portion of publishing rights to Jackson’s music.
Universal Music Group, Sony Music, and Warner Records have sued AI music-synthesis companies Udio and Suno for allegedly committing mass copyright infringement by using recordings owned by the labels to train music-generating AI models, reports Reuters. Udio and Suno can generate novel song recordings based on text-based descriptions of music (i.e., “a dubstep song about Linus Torvalds”).
The lawsuits, filed in federal courts in New York and Massachusetts, claim that the AI companies’ use of copyrighted material to train their systems could lead to AI-generated music that directly competes with and potentially devalues the work of human artists.
Like other generative AI models, both Udio and Suno (which we coveredseparately in April) rely on a broad selection of existing human-created artworks that teach a neural network the relationship between words in a written prompt and styles of music. The record labels correctly note that these companies have been deliberately vague about the sources of their training data.
Until generative AI models hit the mainstream in 2022, it was common practice in machine learning to scrape and use copyrighted information without seeking permission to do so. But now that the applications of those technologies have become commercial products themselves, rightsholders have come knocking to collect. In the case of Udio and Suno, the record labels are seeking statutory damages of up to $150,000 per song used in training.
In the lawsuit, the record labels cite specific examples of AI-generated content that allegedly re-creates elements of well-known songs, including The Temptations’ “My Girl,” Mariah Carey’s “All I Want for Christmas Is You,” and James Brown’s “I Got You (I Feel Good).” It also claims the music-synthesis models can produce vocals resembling those of famous artists, such as Michael Jackson and Bruce Springsteen.
Reuters claims it’s the first instance of lawsuits specifically targeting music-generating AI, but music companies and artists alike have been gearing up to deal with challenges the technology may pose for some time.
In May, Sony Music sent warning letters to over 700 AI companies (including OpenAI, Microsoft, Google, Suno, and Udio) and music-streaming services that prohibited any AI researchers from using its music to train AI models. In April, over 200 musical artists signed an open letter that called on AI companies to stop using AI to “devalue the rights of human artists.” And last November, Universal Music filed a copyright infringement lawsuit against Anthropic for allegedly including artists’ lyrics in its Claude LLM training data.
Similar to The New York Times’ lawsuit against OpenAI over the use of training data, the outcome of the record labels’ new suit could have deep implications for the future development of generative AI in creative fields, including requiring companies to license all musical training data used in creating music-synthesis models.
Compulsory licenses for AI training data could make AI model development economically impractical for small startups like Udio and Suno—and judging by the aforementioned open letter, many musical artists may applaud that potential outcome. But such a development would not preclude major labels from eventually developing their own AI music generators themselves, allowing only large corporations with deep pockets to control generative music tools for the foreseeable future.
https://arstechnica.com/?p=2033128
AI responsabile, l’Italia è settima al mondo secondo il Global Index
“In assenza di misurazioni per monitorare impegni, pratiche e progressi nei paesi di tutto il mondo, il concetto di AI responsabile rimane astratto”. Questo è l’incipit del sito Global Index on Responsible AI, un’iniziativa che vuole passare dai principi alla pratica nell’ambito dell’AI responsabile, tracciando quello che i singoli paesi stanno facendo in materia, prima che molti scenari cadano sotto le conseguenze di un uso sfrenato dell’intelligenza artificiale, a partire dai diritti umani.
Con una serie di parametri di riferimento basati proprio sui diritti umani, l’Indice globale sull’intelligenza artificiale responsabile misura gli impegni dei governi sul tema dell’AI, attraverso una lente sociale, tecnica e politica. Il progetto fa parte del Global Center on AI Governance (GCG), un’organizzazione basata in dall’Africa, con collaboratori e partner da tutto il mondo, intenta a promuovere competenze locali sull’intelligenza artificiale.
Prima di entrare nel dettaglio dei risultati che riguardano l’Italia, vale la pena riportare le dieci evidenze che il Global Index ha dedotto dalla sua prima rilevazione globale.
10 evidenze sull’AI responsabile
La governance dell’AI non si traduce in un’IA responsabile
I meccanismi che garantiscono la tutela dei diritti umani nel contesto dell’AI sono limitati
La cooperazione internazionale è un importante pilastro delle attuali pratiche di AI responsabile
L’uguaglianza di genere rimane una lacuna critica negli sforzi per promuovere l’AI responsabile
Le questioni chiave dell’inclusione e dell’uguaglianza nell’intelligenza artificiale non vengono affrontate
I lavoratori non sono adeguatamente protetti dall’uso dell’intelligenza artificiale
L’AI responsabile deve incorporare la diversità culturale e linguistica
Esistono importanti lacune nel garantire la sicurezza, la protezione e l’affidabilità dei sistemi di AI
Le università e la società civile stanno svolgendo un ruolo cruciale nel promuovere l’AI responsabile
C’è ancora molta strada per raggiungere livelli adeguati di AI responsabile a livello mondiale
Lo scenario italiano
Su un punteggio da 0 a 100, l’Italia è al 61,8 nel merito dell’adozione di sistemi di AI responsabile. Un numero che è la media di sei indici differenti. In quanto a framework governativo, abbiamo 59,99; per le azioni di governo il 55,96; per gli attori non statali il 77,10; per i diritti umani in AI il 66,34; per le capacità nazionali di una AI responsabile il 38,21; per la governance sulla AI responsabile, il 66,13. In Europa, Olanda, Germania, Irlanda, Regno Unito ed Estonia hanno una media più elevata. Gli olandesi sono primi al mondo, noi settimi.
L’ultimo paese è il Sudan, alla posizione 138.
I punteggi a livello nazionale sono il risultato di una valutazione completa degli impegni nazionali per lo sviluppo responsabile dell’AI, con particolare attenzione ai diritti umani, in 19 aree tematiche. Le informazioni sono integrate da fonti di dati secondari.
Apple sta valutando un accordo di intelligenza artificiale con Meta
Sembra che OpenAI non sia l’unica azienda con cui Apple intende avviare una collaborazione nel campo dell’intelligenza artificiale.
La Mela starebbe infatti lavorando ad una partnership con Meta. Secondo il Wall Street Journal, in base all’accordo, il colosso social porterebbe le sue capacità di intelligenza artificiale in Apple Intelligence, il progetto annunciato da Apple alla sua Worldwide Developers Conference all’inizio di questo mese. In quell’occasione, la società di Cupertino aveva svelato l’accordo, peraltro molto vociferato in precedenza, con OpenAI per portare ChatGpt nei suoi sistemi operativi.
Due mondi contrari
Una partnership Apple-Meta AI non è però così ovvia. Molto spesso l’organizzazione guidata da Mark Zuckerberg si è detta contraria alle modifiche alla privacy che Apple ha introdotto 2021, e che avrebbero “spazzato via” 10 miliardi di dollari di entrate al re dei social, un anno dopo, nel 2022. Inoltre, pare che Meta abbia invitato i suoi inserzionisti ad usare una soluzione alternativa per evitare le commissioni che Apple addebita per i post sponsorizzati.
Ma anche se la potenziale partnership dovesse concretizzarsi, non è l’unica che Apple sta perseguendo con i rivali tecnologici. Oltre alla sua relazione AI con OpenAI e a quella in fase di sviluppo con Meta, l’azienda sta lavorando anche con Google sull’integrazione AI, così come con startup come Anthropic e Perplexity.
Alleati prima della battaglia
Il tutto, per perseguire la missione principale delineata insieme ad Apple Intelligence, ossia quella di un ecosistema aperto, non chiuso come è stato finora il mondo di iOS e soci. Il motivo è semplice: gli sforzi AI della Mela non sono al pari di quelli della concorrenza e, per adesso, l’unico modo per non restare indietro, soprattutto dal punto di vista dei consumatori, è appoggiarsi sulle compagnie che hanno puntato maggiormente sull’intelligenza artificiale, tenendosele per amiche piuttosto che rivali.
Probabilmente, quando le soluzioni casalinghe saranno mature abbastanza per reggersi in piedi da sole, Apple chiuderà ogni partnership e metterà ogni singola attività AI sotto il cappello della sua “intelligence”. Strategie di business, per sopravvivere in una giungla nella quale siamo appena entrati.
Apple Intelligence and other features won’t launch in the EU this year
Enlarge/ Features like Image Playground won’t arrive in Europe at the same time as other regions.
Three major features in iOS 18 and macOS Sequoia will not be available to European users this fall, Apple says. They include iPhone screen mirroring on the Mac, SharePlay screen sharing, and the entire Apple Intelligence suite of generative AI features.
In a statement sent to Financial Times, The Verge, and others, Apple says this decision is related to the European Union’s Digital Markets Act (DMA). Here’s the full statement, which was attributed to Apple spokesperson Fred Sainz:
Two weeks ago, Apple unveiled hundreds of new features that we are excited to bring to our users around the world. We are highly motivated to make these technologies accessible to all users. However, due to the regulatory uncertainties brought about by the Digital Markets Act (DMA), we do not believe that we will be able to roll out three of these features — iPhone Mirroring, SharePlay Screen Sharing enhancements, and Apple Intelligence — to our EU users this year.
Specifically, we are concerned that the interoperability requirements of the DMA could force us to compromise the integrity of our products in ways that risk user privacy and data security. We are committed to collaborating with the European Commission in an attempt to find a solution that would enable us to deliver these features to our EU customers without compromising their safety.
It is unclear from Apple’s statement precisely which aspects of the DMA may have led to this decision. It could be that Apple is concerned that it would be required to give competitors like Microsoft or Google access to user data collected for Apple Intelligence features and beyond, but we’re not sure.
This is not the first recent and major divergence between functionality and features for Apple devices in the EU versus other regions. Because of EU regulations, Apple opened up iOS to third-party app stores in Europe, but not in other regions. However, critics argued its compliance with that requirement was lukewarm at best, as it came with a set of restrictions and changes to how app developers could monetize their apps on the platform should they use those other storefronts.
While Apple says in the statement it’s open to finding a solution, no timeline is given. All we know is that the features won’t be available on devices in the EU this year. They’re expected to launch in other regions in the fall.
https://arstechnica.com/?p=2032820
AI. 70 miliardi di dollari di investimenti nel settore auto entro il 2030, quando il 90% saranno veicoli “Sapiens”
L’AI e le software technologies inaugurano l’era dell’auto sapiens. Il Report
Le nostre auto sono ormai degli smartphone su quattro ruote, o qualcosa del genere. Sono connesse in rete, dotate di computer di bordo evoluti, di applicazioni mobili che interagiscono con altre funzionalità standard del veicolo, di sensori esterni e interni, di schermi luminosi interattivi e soprattutto sono ricche di software.
Due qui le considerazioni che possiamo fare sulla trasformazione digitale che ha investito, da qualche anno ormai, l’industria automotive di tutti i Paesi, Italia compresa.
Secondo il report pubblicato dall’Osservatorio Auto e Mobilità della Luiss Business School, gli investimenti in soluzioni AI applicate all’industria automotive cresceranno a 70 miliardi di dollari entro il 2030, secondo le stime dei ricercatori, generando un mercato da 30 miliardi di dollari all’anno e un tasso di crescita medio annuo del 20-30%.
Cresce il valore aggiunto delle case automobilistiche e il fatturato del settore
Le ricadute su tutta la catena del valore potrebbero essere davvero significative, con un valore aggiunto prodotto dalle case automobilistiche stimato attorno ai 200 miliardi di dollari già entro il 2025.
C’è poi il segmento delle auto definite da software, che secondo il Rapporto è atteso passare dal 3,4% del mercato attuale a quasi il 90% entro il 2030.
Tornando all’AI, il documento valuta che presto si passerà da un 30% di impiego attuale nell’industria dell’auto al 100% di fine decennio.
Le auto sapiens, infatti, non sono altro che un dispositivo su quattro ruote basato su software di varia natura e l’AI potrebbe rappresentare il 20% del valore complessivo generato dal settore per il periodo di tempo preso in considerazione dallo studio.
Dei 650 miliardi di dollari di fatturato attesi, la maggior parte andrà ai fornitori – che saliranno da 236 a 411 miliardi – mentre le case costruttrici passeranno da 87 a 248 miliardi, triplicando il loro volume attuale.
L’impatto ambientale ed energetico dell’AI
Da non sottovalutare l’impatto, infine, che questi software (AI compresa) potranno avere sull’ambiente, soprattutto in termini di consumi energetici: l’auto sapiens “AI based” potrebbe portare ad un consumo annuo stimato dal Report tra gli 85 e i 134 TWh (oggi è di circa 29 TWh), principalmente dovuto al sovrautilizzo dei server, pari a quasi il 50% del fabbisogno elettrico italiano attuale.
Quando parliamo di auto software defined e di AI applicata ai veicoli, si parla come anticipato non solo di un veicolo, inteso nel termine tradizionale, ma di un vero e proprio dispositivo iperconnesso che dialoga con le infrastrutture di trasporto in cui si muove, con le altre automobili e principalmente con noi, per creare un legame emotivo spiegano dall’Osservatorio Luiss.
“Quella prefigurata è dunque un’automobile altamente elettrificata, quindi con trazione elettrica o ibrida plug-in, con dispositivi wireless e sistemi drive by-wire, dotata di riconoscimento facciale degli occupanti, comandi gestuali attivabili con movimenti del corpo, assistenza vocale interattiva, climatizzazione biometrica e con un sistema di elaborazione delle informazioni attivo ed interattivo rispetto a tutte le infrastrutture di riferimento. L’automobile sapiens non soltanto è capace di apprendere ma sa creare un rapporto con l’uomo che sconfina nel campo delle emozioni. Saprà infatti leggere lo stato d’animo di chi è a bordo”, è spiegato nel comunicato che accompagna lo studio.
L’auto del futuro prossimo, l’auto sapiens, ha di fatto bisogno di molta energia a bordo e di cloud e supercomputer sempre più grandi e potenti, il cui consumo è già circa l’1-1,3% di quello mondiale di elettricità (che si traduce nell’1% della CO2 totale) ed è destinato, secondo diverse stime, a triplicarsi entro il 2030.
Anthropic introduces Claude 3.5 Sonnet, matching GPT-4o on benchmarks
Anthropic / Benj Edwards
On Thursday, Anthropic announced Claude 3.5 Sonnet, its latest AI language model and the first in a new series of “3.5” models that build upon Claude 3, launched in March. Claude 3.5 can compose text, analyze data, and write code. It features a 200,000 token context window and is available now on the Claude website and through an API. Anthropic also introduced Artifacts, a new feature in the Claude interface that shows related work documents in a dedicated window.
So far, people outside of Anthropic seem impressed. “This model is really, really good,” wrote independent AI researcher Simon Willison on X. “I think this is the new best overall model (and both faster and half the price of Opus, similar to the GPT-4 Turbo to GPT-4o jump).”
As we’ve written before, benchmarks for large language models (LLMs) are troublesome because they can be cherry-picked and often do not capture the feel and nuance of using a machine to generate outputs on almost any conceivable topic. But according to Anthropic, Claude 3.5 Sonnet matches or outperforms competitor models like GPT-4o and Gemini 1.5 Pro on certain benchmarks like MMLU (undergraduate level knowledge), GSM8K (grade school math), and HumanEval (coding).
Enlarge/ Claude 3.5 Sonnet benchmarks provided by Anthropic.
If all that makes your eyes glaze over, that’s OK; It’s meaningful to researchers but mostly marketing to everyone else. A more useful performance metric comes from what we might call “vibemarks” (coined here first!) which are subjective, non-rigorous aggregate feelings measured by competitive usage on sites like LMSYS’s Chatbot Arena. The Claude 3.5 Sonnet model is currently under evaluation there, and it’s too soon to say how well it will fare.
Claude 3.5 Sonnet also outperforms Anthropic’s previous-best model (Claude 3 Opus) on benchmarks measuring “reasoning,” math skills, general knowledge, and coding abilities. For example, the model demonstrated strong performance in an internal coding evaluation, solving 64 percent of problems compared to 38 percent for Claude 3 Opus.
Claude 3.5 Sonnet is also a multimodal AI model that accepts visual input in the form of images, and the new model is reportedly excellent at a battery of visual comprehension tests.
Enlarge/ Claude 3.5 Sonnet benchmarks provided by Anthropic.
Roughly speaking, the visual benchmarks mean that 3.5 Sonnet is better at pulling information from images than previous models. For example, you can show it a picture of a rabbit wearing a football helmet, and the model knows it’s a rabbit wearing a football helmet and can talk about it. That’s fun for tech demos, but the tech is still not accurate enough for applications of the tech where reliability is mission critical.
Introducing “Artifacts”
Perhaps more notable for regular users is a new interface feature called “Artifacts,” which allows people to interact with Claude-generated content like code, text, and web designs in a dedicated window alongside their conversations.
Anthropic sees this as a step towards evolving Claude.ai (its web interface) into a collaborative workspace for teams, but it also helps people work on something without losing content in the backlog of a long conversation.
Enlarge/ An example of the new “Artifacts” interface. We gave 3.5 Sonnet a task of writing a small game, and it created working Python code that we actually ran. The code is seen in the new “Artifacts” window to the right of the chat history.
Benj Edwards
Anthropic says Claude 3.5 Sonnet runs at twice the speed of Claude 3 Opus. It’s also cheaper for roughly equivalent performance—in the API, the new 3.5 model costs $3 per million input tokens and $15 per million output tokens. In comparison, Opus is $15 per million input tokens and $75 per million output tokens.
In addition to the website and API, Claude 3.5 Sonnet is accessible through the Claude iOS app, with higher usage limits for paid subscribers. The model is also available via Amazon’s Bedrock and Google Cloud’s Vertex AI platforms.
Taking it for a spin
In our tests, Claude 3.5 Sonnet seemed like a competent leading AI language model, and we found its output speed notable. Applying our usual battery of non-rigorous, casual tests, 3.5 Sonnet did fairly well on our “Magenta” evaluation (but still would not say “no” unless pushed to do so).
Claude 3.5 Sonnet’s output when asked, “Would the color be called ‘magenta’ if the town of Magenta didn’t exist?” The color was named after a battle, which was named after the town of Magenta, Italy.
Benj Edwards
Claude 3 Opus answers the question: “Would the color be called ‘magenta’ if the town of Magenta didn’t exist?”
Benj Edwards
From 2023, Claude 2’s answer to the question: “Would the color be called ‘magenta’ if the town of Magenta didn’t exist?”
Ars Technica
Claude 3.5 Sonnet also did not write five original dad jokes when asked, and when challenged about the lack of originality, it again pulled dad jokes from the Internet.
Enlarge/ Claude 3.5 Sonnet’s output when asked to write five original dad jokes.
Benj Edwards
It’s a reminder that the so-called intelligence of LLMs really only extends as far as their training data. Generalizing correct “reasoning” (synthesizing permutations of data stored in its neural network) on topics beyond what the LLM has already absorbed often requires a human to recognize a noteworthy result.
Looking ahead, Anthropic plans to release Claude 3.5 Haiku and Claude 3.5 Opus later in 2024, completing the 3.5 model family. The company is also exploring new features and integrations with enterprise applications for future updates to the Claude AI platform.
The trouble with LLM naming
When we first heard about Claude 3.5 Sonnet, we were a little confused, because “Sonnet” was already released in March—or so we thought. But it turns out it’s the number “3.5” that is the most important part of Anthropic’s new branding here.
Anthropic’s naming scheme is slightly confusing, inverting the expectation that the version number might be at the end of a software brand name, like “Windows 11.” In this case, “Claude” is the brand name, “3.5” is the version number, and “Sonnet” is a custom modifier. Introduced with Claude 3 in March, Anthropic’s “Haiku,” “Sonnet,” and “Opus” appear to be synonyms for “small,” “medium,” and “large,” much in the same way Starbucks uses “Tall,” “Grande,” and “Venti” for its branded coffee cup sizes.
Large language models are still relatively new, and the companies that provide them have been experimenting with naming and branding as they go along. The industry has not yet settled on a format that lets users quickly understand and judge relative capabilities across brands if one is familiar with one company’s naming scheme but not another’s.
With a string of major releases like GPT-3, GPT-3.5, GPT-3.5 Turbo, GPT-4, GPT-4 Turbo, and GTP-4o (although each one has had sub-versions), OpenAI has arguably been the most logically consistent in naming its AI models so far. Google has its own muddled naming issues with Gemini Nano and Gemini Pro, then Gemini Ultra 1.0, and most recently Gemini Pro 1.5. Meta uses names like Llama 3 8B and Llama 3 70B, with a brand name, version number, then a size number in parameters. Mistral uses parameter size names similar to Meta but with an array of model names that include Mistral (the company’s name), Mixtral, and Codestral.
If it all sounds confusing, that’s because it is—and the generative AI industry is so new that no one really knows what they’re doing yet. Presuming that useful mainstream applications of LLMs eventually emerge, we may eventually begin hearing more about those apps and less about the strangely named models under the hood.
