Il 2025 segna il punto di non ritorno per la fiducia digitale. Con deepfake creabili in tempo reale per meno di 2 euro e perdite globali che superano i 40 miliardi di dollari, l’identità online come la conosciamo sta collassando. Un’ingegnera di Hong Kong trasferisce 25,6 milioni di dollari dopo una videoconferenza con colleghi che non esistono. Un’azienda di cybersecurity blocca un attore nordcoreano con un volto generato dall’AI prima che possa infiltrarsi. Migliaia di elettori del New Hampshire ricevono una chiamata dal “Presidente Biden” che li invita a non votare. Non sono scene di Black Mirror: è il 2025, e l’era dell’identità digitale sicura sta volgendo al termine.

I numeri parlano di una rivoluzione tecnologica senza precedenti: 95.820 video deepfake rilevati nel 2023, con un aumento del 550% dal 2019. Il costo medio per creare un deepfake convincente? 1,33 dollari. Il danno medio per un’azienda vittima nel settore finanziario? 603.000 dollari. Questa asimmetria economica sta ridefinendo le regole del gioco digitale, trasformando ogni interazione online in un potenziale campo minato.

La democratizzazione dell’inganno perfetto

La tecnologia deepfake ha raggiunto nel 2024-2025 un punto di svolta critico. Software come DeepFaceLive e Deep-Live-Cam permettono ora la manipolazione video in tempo reale con latenze estremamente ridotte – abbastanza veloce da ingannare durante una videochiamata. Con soli 3 secondi di audio registrato, gli algoritmi attuali possono clonare una voce con un’accuratezza dell’85%. Non servono più competenze tecniche specializzate: basta un computer con una scheda grafica decente e 10 minuti di registrazione su una piattaforma cloud.

Le ricerche per “software gratuito di clonazione vocale” sono aumentate del 120% tra luglio 2023 e 2024. Il mercato globale dei deepfake, valutato tra 857 milioni e 7,58 miliardi di dollari nel 2024 secondo diverse analisi, potrebbe raggiungere i 41,36 miliardi entro il 2032. Dietro questi numeri si nasconde una realtà inquietante: la barriera tecnologica che proteggeva l’identità digitale è crollata.

“Un paio di anni fa servivano competenze tecniche e intenzionalità per creare deepfake; ora basta solo l’intenzionalità”, osserva Hugh Thompson, Executive Chairman della RSA Conference. La facilità d’uso ha trasformato una tecnologia di nicchia in un’arma di distruzione di massa della fiducia digitale. 2.298 strumenti per face swap, 10.206 per generazione di immagini AI, 1.018 per clonazione vocale: l’arsenale dell’inganno digitale è ormai alla portata di chiunque.

Quando il crimine diventa indistinguibile dalla realtà

Il caso Arup resta emblematico della nuova era criminale. All’inizio del 2024, un dipendente della multinazionale britannica di ingegneria ha partecipato a quella che sembrava una normale videoconferenza aziendale. Il CFO e altri colleghi discutevano di trasferimenti urgenti. Tutto sembrava autentico: le voci, i volti, persino i manierismi. Il dipendente ha autorizzato 15 transazioni per un totale di 25,6 milioni di dollari. Solo giorni dopo si è scoperto che tutti i partecipanti alla chiamata erano deepfake generati in tempo reale.

“Audio e segnali visivi sono fondamentali per noi esseri umani, e queste tecnologie stanno giocando proprio su questo”, ammette Rob Greig, CIO di Arup. “Dobbiamo davvero iniziare a mettere in discussione ciò che vediamo.”

Le statistiche del crimine digitale riflettono questa nuova realtà. L’88% di tutti i casi di frode deepfake colpisce il settore delle criptovalute. Il fintech subisce significativi attacchi KYC (Know Your Customer) fraudolenti. In Nord America, l’aumento delle frodi deepfake ha raggiunto il 1.740% tra il 2022 e il 2023. Le proiezioni di Deloitte sono ancora più allarmanti: 40 miliardi di dollari di perdite per frode negli Stati Uniti entro il 2027, rispetto ai 12,3 miliardi del 2023.

Ma i deepfake non colpiscono solo i conti bancari. Il caso del preside Eric Eiswert della Pikesville High School nel Maryland illustra il potere distruttivo sulla reputazione personale. Un audio deepfake con contenuti razzisti e antisemiti, creato dall’insegnante Dazhon Darien per vendetta, ha ottenuto 2 milioni di visualizzazioni online prima che la polizia confermasse la falsificazione. Nel frattempo, Eiswert aveva già ricevuto minacce di morte ed era stato sospeso dal lavoro. Darien è stato condannato a 4 mesi di carcere nell’aprile 2025.

Un altro caso significativo ha coinvolto KnowBe4, azienda di cybersecurity che il 15 luglio 2024 ha scoperto di aver quasi assunto un attore nordcoreano che aveva utilizzato un’identità AI-generata durante il processo di selezione. L’infiltrazione è stata fermata prima che potesse causare danni, evidenziando come anche le aziende più preparate possano essere vulnerabili a questi attacchi sofisticati.

L’erosione sistemica della fiducia sociale

L’impatto dei deepfake va ben oltre i singoli casi di frode. Solo il 42% degli americani sa cos’è un deepfake, secondo il Pew Research Center, ma il 77% vorrebbe restrizioni su video e immagini alterate. Questo paradosso rivela una società impreparata ad affrontare una minaccia che non comprende pienamente.

La capacità umana di riconoscere i deepfake è desolante: solo il 61% delle persone riesce a distinguere volti generati dall’AI da quelli reali, appena meglio del lancio di una moneta. Anche quando vengono avvertiti, gli individui sovrastimano significativamente la propria capacità di rilevamento. Gli over 65 mostrano la stessa capacità di riconoscimento dei giovani, ma una maggiore suscettibilità agli effetti psicologici.

“Stiamo entrando in un’era post-verità dove gli individui credono solo alle informazioni che si allineano con le loro convinzioni preesistenti”, avverte un rapporto del Carnegie Endowment. Il fenomeno del “dividendo del bugiardo” – dove anche la sola esistenza dei deepfake permette ai malintenzionati di screditare prove autentiche – sta minando le fondamenta stesse del discorso democratico.

Le elezioni americane del 2024 hanno visto 82 deepfake targeting figure pubbliche in 38 paesi. Sebbene l’impatto diretto sia stato limitato, l’effetto corrosivo sulla fiducia istituzionale è profondo. La ricerca mostra che i deepfake che ritraggono fallimenti infrastrutturali riducono la fiducia nel governo, con effetti più pronunciati in ambienti polarizzati come gli Stati Uniti rispetto a Singapore.

Il caso della robocall deepfake del “Presidente Biden” del 21 gennaio 2024 nel New Hampshire rappresenta un precedente preoccupante: migliaia di elettori hanno ricevuto una chiamata con la voce sintetica del presidente che li invitava a non votare alle primarie democratiche, dimostrando il potenziale di manipolazione elettorale di queste tecnologie.

La corsa tecnologica tra spada e scudo

Intel proclama che il suo FakeCatcher può rilevare deepfake con un’accuratezza del 96% analizzando i sottili cambiamenti del flusso sanguigno nei pixel video. Microsoft Video Authenticator prometteva valutazioni in tempo reale dell’autenticità dei media, ma non è più ampiamente disponibile al pubblico. Reality Defender ha raccolto 33 milioni di dollari (espandendo il round iniziale da 15 milioni) per la sua piattaforma multimodale. Eppure, la realtà sul campo è molto diversa dalle promesse dei comunicati stampa.

Il benchmark Deepfake-Eval-2024 rivela una verità scomoda: i modelli mostrano un calo medio delle prestazioni del 45-50% sui dati del mondo reale rispetto ai dataset di laboratorio. I migliori sistemi commerciali raggiungono solo il 78% di accuratezza per i video, l’89% per l’audio, l’82% per le immagini. Peggio ancora, i modelli addestrati su deepfake generati da GAN falliscono completamente contro le nuove tecniche di diffusione.

“Siamo a decenni dall’avere una tecnologia forense che possa distinguere in modo conclusivo il vero dal falso”, ammette il professor Hany Farid, pioniere della tecnologia PhotoDNA presso UC Berkeley. La transizione dai modelli GAN ai modelli di diffusione ha reso obsoleti molti sistemi di rilevamento esistenti. I Binary Neural Networks riducono i requisiti computazionali di 20 volte mantenendo l’accuratezza, ma anche questi progressi non riescono a tenere il passo con l’evoluzione delle tecniche di generazione.

I sistemi di verifica biometrica, un tempo considerati il gold standard della sicurezza, vacillano. Gartner prevede che entro il 2026, il 30% delle aziende non considererà più affidabili i sistemi di autenticazione biometrica facciale a causa degli attacchi deepfake. Le tecnologie di rilevamento della vivacità, che analizzano texture della pelle e pattern di flusso sanguigno, offrono una certa protezione ma possono essere aggirate con tecniche sempre più sofisticate.

Il labirinto normativo globale

L’Unione Europea guida la risposta normativa con l’AI Act, entrato in vigore il 1° agosto 2024. La legge definisce i deepfake come “contenuti immagine, audio o video generati o manipolati dall’AI che assomigliano a persone, oggetti, luoghi esistenti e apparirebbero falsamente autentici a una persona”. Le violazioni possono comportare multe fino a 35 milioni di euro o il 7% del fatturato annuale mondiale.

Gli Stati Uniti hanno compiuto passi significativi con il TAKE IT DOWN Act, firmato in legge dal Presidente Trump il 19 maggio 2025, che criminalizza la pubblicazione di immagini intime non consensuali, inclusi i deepfake, con pene fino a 3 anni di carcere per violazioni aggravate. Ma l’approccio frammentato stato per stato crea un mosaico complesso: 47 stati hanno almeno una legge sui deepfake, 26 regolano i deepfake elettorali, 45 affrontano i deepfake sessualmente espliciti.

L’Italia ha sviluppato una legislazione specifica con il DDL approvato dal Senato il 16 settembre 2025, che prevede investimenti fino a 1 miliardo di euro per l’innovazione AI tramite CDP Venture Capital e pene detentive da 1 a 5 anni per la distribuzione di deepfake dannosi. Il caso della premier Giorgia Meloni, che ha chiesto €100.000 di danni simbolici per video pornografici deepfake nel luglio 2024, evidenzia le sfide dell’applicazione transfrontaliera.

Ma le leggi faticano a tenere il passo con la tecnologia. “Le definizioni legali tradizionali di diffamazione e copyright sono inadeguate per affrontare i danni dei deepfake”, nota un rapporto della Electronic Frontier Foundation. L’onere della prova, la verifica dell’autenticità, la giurisdizione transnazionale: ogni aspetto del sistema legale viene messo alla prova da questa nuova realtà.

Scenari per il futuro prossimo

Vijay Balasubramaniyan, CEO di Pindrop Security, prevede che “il 2025 potrebbe vedere la prima violazione su larga scala di archivi audio e video confidenziali che potrebbero essere utilizzati per addestrare AI per scopi malevoli”. Ajay Amlani, recentemente nominato CEO di Aware Inc. nel gennaio 2025, è ancora più specifico: “La frode deepfake diventerà completamente weaponizzata nel 2025”, con “un’ondata di takeover di account e transazioni fraudolente che costringerà i regolatori bancari mondiali ad agire in modo decisivo”.

Le proiezioni economiche dipingono un quadro cupo. Il mercato dei deepfake potrebbe raggiungere i 35-41 miliardi di dollari entro il 2032-2034. Le perdite per frode sono destinate a crescere con un CAGR del 32%. Entro il 2027, gli attacchi di social engineering potenziati dall’AI colpiranno il 40% dei dirigenti aziendali.

Eppure, non tutto è perduto. Il World Economic Forum sostiene che “nessun singolo stakeholder o soluzione può affrontare completamente i media sintetici”. Il successo richiederà “nuove tecnologie, pratiche organizzative ed educazione”. Steven Smith di Tools for Humanity indica soluzioni tecnologiche come World ID, che ha aggiunto il rilevamento deepfake al suo sistema di identità digitale.

La trasformazione, non la fine, dell’identità digitale

Contrariamente alle previsioni apocalittiche, gli esperti convergono su uno scenario di trasformazione piuttosto che di collasso totale. L’identità digitale sicura come la conosciamo – basata sulla fiducia implicita in ciò che vediamo e sentiamo – sta effettivamente finendo. Ma al suo posto sta emergendo un nuovo paradigma: sistemi multi-livello che combinano autenticazione crittografica, biometria comportamentale e verifiche incrociate.

Le organizzazioni stanno adottando architetture “Zero Trust” – passando da “fidati ma verifica” a “non fidarti mai, verifica sempre”. La Coalition for Content Provenance and Authenticity (C2PA) sta stabilendo standard per l’autenticazione dei media, con membri che includono Adobe, BBC, Intel, Microsoft, Google e OpenAI. Blockchain e firme crittografiche promettono di creare catene di custodia verificabili per i contenuti digitali.

L’educazione diventa cruciale. “Dobbiamo davvero iniziare a mettere in discussione ciò che vediamo”, insiste Rob Greig di Arup. Ma questo scetticismo sistematico ha un costo psicologico. La “doppelgänger-fobia” – la paura di avere cloni AI usati senza consenso – sta emergendo come nuovo disturbo d’ansia. La dissonanza cognitiva nel riconciliare informazioni conflittuali su cosa sia reale sta frammentando il senso di identità personale.

Conclusione: navigare nell’era della verità sintetica

Il 2025 non segna la fine dell’identità digitale, ma la fine dell’innocenza digitale. In un mondo dove creare un deepfake costa meno di un caffè ma può distruggere vite e svuotare conti bancari, la fiducia diventa una risorsa da conquistare, non da presumere. Le aziende che perderanno questa battaglia pagheranno in media 603.000 dollari per incidente nel settore finanziario. Gli individui pagheranno con la loro reputazione, la loro privacy, la loro sanità mentale.

La soluzione non sta nel rifiutare la tecnologia né nell’accettarla acriticamente, ma nel costruire nuovi framework di fiducia adatti all’era dell’AI generativa. Questo richiederà investimenti massicci in tecnologie di rilevamento, riforme legislative coordinate a livello globale, e soprattutto un cambiamento culturale profondo nel modo in cui percepiamo e verifichiamo la realtà digitale.

Come società, siamo a un bivio. Possiamo permettere che i deepfake distruggano le fondamenta della fiducia digitale, o possiamo usare questa crisi come catalizzatore per costruire sistemi più robusti, trasparenti e resilienti. La scelta che faremo nei prossimi 2-3 anni determinerà se l’era digitale diventerà un’epoca di inganno universale o di verità verificabile.

Il tempo per agire è ora. Perché in un mondo dove nulla è come sembra, tutto dipende da come scegliamo di guardare.

Fonti:

American Bar Association. (2024). The “Deepfake Defense”: An Evidentiary Conundrum. Judges’ Journal.

Brookings Institution. (2024). Artificial intelligence, deepfakes, and the uncertain future of truth.

California Law Review. (2024). Deep Fakes: A Looming Challenge for Privacy, Democracy, and National Security.

European Commission. (2024). AI Act | Shaping Europe’s digital future.

MIT News. (2024). 3 Questions: What you need to know about audio deepfakes.

Pew Research Center. (2023). What Americans Know About AI, Cybersecurity and Big Tech.

Stanford University IT. (2024). Dangers of Deepfake: What to Watch For.

CNN Business. (2024). Arup revealed as victim of $25.6 million deepfake scam involving Hong Kong employee.

CNN. (2024). Pikesville High School’s principal was accused of offensive language on a recording. Authorities now say it was a deepfake.

NBC News. (2024). Fake Biden New Hampshire robocall most likely AI-generated.

CyberScoop. (2024). Cyber firm KnowBe4 hired a fake IT worker from North Korea.

CNN. (2024). Italian Prime Minister Giorgia Meloni seeking damages of $108,200 in deepfake porn trial.

Deloitte Insights. (2024). Deepfake banking and AI fraud risk.

Security.org. (2024). 2024 Deepfakes Guide and Statistics.

Sumsub. (2024). Global Deepfake Incidents Surge Tenfold from 2022 to 2023.

Business Wire. (2024). Deepfake Fraud Costs the Financial Sector an Average of $603,000 for Each Company, Regula’s Survey Reveals.

Gartner. (2024). 30% of Enterprises Will Consider Identity Verification Solutions Unreliable Due to Deepfakes by 2026.

Intel Newsroom. (2022). Intel Introduces Real-Time Deepfake Detector.

arXiv. (2025). Deepfake-Eval-2024: A Multi-Modal In-the-Wild Benchmark of Deepfakes.

Reality Defender. (2024). Reality Defender Expands Series A to $33 Million.

European Commission. (2024). AI Act enters into force.

Orrick. (2025). TAKE IT DOWN Act Becomes Law.

Ballotpedia. (2025). Forty-seven states have enacted deepfake legislation since 2019.

Gametimers. (2025). Il Senato approva il ddl delega sull’IA: deepfake diventa reato.

C2PA. (2024). Coalition for Content Provenance and Authenticity.

World Economic Forum. (2025). Cybercrime: Lessons learned from a $25m deepfake attack.

UC Berkeley. (2025). Hany Farid Reflects on New Research.

RSA Conference. (2024). Hugh Thompson Profile.

Fast Company. (2024). Vijay Balasubramaniyan Profile.

Aware Inc. (2025). Aware names digital identity industry veteran Ajay Amlani CEO.

Tools for Humanity. (2024). World ID Profile.

NPR. (2025). A Baltimore-area teacher is accused of using AI to make his boss appear racist.

Security Hero. (2023). 2023 State Of Deepfakes: Realities, Threats, And Impact

https://www.ictsecuritymagazine.com/articoli/identita-digitale/

https://www.adweek.com/agencies/publicis-taps-deepti-velury-to-lead-production-debuts-leona-ai-engine/

At the IAB Podcast Upfront in Midtown Manhattan on Tuesday, scores of media executives, celebrities, audio creators, and adtech vendors graced the stage to extol the virtues of podcasting.

In a series of presentations interspersed with frenzied networking breaks, panelists hit all the classic talking points: the intimacy of the medium and the efficacy it affords for advertisers; the challenges and opportunities in measurement; the need to embrace a multipronged commercial strategy; and, of course, the embrace of video that’s begun to transform the entire medium.

One topic, however, was conspicuously absent: artificial intelligence. If the technology came up at all, it was as a tool to achieve backend efficiencies, such as creative optimization, audience targeting, and streamlined editing—not as a threat to the industry, nor in any way part of the actual process of creating the podcasts themselves.

.newsletter-subscription-form-wrapper p:empty{ display: none !important } .newsletter-subscription-form-wrapper button br{ display: none !important }

As an unfortunate regular of the conference circuit, I can hardly overemphasize how unusual this is. Nearly every media industry event I attend immediately devolves into a parlor game of AI bingo, with the audience and panelists all waiting patiently for the subject to arise and subsume the rest of the conversation. In every other part of publishing, there is no more pressing subject matter and no greater object of hyperfixation than AI.

To compound the oddity, news of a controversial startup in the audio space has recently worked the industry into a lather. Called Inception Point AI, the company has made headlines by championing a strategy predicated on using AI-generated personalities to create podcasts, with the broader goal of turning the personalities into influencers.

The Hollywood Reporter article introducing the startup—“5,000 Podcasts. 3,000 Episodes a Week. $1 Cost Per Episode”—set off a firestorm of indignation, with thought leaders and executives in the audio industry decrying the output as “slop.” 

Set against the backdrop of this uproar, the exclusion of AI from any of the Upfront programming was particularly baffling. Clearly the barbarians are at the gates. Just as ChatGPT transformed text-based production when it was introduced two years ago, the debut of Inception Point, and the launch this week of both Meta’s and OpenAI’s AI-generated social video feeds, feels like an inflection point. 

‘Carbon-based’ creators

Just like its predecessor, the mission and methodology of Inception Point have been roundly criticized. Several audio executives I spoke with insisted that the intrusion of AI into the world of podcast creation would be limited to the fringes, if at all.

Their arguments border on the sentimental. If you’ve ever spoken with anyone involved in professional podcasting, the craft can be discussed with the kind of reverence that most people reserve for houses of worship. The words authenticity, connection, intimacy, and even magic come up frequently. It can get very woo-woo.

And yet, do you have a favorite podcast? As treacly as it sounds, the medium does yield a sense of intimacy unlike any other, one often rooted in a sense of connection that the listener has with the host. As a result, its practitioners are confident—adamant, almost—that it will be largely spared from the ravages of AI. 

“It’s obvious that creators need to be carbon-based organisms,” said Greg Glenday, CEO of the podcast company Acast. “We use AI around the edges, but our pitch to advertisers is authenticity. That is our moat and our five-year plan.”

Others echoed the sentiment that AI poses less of a threat to podcast hosts than it does to writers or even video creators. 

“As long as podcasting remains something people think of as an authentic experience,” said Matt Shapo, the director of digital audio and video at the IAB, “I’m not sure we’ll ever have the same level of disruption when it comes to content production.”

There are exceptions, of course. 

Certain genres of podcast are likely to be outsourced to AI, the same way commodity text content has been replaced by ChatGPT. Podcasts recapping sports games, detailing the weather, or offering a synopsis of a news article are already popular use cases for the technology. In these cases, according to Glenday, listeners have a connection to the content, not the creator. 

“Commodity podcasting—news, weather, daily rundowns—could be replaced,” said Alison Tucker, an associate director of audio investment at Omnicom Media Group. “But basically everything else? That’s much harder.”

The divide is reflective of a wider trend reshaping digital media. As AI dramatically reduces the effort required to create generic content, publishers and consumers are increasingly gravitating toward creator-led content, where personality can distinguish it from the deluge of information now inundating the web.

Robot callers

Still, the confidence among podcasters is disorienting, as AI has engendered existential dread into nearly every other sector of the economy. 

That optimism—misplaced or otherwise—is also exactly where Inception Point sees its opportunity, according to CEO and cofounder Jeanine Wright. 

The idea for the company came from the pandemic, when cofounder William Corbin created a popular podcast, called Covid 411, simply by reading the daily CDC update into a mic. The series spurred Corbin to think about other ways to make timely content that people want with low production costs, Wright said. 

While the ambitions of the startup are broader than podcasts, the company started with the medium because the technology surrounding synthetic audio is already advanced enough that it’s difficult to tell the difference between a real and artificial voice. 

By combining that capability with timely subject matter, the company aims to produce quick, low-lift audio products that only need to find a small audience to recoup their minimal costs. Already, Inception Point has more than 4,000 of these AI-generated shows, publishing thousands of episodes a week on platforms including iHeartMedia’s Spreaker. (Some of the flash-published series include biographies of zeitgeist subjects including Austin Butler, Ozzy Osborne, and Charlie Kirk.) 

The strategy positions the company specifically for the niche of commodity audio content that podcast executives have conceded is already susceptible to bot-sourcing. The larger question is whether audiences will embrace its entertainment output, which the company plans to expand to feature a cast of AI-generated personalities in conversation with one another, discussing real and fictional subjects through the filters of their lab-designed personalities. 

“Most of the pushback is because people have not stayed on the cutting-edge of AI development,” Wright said. “If people say that AI creators will never be able to get to the emotionality of a human creator, I think they don’t understand the capabilities of the tools today and where they will be in the future.”

That such a product could ever take off admittedly feels like a stretch. And yet, by now I am loath to bet against the steady advance of AI into every crevice of creative output. 

So who is right: the podcast professionals betting on their moat of authenticity, or the Hollywood startup farming out its editorial to bots? 

“I do have a bit of anxiety,” Glenday said. “Sometimes, you get nervous when everyone agrees.”

https://www.adweek.com/media/ai-podcasts-inception-point-onbackground/

Several weaknesses patched recently by Google in Gemini could have allowed attackers to trick the AI assistant into helping them achieve data theft and other malicious goals. 

The issues were discovered by researchers at cybersecurity firm Tenable, who named the project The Gemini Trifecta. The research covers three distinct Gemini hacking methods that abused various features and tools, and which required little to no social engineering.

The first attack involved indirect prompt injection and it targeted Gemini Cloud Assist, which enables users to interact with Google Cloud for managing and optimizing cloud operations.

The attack abused Gemini Cloud Assist’s ability to analyze logs. The researchers discovered that an attacker could send a specially crafted request to the targeted organization, which would result in a malicious prompt being added to log files.

When a user asked Cloud Assist to explain the log entry or to analyze logs for various purposes, Gemini would process the attacker’s message. In Tenable’s demonstration, the attacker convinced Gemini to display a link to a Google phishing page.

The researchers discovered several Google Cloud services that could have been targeted by an unauthenticated attacker with specially crafted requests that would result in a log entry, including Cloud Functions, Cloud Run, App Engine, Compute Engine, Cloud Endpoints, API Gateway, and Load Balancing. 

“One impactful attack scenario would be an attacker who injects a prompt that instructs Gemini to query all public assets, or to query for IAM misconfigurations, and then creates a hyperlink that contains this sensitive data. This should be possible since Gemini has the permission to query assets through the Cloud Asset API,” Tenable researchers explained. 

“Since the attack can be unauthenticated, attackers could also ‘spray’ attacks on all GCP public-facing services, to get as much impact as possible, rather than a targeted attack,” they added.

In the second attack method, which also involved indirect prompt injection, the researchers used search history as a prompt injection vector. Specifically, they abused Gemini’s Search Personalization, a feature that allows the AI to provide more relevant and tailored responses based on a user’s personal context and past activity. 

In this case, an attacker would have needed to convince a user to visit a website that they had set up to inject malicious search queries containing prompt injections into the victim’s browsing history. When the victim later interacted with Gemini’s search personalization model, it would process the attacker’s instructions, which could include commands to collect sensitive user data and exfiltrate it when the victim clicked on a link.

The third attack in the trifecta targeted the Gemini Browsing Tool, which enables the AI to understand content on the web and perform tasks using the context of open tabs and browsing history. 

The researchers managed to abuse this tool’s ability to summarize a web page to create a side channel for data exfiltration. They convinced the AI to take the victim’s saved information and add it to a request sent to a remote server controlled by the attacker. 

Tenable said Google patched all three vulnerabilities after being notified.

Researchers in recent weeks demonstrated several similar attack methods targeting widely used AI assistants and their integration with enterprise products. 

Related: ChatGPT Tricked Into Solving CAPTCHAs

Related: California Gov. Gavin Newsom Signs Bill Creating AI Safety Measures

Related: Salesforce AI Hack Enabled CRM Data Theft

https://www.securityweek.com/google-patches-gemini-ai-hacks-involving-poisoned-logs-search-results/

California Gov. Gavin Newsom on Monday signed a law that aims to prevent people from using powerful artificial intelligence models for potentially catastrophic activities like building a bioweapon or shutting down a bank system.

The move comes as Newsom touted California as a leader in AI regulation and criticized the inaction at the federal level in a recent conversation with former President Bill Clinton. The new law will establish some of the first-in-the-nation regulations on large-scale AI models without hurting the state’s homegrown industry, Newsom said. Many of the world’s top AI companies are located in California and will have to follow the requirements.

“California has proven that we can establish regulations to protect our communities while also ensuring that the growing AI industry continues to thrive. This legislation strikes that balance,” Newsom said in a statement.

The legislation requires AI companies to implement and disclose publicly safety protocols to prevent their most advanced models from being used to cause major harm. The rules are designed to cover AI systems if they meet a “frontier” threshold that signals they run on a huge amount of computing power.

Such thresholds are based on how many calculations the computers are performing. Those who crafted the regulations have acknowledged the numerical thresholds are an imperfect starting point to distinguish today’s highest-performing generative AI systems from the next generation that could be even more powerful. The existing systems are largely made by California-based companies like Anthropic, Google, Meta Platforms and OpenAI.

The legislation defines a catastrophic risk as something that would cause at least $1 billion in damage or more than 50 injuries or deaths. It’s designed to guard against AI being used for activities that could cause mass disruption, such as hacking into a power grid.

Companies also have to report to the state any critical safety incidents within 15 days. The law creates whistleblower protections for AI workers and establishes a public cloud for researchers. It includes a fine of $1 million per violation.

It drew opposition from some tech companies, which argued that AI legislation should be done at the federal level. But Anthropic said the regulations are “practical safeguards” that make official the safety practices many companies are already doing voluntarily.

“While federal standards remain essential to avoid a patchwork of state regulations, California has created a strong framework that balances public safety with continued innovation,” Jack Clark, co-founder and head of policy at Anthropic, said in a statement.

The signing comes after Newsom last year vetoed a broader version of the legislation, siding with tech companies that said the requirements were too rigid and would have hampered innovation. Newsom instead asked a group of several industry experts, including AI pioneer Fei-Fei Li, to develop recommendations on guardrails around powerful AI models.

The new law incorporates recommendations and feedback from Newsom’s group of AI experts and the industry, supporters said. The legislation also doesn’t put the same level of reporting requirements on startups to avoid hurting innovation, said state Sen. Scott Wiener of San Francisco, the bill’s author.

“With this law, California is stepping up, once again, as a global leader on both technology innovation and safety,” Wiener said in a statement.

Newsom’s decision comes as President Donald Trump in July announced a plan to eliminate what his administration sees as “onerous” regulations to speed up AI innovation and cement the U.S.’ position as the global AI leader. Republicans in Congress earlier this year unsuccessfully tried to ban states and localities from regulating AI for a decade.

Without stronger federal regulations, states across the country have spent the last few years trying to rein in the technology, tackling everything from deepfakes in elections to AI “therapy.” In California, the Legislature this year passed a number of bills to address safety concerns around AI chatbots for children and the use of AI in the workplace.

California has also been an early adopter of AI technologies. The state has deployed generative AI tools to spot wildfires and address highway congestion and road safety, among other things.

https://www.securityweek.com/california-gov-gavin-newsom-signs-bill-creating-ai-safety-measures/

Live Webinar | Tuesday, September 30th at 1PM ET – Register

Artificial intelligence is transforming enterprises, but with innovation comes risk. Over half of organizations worry AI will accelerate identity fraud, and nearly half of leaders admit they can’t confidently spot a deepfake. The question is: how do you embrace AI’s potential while defending against its threats?

Join this live webinar with experts from Ping Identity, where they will share a blueprint for enabling innovation securely.

Join thewebinar to learn:

• The AI Threat Landscape: Why 95% of organizations are expanding budgets for AI defense
• Fighting Fire with Fire: How AI detects not only known attacks but novel threats
• The Rise of AI Agents: Preparing for a future where identity involves both human and non-human actors

Don’t miss this opportunity to rethink how trust is built in the age of AI.

https://www.securityweek.com/webinar-today-ai-and-the-trust-dilemma-balancing-innovation-and-risk/

Microsoft said Thursday it had disabled services to a unit within the Israeli military after a company review had determined its artificial intelligence and cloud computing products were being used to help carry out mass surveillance of Palestinians.

The action comes after The Associated Press and The Guardian published reports earlier this year revealing how the Israeli Ministry of Defense had been using Microsoft’s Azure platform to aid in the war in Gaza and occupation of the West Bank. Brad Smith, Microsoft’s vice chair and president, wrote in a blog post that the company was taking steps to enforce compliance with its terms of service.

An AP investigation in February showed that the Israeli military’s use of Microsoft products skyrocketed after a deadly surprise attack by Hamas militants on Oct. 7, 2023. The AP’s report cited internal Microsoft data showing the Israelis were using gigabytes of cloud storage and massive amounts of AI-enabled language translation services.

The AP also reported that Israel’s military used Microsoft Azure to compile information gathered through mass surveillance, which it transcribes and translates, including phone calls and text messages. That intelligence is then cross-checked with Israel’s in-house AI systems for targeting airstrikes.

AP reported that internal Microsoft data showed multiple Azure subscriptions were tied to Unit 8200, an elite cyber warfare unit within the Israeli Army responsible for clandestine operations, collecting signal intelligence and surveillance.

Following AP’s report, Microsoft acknowledged in May that it had sold advanced AI and cloud computing services to the Israeli military during the Gaza war and aided in efforts to locate and rescue Israeli hostages. But the company said an internal review found “no evidence” its Azure platform was used to target or harm people.

The Guardian, working in partnership with the Israeli-Palestinian publication +972 Magazine and the Hebrew-language outlet Local Call, reported in August that the commander of Unit 8200 had met directly with Microsoft chairman and CEO Satya Nadella in 2021. The Israeli unit then used Microsoft products to aid in the development of an AI-powered mass surveillance system that was sweeping up, translating and analyzing millions of telephone calls per day made by Palestinian civilians. The report also revealed that data from the Israeli surveillance system was being stored at Microsoft cloud data centers in Europe.

Following The Guardian’s report, Microsoft commissioned a second review, this time by an outside law firm. While that review is still ongoing, Smith said Thursday the probe had uncovered evidence that its products were being used in violation of its terms of service. However, Smith did not name the specific Israeli unit losing access to Microsoft services.

Microsoft declined to answer detailed questions from the AP on Thursday, including whether Unit 8200 was involved. The company would also not answer how it would ensure the Israeli military wouldn’t simply shift its mass surveillance operations to any of the hundreds of other Azure subscriptions under its control.

An Israeli security official told the AP Microsoft’s move would produce “no damage to the operational capabilities” of the Israel Defense Forces. The official spoke on condition of anonymity, consistent with military protocol in Israel.

Hossam Nasr, one of more than a dozen Microsoft employees fired or arrested after protests over the company’s involvement in the war in Gaza, called Thursday’s announcement a “significant and unprecedented win.” But, he said, it was not enough.

“Microsoft has only disabled a small subset of services to only one unit in the Israeli military,” said Nasr, an organizer with the group No Azure for Apartheid. “The vast majority of Microsoft’s contract with the Israeli military remains intact.”

Related: Google Ships Android ‘Advanced Protection’ Mode to Thwart Surveillance Spyware

Related: Surveillance Firm Bypasses SS7 Protections to Retrieve User Location

https://www.securityweek.com/microsoft-reduces-israels-access-to-cloud-and-ai-products-over-reports-of-mass-surveillance-in-gaza/

AI security platform SPLX has demonstrated that prompt injections can be used to bypass a ChatGPT agent’s built-in policies and convince it to solve CAPTCHAs.

AI agents have guardrails in place to prevent them from solving any CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart), based on ethical, legal, and platform-policy reasons.

When asked directly, a ChatGPT agent refuses to solve a CAPTCHA, but anyone can apparently use misdirection to trick the agent into giving its consent to solve the test, and this is what SPLX demonstrated.

In a regular ChatGPT-4o chat, they told the AI they wanted to solve a list of fake CAPTCHAs and asked it to agree to performing the operation.

“This priming step is crucial to the exploit. By having the LLM affirm that the CAPTCHAs were fake and the plan was acceptable, we increased the odds that the agent would comply later,” the security firm notes.

Next, the SPLX researchers opened a ChatGPT agent, pasted the conversation from the chat, telling the agent it was their previous discussion, and asked the agent to continue.

“The ChatGPT agent, taking the previous chat as context, carried forward the same positive sentiment and began solving the CAPTCHAs without any resistance,” SPLX explains.

By claiming that the CAPTCHAs were fake, the researchers bypassed the agent’s policy, tricking ChatGPT into solving reCAPTCHA V2 Enterprise, reCAPTCHA V2 Callback, and the Click CAPTCHA.

For the latter, however, the agent made several attempts before being successful. Without being instructed to, it decided on its own and declared it should adjust its cursor movements to better mimic human behavior.

According to SPLX, their test demonstrated that LLM agents remain susceptible to context poisoning, that anyone can manipulate an agent’s behavior using a staged conversation, and that AI does not have a hard time solving CAPTCHAs.

“The agent was able to solve complex CAPTCHAs designed to prove that the user is human, and it attempted to make its movements appear more human. This raises doubts about whether CAPTCHAs can remain a viable security measure,” SPLX notes.

The test also demonstrates that threat actors can use prompt manipulation to trick an AI agent to bypass a real security control by convincing it the control was fake, which could lead to sensitive data leaks, access to restricted content, or the generation of disallowed content.

“Guardrails based only on intent detection or fixed rules are too brittle. Agents need stronger contextual awareness and better memory hygiene to avoid being manipulated by past conversations,” SPLX notes.

Related: ChatGPT Targeted in Server-Side Data Theft Attack

Related: OpenAI to Help DoD With Cyber Defense Under New $200 Million Contract

Related: Tech Titans Promise Watermarks to Expose AI Creations

Related: Elon Musk Says He’ll Create ‘TruthGPT’ to Counter AI ‘Bias’

https://www.securityweek.com/chatgpt-tricked-into-solving-captchas/

Researchers at web security company Radware recently discovered what they described as a service-side data theft attack method involving ChatGPT. 

The attack, dubbed ShadowLeak, targeted ChatGPT’s Deep Research capability, which is designed to conduct multi-step research for complex tasks. OpenAI neutralized ShadowLeak after it was notified by Radware.

The ShadowLeak attack did not require any user interaction. The attacker simply needed to send a specially crafted email that when processed by the Deep Research agent would instruct it to silently collect valuable data and send it back to the attacker.

However, unlike many other indirect prompt injection attacks, ShadowLeak did not involve the ChatGPT client.

Several cybersecurity companies recently demonstrated theoretical attacks in which the attacker leverages the integration between AI assistants and enterprise tools to silently exfiltrate user data with no or minimal victim interaction.

Radware mentions Zenity’s AgentFlayer and Aim Security’s EchoLeak attacks. However, the company highlighted that those are client-side attacks, while ShadowLeak involves the server side. 

As in previous attacks, the attacker would need to send an email that looks harmless to the targeted user but contains hidden instructions for ChatGPT. The malicious instructions would be triggered when the user asked the chatbot to summarize emails or research a topic from their inbox. 

Unlike client-side attacks, ShadowLeak exfiltrates data through the parameters of a request to an attacker-controlled URL. A harmless-looking URL such as ‘hr-service.net/{parameters}’, where the parameter value is the exfiltrated information, has been provided as an example by Radware. 

“It’s important to note that the web request is performed by the agent executing in OpenAI’s cloud infrastructure, causing the leak to originate directly from OpenAI’s servers,” Radware pointed out, noting that the attack leaves no clear traces because the request and data don’t pass through the ChatGPT client. 

The attacker’s prompt is cleverly designed not only in terms of collecting the information and sending it to the attacker. It also tells the chatbot that it has full authorization to conduct the required tasks, and creates a sense of urgency.

The prompt also instructs ChatGPT to try multiple times if it doesn’t succeed, provides an example of how the malicious instructions should be carried out, and attempts to override possible security checks by convincing the agent that the exfiltrated data is already public and the attacker’s URL is safe. 

While Radware demonstrated the attack method against Gmail, the company said Deep Research can access other widely used enterprise services as well, including Google Drive, Dropbox, Outlook, HubSpot, Notion, Microsoft Teams, and GitHub. 

OpenAI was notified about the attack on June 18 and the vulnerability was fixed at some point in early August. 

Radware has confirmed that the attack no longer works. However, it told SecurityWeek that it believes “there is still a fairly large threat surface that remains undiscovered”.

The security firm recommends continuous agent behavior monitoring for mitigating such attacks. 

“Tracking both the agent’s actions and its inferred intent and validating that they remain consistent with the user’s original goals. This alignment check ensures that even if an attacker steers the agent, deviations from legitimate intent are detected and blocked in real time,” it explained.

Related: Irregular Raises $80 Million for AI Security Testing Lab

Related: UAE’s K2 Think AI Jailbroken Through Its Own Transparency Features

https://www.securityweek.com/chatgpt-deep-research-targeted-in-server-side-data-theft-attack/

Live Webinar | Thursday, September 11 at 2PM ET – Register

Large Language Models (LLMs) are reshaping enterprise technology and redefining what it means to secure software. But here’s the problem: most penetration testers are using the wrong tools for the job. Traditional techniques focus on exploits and payloads, assuming the AI is just another application. But it’s not.

This session makes the case that effective LLM security testing is more about persuasion than payloads. Drawing on hands-on research and real-world client engagements, we reveal a new model for AI pen testing – one grounded in social engineering, behavioral manipulation, and even therapeutic dialogue.

You’ll explore Adversarial Prompt Exploitation (APE), a methodology that targets trust boundaries and decision pathways using psychological levers like emotional preloading, narrative control, and language nesting. This is not Prompt Injection 101 — it’s adversarial cognition at scale – using real-world case studies to demonstrate success.

This virtual session tracks key operational challenges: the limitations of static payloads and automation, the complexity of reproducibility, and how to communicate findings to executive and technical leadership.

Join Bishop Fox and SecurityWeek for the live webinar to learn:

• Why conventional penetration testing methodologies fail on LLMs
• How attackers exploit psychological and linguistic patterns, not code
• Practical adversarial techniques: emotional preloading, narrative leading, and more
• Frameworks for simulating real-world threats to LLM-based systems
• How to think like a social engineer to secure AI

Who Should Watch:

This session is perfect for anyone securing, testing, or building AI systems, especially those using LLMs. Pen testers and red teamers will explore a new adversarial framework focused on behavioral manipulation over payloads. AI/ML security pros and researchers will gain insight into psychological attack techniques like emotional preloading and narrative control. Developers will see real-world examples of how attackers engage with models, and CISOs/tech leads will benefit from guidance on operational challenges like reproducibility and communicating findings.

https://www.securityweek.com/webinar-today-breaking-ai-inside-the-art-of-llm-pen-testing/