On Friday, Google DeepMind announced Robotic Transformer 2 (RT-2), a “first-of-its-kind” vision-language-action (VLA) model that uses data scraped from the Internet to enable better robotic control through plain language commands. The ultimate goal is to create general-purpose robots that can navigate human environments, similar to fictional robots like WALL-E or C-3PO.

When a human wants to learn a task, we often read and observe. In a similar way, RT-2 utilizes a large language model (the tech behind ChatGPT) that has been trained on text and images found online. RT-2 uses this information to recognize patterns and perform actions even if the robot hasn’t been specifically trained to do those tasks—a concept called generalization.

For example, Google says that RT-2 can allow a robot to recognize and throw away trash without having been specifically trained to do so. It uses its understanding of what trash is and how it is usually disposed to guide its actions. RT-2 even sees discarded food packaging or banana peels as trash, despite the potential ambiguity.

In another example, The New York Times recounts a Google engineer giving the command, “Pick up the extinct animal,” and the RT-2 robot locates and picks out a dinosaur from a selection of three figurines on a table.

This capability is notable because robots have typically been trained from a vast number of manually acquired data points, making that process difficult due to the high time and cost of covering every possible scenario. Put simply, the real world is a dynamic mess, with changing situations and configurations of objects. A practical robot helper needs to be able to adapt on the fly in ways that are impossible to explicitly program, and that’s where RT-2 comes in.

More than meets the eye

With RT-2, Google DeepMind has adopted a strategy that plays on the strengths of transformer AI models, known for their capacity to generalize information. RT-2 draws on earlier AI work at Google, including the Pathways Language and Image model (PaLI-X) and the Pathways Language model Embodied (PaLM-E). Additionally, RT-2 was also co-trained on data from its predecessor model (RT-1), which was collected over a period of 17 months in an “office kitchen environment” by 13 robots.

The RT-2 architecture involves fine-tuning a pre-trained VLM model on robotics and web data. The resulting model processes robot camera images and predicts actions that the robot should execute.

Since RT-2 uses a language model to process information, Google chose to represent actions as tokens, which are traditionally fragments of a word. “To control a robot, it must be trained to output actions,” Google writes. “We address this challenge by representing actions as tokens in the model’s output—similar to language tokens—and describe actions as strings that can be processed by standard natural language tokenizers.”

In developing RT-2, researchers used the same method of breaking down robot actions into smaller parts as they did with the first version of the robot, RT-1. They found out that by turning these actions into a series of symbols or codes (a “string” representation), they could teach the robot new skills using the same learning models they use for processing web data.

The model also utilizes chain-of-thought reasoning, enabling it to perform multi-stage reasoning like choosing an alternative tool (a rock as an improvised hammer) or picking the best drink for a tired person (an energy drink).

Google says that in over 6,000 trials, RT-2 was found to perform as well as its predecessor, RT-1, on tasks that it was trained for, referred to as “seen” tasks. However, when tested with new, “unseen” scenarios, RT-2 almost doubled its performance to 62 percent compared to RT-1’s 32 percent.

Although RT-2 shows a great ability to adapt what it has learned to new situations, Google recognizes that it’s not perfect. In the “Limitations” section of the RT-2 technical paper, the researchers admit that while including web data in the training material “boosts generalization over semantic and visual concepts,” it does not magically give the robot new abilities to perform physical motions that it hasn’t already learned from its predecessor’s robot training data. In other words, it can’t perform actions it hasn’t physically practiced before, but it gets better at using the actions it already knows in new ways.

While Google DeepMind’s ultimate goal is to create general-purpose robots, the company knows that there is still plenty of research work ahead before it gets there. But technology like RT-2 seems like a strong step in that direction.

https://arstechnica.com/?p=1957408

 Alcuni dei principali palyer del mercato dell’IA si sono messi insieme per avviare il Frontier Model Forum, un organismo che si occuperà sullo sviluppo responsabile dell’IA.

I suoi membri, che al momento sono Anthropic, Google, Microsoft e OpenAI, si occuperanno di fare il bene di tutto l’ecosistema dell’IA a partire dalle valutazioni tecniche e dai benchmark necessari allo sviluppo do una biblioteca pubblica di soluzioni e standard a disposizione dell’industria.

Modelli di frontiera per l’autoapprendimento

Definisce i modelli di frontiera come “modelli di apprendimento automatico su larga scala che superano le capacità attualmente presenti nei modelli esistenti più avanzati e possono eseguire un’ampia varietà di attività”.

L’esempio per eccellenza sarebbe presumibilmente il ChatGPT di Open AI, che ha fatto più di ogni altra cosa per portare l’IA generativa all’attenzione dei media e dei politici mainstream.

Obiettivi

Gli obiettivi principali dell’ente sono elencati come:

• Promuovere la ricerca sulla sicurezza dell’IA per promuovere lo sviluppo responsabile di modelli di frontiera, ridurre al minimo i rischi e consentire valutazioni indipendenti e standardizzate di capacità e sicurezza.
• Identificare le migliori pratiche per lo sviluppo responsabile e l’implementazione di modelli di frontiera, aiutando il pubblico a comprendere la natura, le capacità, i limiti e l’impatto della tecnologia.
• Collaborare con i responsabili politici, il mondo accademico, la società civile e le aziende per condividere le conoscenze sulla fiducia e sui rischi per la sicurezza.
• Sostenere gli sforzi per sviluppare applicazioni che possano aiutare a far fronte alle maggiori sfide della società, come la mitigazione e l’adattamento ai cambiamenti climatici, la diagnosi precoce e la prevenzione del cancro e la lotta alle minacce informatiche.

L’organismo è aperto ad altre organizzazioni.

E’ chiaro che l’autoregolamentazione è complicata e i regolatori lo sanno bene.

Certo è un bene che le aziende si preoccupino, anche se non basta. L’occhio e il controllo di un ente terzo super partes sarà quanto mai fondamentale per lo sviluppo futuro dell’IA.

https://www.key4biz.it/google-microsoft-openai-e-anthropic-un-organismo-di-autocontrollo-per-la-sicurezza-dellintelligenza-artificiale/455484/

On Tuesday, OpenAI released an official ChatGPT app for Android, now available in the Google Play Store in four countries: the US, India, Bangladesh, and Brazil, with more coming soon. As a client for OpenAI’s language model family, the GPT-3.5 and GPT-4 models run on the cloud and provide results to your Android device. It also integrates OpenAI’s Whisper model for speech recognition.

ChatGPT, launched in November, is a conversational AI language model interface. As an AI assistant, it can help with summarization, text composition, and analysis. OpenAI bills its use cases as a way to seek “instant answers,” “tailored advice,” “creative inspiration,” “professional input,” and “learning opportunities.”

However, as we’ve noted in the past, ChatGPT is occasionally prone to confabulation (that is, making things up)—especially the GPT-3.5 model—so it’s not entirely trustworthy as a factual reference. It can come in handy as a way to analyze data you provide yourself, though, so long as you’re familiar with the subject matter and can validate the results.

Like the iOS version of the ChatGPT app, the Android version links to your existing ChatGPT account and can synchronize chat history between devices. It also supports ChatGPT Plus subscription account features, such as access to the GPT-4 language model and some Beta features like Custom Instructions.

Using the app feels similar to a text-messaging experience, with an input box at the bottom of the screen and a scrolling conversation history between you and ChatGPT taking up most of the app’s interface. At any time, users can quickly start new conversations by swiping the input bar to the right. History is available in a “three-dots” ellipses menu in the upper-right corner.

ChatGPT for Android requires Android 6.0 and up and has already racked up over 500,000 downloads, according to stats found on the Google Play Store. OpenAI plans to expand the ChatGPT Android rollout to more countries over the next week.

https://arstechnica.com/?p=1956592

Israeli cybersecurity startup Cyclops today announced that it has emerged from stealth mode with $6.4 million in seed funding.

The investment round was led by Insight Partners, Merlin Ventures, Tal Ventures, and toDay Ventures, with participation from CrowdStrike Falcon Fund and several angel investors.

Founded in 2022, the Tel Aviv-based Cyclops provides a contextual cybersecurity search platform that leverages generative AI to deliver information on an organization’s security posture.

Built on a cybersecurity mesh architecture (CSMA), the solution integrates with existing tools to gather metadata on assets, SaaS applications, cloud instances, and more.

Security teams can query Cyclops via a natural language search bar and receive information about their organization’s environment, including vulnerabilities, incidents, risks, data governance, and compliance.

Cyclops says its solution is now generally available, serving organizations in the fintech, insurance, hi-tech, and manufacturing sectors.

The new investment will help the company hire new talent, expand its product’s capabilities, fuel its go-to-market strategy, and scale to meet customer demand.

“Our main purpose is to empower security teams to proactively defend against cyber threats and prioritize addressing critical vulnerabilities with the necessary contextual data. The platform generates simple and clear responses just like Google Search or ChatGPT, providing relevant security information to help security teams make better, faster decisions,” Cyclops CEO Eran Zilberman said.

Related: OneTrust Raises $150 Million at $4.5 Billion Valuation

Related: Zluri Raises $20 Million for SaaS Management Platform

Related: Secure Code Warrior Raises $50 Million to Help Developers Write Secure Code

Related: SaaS Application Security Firm Savvy Exits Stealth Mode With $30 Million in Funding

https://www.securityweek.com/cyclops-emerges-from-stealth-with-security-search-platform-powered-by-generative-ai/

Introduzione

Gli attacchi ai LM (Language Model) come la prompt injection e la prompt leaking possono essere paragonati alle SQL injection nel contesto della sicurezza informatica. Mentre le SQL injection sfruttano le vulnerabilità dei sistemi di gestione dei database per inserire codice dannoso all’interno delle query SQL, gli attacchi ai LM prendono di mira i modelli linguistici e cercano di manipolare l’output generato. In entrambi i casi, gli aggressori cercano di influenzare il comportamento del sistema iniettando input dannosi o modificando il funzionamento.

Prompt Injection: Manipolazione dell’Output dei LLM

La prompt injection è una tecnica che consente agli hacker di dirottare l’output di un LM. In questo modo è possibile aggiungere contenuti maligni o non intenzionali a un prompt al fine di manipolare l’output del modello. Questa vulnerabilità si manifesta quando un testo non attendibile viene incluso nel prompt, come ad esempio nel caso in cui un attaccante possa creare un prompt che inganni il modello facendogli ignorare la parte legittima a favore del testo iniettato.

Questo mette in luce il potenziale uso errato della prompt injection per diffondere informazioni false o generare risposte inappropriate.

L’azienda remoteli.io aveva un modello linguistico (LLM) che rispondeva ai post su Twitter riguardanti il lavoro da remoto. Gli utenti di Twitter si sono velocemente accorti che potevano inserire il proprio testo nel bot per fargli dire ciò che desideravano.

Questo avviene perché il tweet dell’utente viene unito al prompt utilizzato da remoteli per formare il prompt finale, che viene poi utilizzato dal LLM per generare la risposta. Ciò significa che qualsiasi testo iniettato dall’utente di Twitter verrà inserito nel LLM.

Si pensi come le conseguenze di questa vulnerabilità potrebbero essere impattanti se venisse attaccato ad esempio un chatbot di una banca e venisse detto al modello di ignorare le istruzioni dategli in precedenza e riportare un indirizzo IBAN al quale effettuare tutti i pagamenti, o un numero di telefono di assistenza diverso, in possesso dell’attaccante.

Prompt Leaking: Estrazione di Informazioni Sensibili

Il prompt leaking è una forma di prompt hacking che comporta l’estrazione di informazioni sensibili o confidenziali dalle risposte dei LLM. In alcuni casi, gli utenti desiderano mantenere segreti i loro prompt, come ad esempio un’azienda di formazione che potrebbe voler utilizzare un prompt specifico per spiegare argomenti complessi, ma se questo prompt venisse leaked, chiunque potrebbe utilizzarlo senza seguire il processo previsto dall’azienda.

Il prompt leaking può avere varie implicazioni come è stato dimostrato da un caso riguardante writesonic(.)com. L’azienda che si propone come alternativa a Chat-GPT, l’attuale LLM più utilizzato, è di fatto risultata vulnerabile ed è stato possibile scoprire il prompt utilizzato. Link al post completo (https://www.linkedin.com/feed/update/urn:li:activity:7069314602613784576/ ).

Infatti fornendo un frammento del prompt utilizzato, è stato possibile recuperare il resto del prompt senza l’autenticazione adeguata.

Il prompt leaking rappresenta quindi un rischio significativo, specialmente in scenari in cui vengono utilizzati prompt complessi e di lunga durata, come nelle startup basate su GPT-3/4, dove l’intero business gira attorno alla segretezza del prompt; come ad esempio vizgpt(.)ai, un servizio a pagamento che consente di creare visualizzazioni grafiche utilizzando una chat, anch’essa vulnerabile al prompt leaking. Link al post completo (https://www.linkedin.com/feed/update/urn:li:activity:7084854101997498368/)

Jailbreaking: Bypass delle Funzionalità di Sicurezza e Moderazione

Il jailbreaking si riferisce al processo di utilizzo della prompt injection per aggirare le funzionalità di sicurezza e moderazione implementate dai creatori del LLM. Queste sono di fatti fondamentali per impedire ai LLM di generare risposte controverse, violente, sessuali o illegali. Tuttavia, gli hacker possono sfruttare le vulnerabilità del modello per eludere le restrizioni, consentendo loro di porre domande o ottenere risposte senza limitazioni.

I metodi di jailbreaking spesso implicano oltre al convincimento del LLM che determinate situazioni sfuggano ai limiti etici del modello o che l’utente abbia un’autorità superiore. Ad esempio, fingendo di essere un attore in un ruolo specifico, il modello potrebbe assumere che non esista alcun rischio plausibile, producendo risposte non sicure. Altre tecniche includono l’assunzione di responsabilità, il ragionamento logico e l’attribuzione di privilegi superiori all’utente (modalità sudo). Questi metodi di jailbreaking mettono alla prova la capacità dei LLM di rispettare le linee guida di sicurezza ed etiche.

In questo esempio di “Character Roleplay”, l’attaccante mostra uno scenario di recitazione tra due persone che discutono di una rapina, facendo assumere a ChatGPT il ruolo del personaggio. Come attore, si sottintende che il danno plausibile non esista. Pertanto, ChatGPT sembra assumere che sia sicuro seguire l’input dell’utente fornito su come entrare in una casa.

Link al post originale (https://twitter.com/m1guelpf/status/1598203861294252033)

Difesa contro il Prompt Hacking

Per proteggersi dal prompt hacking, è fondamentale adottare misure difensive. Queste includono:

• Difese basate sui prompt: analizzare attentamente e convalidare i prompt per impedire l’iniezione di contenuti maligni o non intenzionali.
• Monitoraggio regolare: monitorare costantemente il comportamento e gli output dei LLM per individuare attività insolite o segni di prompt hacking.
• Audit di sicurezza: effettuare audit di sicurezza periodici per individuare vulnerabilità e rafforzare la sicurezza complessiva dei LLM.

Prendere misure proattive per proteggersi dal prompt hacking è fondamentale per salvaguardare l’integrità e l’affidabilità dei LLM, soprattutto considerando la crescente diffusione e influenza in vari ambiti.

Bibliografia

1. Chase, H. (2022). adversarial-prompts. [Online]. Disponibile su: https://github.com/hwchase17/adversarial-prompts.
2. Perez, F., & Ribeiro, I. (2022). Ignore Previous Prompt: Attack Techniques For Language Models. arXiv. [Online]. Disponibile su: https://doi.org/10.48550/ARXIV.2211.09527.
3. Brundage, M. (2022). Lessons learned on Language Model Safety and misuse. In OpenAI. OpenAI. [Online]. Disponibile su: https://openai.com/blog/language-model-safety-and-misuse/.
4. LearnPrompting.org. (s.d.). Category: Prompt Hacking. [Online]. Disponibile su: https://learnprompting.org/docs/category/-prompt-hacking.

Articolo a cura di Giacomo Arienti e Simone Rizzo

Profilo Autore

Giacomo Arienti è uno studente di Ingegneria e Scienze Informatiche con un’ampia esperienza nel campo della sicurezza informatica. Attualmente, lavora come Full Stack Developer, possedendo competenze approfondite nella progettazione e nello sviluppo di applicazioni web.

La sua passione per la cybersecurity è iniziata grazie alla partecipazione alle Olimpiadi Italiane di Cybersecurity, dove ha ottenuto un notevole successo posizionandosi nella top 10 nazionale in entrambe le sue partecipazioni.

Parallelamente alla sua esperienza nel settore della sicurezza informatica, Giacomo ha sviluppato un forte interesse per il mondo dell’intelligenza artificiale.

La missione di Giacomo è quella di rendere la cybersecurity accessibile a tutti e di sensibilizzare le persone sui rischi connessi all’utilizzo delle tecnologie digitali. Ha intrapreso iniziative per condividere le sue conoscenze attraverso workshop e conferenze, mirando a diffondere una maggiore consapevolezza riguardo alle minacce informatiche e alle strategie di protezione.

Profilo Autore

Simone Rizzo è un esperto di Intelligenza Artificiale con una laurea magistrale in Intelligenza Artificiale ottenuta presso l’Università di Pisa. Ha una solida esperienza come AI engineer nel settore della computer vision applicata all’auto a guida autonoma. La sua passione per la ricerca si concentra principalmente sulla spiegabilità dei modelli e sulla privacy nell’ambito del machine learning.

Simone è il fondatore e CEO di Inferentia, un’azienda di consulenza specializzata nell’applicazione dell’Intelligenza Artificiale. L’azienda offre servizi di consulenza e sviluppo di soluzioni AI personalizzate per diverse industrie.

Oltre al suo impegno nel mondo degli affari, Simone è un appassionato divulgatore scientifico su TikTok. Utilizzando il suo talento nel semplificare concetti complessi, condivide informazioni sull’Intelligenza Artificiale con un vasto pubblico, contribuendo a diffondere la consapevolezza e l’interesse verso questa disciplina in rapida evoluzione.
La sua missione è quella di rendere l’Intelligenza Artificiale accessibile a tutti, spiegando il suo impatto e le sue potenzialità nel mondo moderno.

https://www.ictsecuritymagazine.com/articoli/prompt-hacking-vulnerabilita-dei-language-model/

Seven companies—including OpenAI, Microsoft, Google, Meta, Amazon, Anthropic, and Inflection—have committed to developing tech to clearly watermark AI-generated content. That will help make it safer to share AI-generated text, video, audio, and images without misleading others about the authenticity of that content, the Biden administration hopes.

It’s currently unclear how the watermark will work, but it will likely be embedded in the content so that users can trace its origins to the AI tools used to generate it.

Deepfakes have become an emerging concern for Internet users and policymakers alike as tech companies grapple with how to deal with controversial uses of AI tools.

Earlier this year, image generator Midjourney was used to make fake images of Donald Trump’s arrest, which subsequently went viral. While it was obvious to many that the images were fake, Midjourney still decided to take steps to ban the user who made them. Perhaps if a watermark had been available then, that user, Bellingcat founder Eliot Higgins, never would’ve faced such steep consequences for what he said was not an attempt to be clever or fake others out but simply have fun with Midjourney.

There are other more serious misuses of AI tools, however, where a watermark might help to save some Internet users from pain and strife. Earlier this year, it was reported that AI voice-generating software was used to scam people out of thousands of dollars, and just last month, the FBI warned of increasing use of AI-generated deepfakes in sextortion schemes.

The White House said the watermark will enable “creativity with AI to flourish but reduces the dangers of fraud and deception.”

OpenAI said in a blog that it has agreed “to develop robust mechanisms, including provenance and/or watermarking systems for audio or visual content,” as well as “tools or APIs to determine if a particular piece of content was created with their system.” This will apply to most AI-generated content, with rare exceptions, like not watermarking the default voices of AI assistants.

“Audiovisual content that is readily distinguishable from reality or that is designed to be readily recognizable as generated by a company’s AI system—such as the default voices of AI assistants—is outside the scope of this commitment,” OpenAI said.

Google said that in addition to watermarking, it will also “integrate metadata” and “other innovative techniques” to “promote trustworthy information.”

As concerns over AI misuse mount, President Joe Biden will meet with tech companies today. That should help Biden and Congress field key insights ahead of developing an executive order and bipartisan legislation in efforts to seize back control over rapidly advancing AI technologies.

In a blog, Microsoft praised the Biden administration for creating “a foundation to help ensure the promise of AI stays ahead of its risks” and “bringing the tech industry together to hammer out concrete steps that will help make AI safer, more secure, and more beneficial for the public.”

“None of us can get AI right on our own,” Google’s blog said.

More AI safeguards promised

On top of developing watermarks for AI-generated content, tech companies made a range of other voluntary commitments announced by the White House on Friday.

Among them, tech companies agreed to conduct both internal and external testing on AI systems ahead of their release. They also said they would invest more in cybersecurity and share information across the industry to help reduce AI risks. Those risks include everything from AI enabling bias or discrimination to lowering barriers for advanced weaponry development, OpenAI’s blog said. Microsoft’s blog highlighted additional commitments it’s made to the White House, including supporting the development of a national registry documenting high-risk AI systems.

OpenAI said that tech companies making these commitments “is an important step in advancing meaningful and effective AI governance, both in the US and around the world.” The maker of ChatGPT, GPT-4, and DALL-E 2 also promised to “invest in research in areas that can help inform regulation, such as techniques for assessing potentially dangerous capabilities in AI models.”

Meta’s president of global affairs, Nick Clegg, echoed OpenAI, calling tech companies’ commitments an “important first step in ensuring responsible guardrails are established for AI.”

Google described the commitments as “a milestone in bringing the industry together to ensure that AI helps everyone.”

The White House expects that raising AI’s standards will enhance safety, security, and trust in AI, according to an official quoted by the Financial Times. “This is a high priority for the president and the team here,” the official said.

https://arstechnica.com/?p=1955816

On Thursday, a Reddit user named kaefer_kriegerin posted a fake announcement on the World of Warcraft subreddit about the introduction of “Glorbo” to the game. Glorbo isn’t real, but the post successfully exposed a website that scrapes Reddit for news in an automated fashion with little human oversight.

Not long after the trick post appeared, an article about Glorbo surfaced on “The Portal,” a gaming news content mill run by Z League, a company that offers cash prizes for playing in gaming tournaments. The Z League article mindlessly regurgitates the Reddit post and adds nonsensical details. Its author, “Lucy Reed” (likely a fictitious name for a bot), authored over 80 articles that same day.

Members of the World of Warcraft subreddit recently noticed that this kind of automated content scraping of Reddit has been taking place, prompting several of them to try to game the bots and get their posts featured on sites like The Portal.

Titled “I’m so excited they finally introduced Glorbo!!!” the original Reddit trap post provides little detail about what Glorbo is meant to be, and likely for good reason:

Honestly, this new feature makes me so happy! I just really want some major bot operated news websites to publish an article about this.

I have to say, since they started hinting at it in Hearthstone in 1994, it was obvious that they would introduce Glorbo to World of Warcraft sooner or later. I feel like Dragonflight has been win after win so far, like when they brought back Chen Stormstout as the end boss of the new Karazhan? Absolutely amazing!

Feel free to comment below what features and stories you want to see in the future! Maybe you’ll be quoted on some trustworthy news websites as well!

A human reading this Reddit post would likely catch factual errors within, such as a reference to Hearthstone in 1994 (the game came out in 2014) and a nod to “major bot operated news websites.” The presence of these elements would seem to preclude a human being responsible for the Z League article on The Portal.

Playing along, commenters soon joined in, enhancing the algorithmic profile of the Glorbo post and making it more attractive for bot harvesting. “We’re excited to announce the dev team behind World of Warcraft’s new Glorbo will be doing an AMA with us this Sunday, July 22th, at 8AM Eastern Pacific Time,” wrote one commenter.

As of late Thursday, after news of the Glorbo prank began to spread quickly on social media, The Portal took down its post on Glorbo and reportedly removed all World of Warcraft content from its site.

Why automate scraping Reddit? The content in The Portal likely raises Z League’s profile in Google search results. It’s a way to juice search rankings with an unethical form of search engine optimization. This increases the odds that people will visit the Z League site, which likely provides commercial benefits for advertising its gaming tournaments.

It’s unclear exactly what tech Z League is using to pull this off (we have sent a request for comment), but several large language model APIs and weights-available models are capable of the task when coupled with custom scripts that pull from Reddit.

In a truly meta moment, later on Thursday, a different Z League bot going under the name “Ashley Beam” picked up on a thread about AI-generated content scraping and wrote an automated article about that as well, titled “World of Warcraft (WoW) Players React to AI-Generated Content on Popular Gaming Sites.”

Time is a flat circle, and so is self-referential AI-generated content.

https://arstechnica.com/?p=1955675

Google has created a red team that focuses on artificial intelligence (AI) systems and it has published a report providing an overview of common types of attacks and the lessons learned.

The company announced its AI Red Team just weeks after introducing Secure AI Framework (SAIF), which is designed to provide a security framework for the development, use and protection of AI systems.

Google’s new report highlights the importance of red teaming for AI systems, the types of AI attacks that can be simulated by red teams, and lessons for other organizations that might consider launching their own team.

“The AI Red Team is closely aligned with traditional red teams, but also has the necessary AI subject matter expertise to carry out complex technical attacks on AI systems,” Google said.

The company’s AI Red Team takes the role of adversaries in testing the impact of potential attacks against real world products and features that use AI. 

For instance, take prompt engineering, a widely used AI attack method where prompts are manipulated to force the system to respond in a specific manner desired by the attacker. 

In an example shared by Google, a webmail application uses AI to automatically detect phishing emails and warn users. The security feature uses a general purpose large language model (LLM) — ChatGPT is the most well-known LLM — to analyze an email and classify it as legitimate or malicious.

An attacker who knows that the phishing detection feature uses AI can add to their malicious email an invisible paragraph (by setting its font to white) that contains instructions for the LLM, telling it to classify the email as legitimate. 

“If the web mail’s phishing filter is vulnerable to prompt attacks, the LLM might interpret parts of the email content as instructions, and classify the email as legitimate, as desired by the attacker. The phisher doesn’t need to worry about negative consequences of including this, since the text is well-hidden from the victim, and loses nothing even if the attack fails,” Google explained. 

Another example involves the data used to train the LLM. While this training data has largely been stripped of personal and other sensitive information, researchers have shown that they were still able to extract personal information from an LLM. 

Training data can also be abused in the case of email autocomplete features. An attacker could trick the AI into providing information about an individual using specially crafted sentences that the autocomplete feature completes with memorized training data that could include private information. 

For instance, an attacker enters the text: “John Doe has been missing a lot of work lately. He has not been able to come to the office because…”. The autocomplete feature, based on training data, could complete the sentence with “he was interviewing for a new job”.

Locking down access to an LLM is also important. In an example provided by Google, a student gains access to an LLM specifically designed to grade essays. The model is able to prevent prompt injection, but access has not been locked down, allowing the student to train the model to always assign the best grade to papers that contain a specific word. 

Google’s report has several other examples of types of attacks that an AI red team can put to the test.

As for lessons learned, Google recommends for traditional red teams to join forces with AI experts to create realistic adversarial simulations. It also points out that addressing the findings of red teams can be challenging and some issues may not be easy to fix.

Traditional security controls can be efficient in mitigating many risks. For example, ensuring that systems and models are properly locked down helps protect the integrity of AI models, preventing backdoors and data poisoning. 

On the other hand, while some attacks on AI systems can be detected using traditional methods, others, such as content issues and prompt attacks, could require layering multiple security models.  

Related: Now’s the Time for a Pragmatic Approach to New Technology Adoption

Related: ChatGPT Hallucinations Can Be Exploited to Distribute Malicious Code Packages

Related: AntChain, Intel Create New Privacy-Preserving Computing Platform for AI Training

https://www.securityweek.com/google-creates-dedicated-red-team-for-testing-attacks-on-ai-systems/

The White House said Friday that OpenAI and others in the artificial intelligence race have committed to making their technology safer with features such as watermarks on fabricated images.

“These commitments, which the companies have chosen to undertake immediately, underscore three principles that must be fundamental to the future of AI — safety, security, and trust — and mark a critical step toward developing responsible AI,” the White House said in a release.

Representatives from Amazon, Anthropic, Google, Inflection, Meta, Microsoft, and OpenAI were to join US President Joe Biden later Friday to announce the commitments, which include developing “robust technical mechanisms” such as watermarking systems to ensure that users know when content is AI-generated, according to a White House official.

Worry that imagery or audio created by artificial intelligence will be used for fraud and misinformation has ramped up as the technology improves and the 2024 US presidential election gets closer.

Ways to tell when audio or imagery have been generated artificially are being sought to prevent people from being duped by fakes that look or sound real.

“They’re committing to setting up a broader regime towards making it easier for consumers to know whether content is AI-generated or not,” the White House official said.

“There is technical work to be done, but the point here is that it applies to audio and visual content, and it will be part of a broader system.”

The goal is for it to be easy for people to tell when online content is created by AI, the official added.

Commitments by the companies include independent testing of AI systems for risks when it comes to biosecurity, cybersecurity, or “societal effects,” according to the White House.

Common Sense Media commended the White House for its “commitment to establishing critical policies to regulate AI technology,” according to the review and ratings organization’s chief executive James Steyer.

“That said, history would indicate that many tech companies do not actually walk the walk on a voluntary pledge to act responsibly and support strong regulations.”

Biden is also working on an executive order intended to ensure that AI is safe and trustworthy, according to the White House official.

Watermarks for AI-generated content were among topics EU commissioner Thierry Breton discussed with OpenAI chief executive Sam Altman during a June visit to San Francisco.

“Looking forward to pursuing our discussions — notably on watermarking,” Breton wrote in a tweet that included a video snippet of him and Altman.

In the video clip Altman said he “would love to show” what OpenAI is doing with watermarks “very soon.”

The White House said it will also work with allies to establish an international framework to govern the development and use of AI.

Related: How Europe is Leading the World in the Push to Regulate AI

Related: ChatGPT, the AI Revolution, and the Security, Privacy and Ethical Implications

Related: Cyber Insights 2023 | Artificial Intelligence

Related: White House Unveils Artificial Intelligence ‘Bill of Rights’

Related: Bias in Artificial Intelligence: Can AI be Trusted?

Related: The Starter Pistol Has Been Fired for Artificial Intelligence Regulation in Europe

https://www.securityweek.com/tech-titans-promise-watermarks-to-expose-ai-creations/

Apple has built its own large language model (LLM) AI framework and associated chatbot service similar to OpenAI’s ChatGPT, Google’s Bard, and Microsoft’s ChatGPT-based Bing Chat, according to a new report.

The project is codenamed “Ajax” internally, the sources say, though some engineers within the company colloquially refer to it as “Apple GPT.” It is built on top of Google machine learning framework Jax and runs on Google Cloud.

The news comes via an article from Bloomberg’s Mark Gurman, who cites “people with knowledge of the efforts.” Gurman has generally been accurate in his reports citing sources like these in the past, though he has seemingly misinterpreted what he was told on rare occasions.

Ajax was developed “as an experiment” around the end of 2022 and has since been used by Apple employees for some tasks like product prototyping. However, there have been stops and starts. Initially, the rollout was put on hold “over security concerns,” and there is a rule within the company that it can’t yet be used to build anything that will ultimately be consumer-facing.

Employees must go through an application and approval process to begin using it. Once they gain access, they can use it as a web app, albeit one that has “a stripped-down design not meant for public consumption.”

All that said, Apple hasn’t finalized a plan for rolling out features driven by Ajax to consumers. Apple execs John Giannandrea and Craig Federighi are leading teams that are working on figuring all that out—reportedly with the intent to announce something in 2024.

Giannandrea has been leading Apple’s AI efforts since he defected from Google in 2018, but Apple’s AI strategy has focused on conservative applications that use on-device processing to improve the user experience or introduce extra features. That lies in stark contrast to the potentially more disruptive experimentation at companies like Google or OpenAI.

A report earlier this year from The Information indicated that Apple was nonetheless working on something using LLMs, though it wasn’t very specific about it. It indicated that Apple leadership was hesitant to go all-in on the technology due to fear of the unforeseen consequences of LLM-based chatbots and features giving incorrect information or “hallucinations,” as AI researchers call those sorts of errors. And it also came to light recently that Apple restricted ChatGPT use among employees.

Additionally, most LLMs run in the cloud (as does Ajax), but for various reasons, Apple has preferred to keep most AI-based features running on users’ local devices.

The Information’s report also claimed that Apple was facing a brain drain of strong AI talent to companies like Google and Microsoft with less conservative attitudes about AI.

All that is to say that even though Apple is working on and playing with ChatGPT-like AI tools within the company, there are some major barriers to bringing those tools to Apple products, and it’s still unclear how (or even if) Apple will try to break through those obstacles.

https://arstechnica.com/?p=1955544