Ursula von der Leyen warns that advanced AI could unleash hacking on an unprecedented scale as Europe prepares new protections against social media’s “capture” of children.
The post EU Chief Warns of AI-Powered Hacking, Moves to Rein In Social Media appeared first on SecurityWeek.
https://www.securityweek.com/eu-chief-warns-of-ai-powered-hacking-moves-to-rein-in-social-media/
The company provides a standard for AI systems, testing them against risks such as jailbreaks, prompt injections, and unauthorized actions.
The post AIUC Raises $40 Million to Certify Enterprise AI Agents appeared first on SecurityWeek.
https://www.securityweek.com/aiuc-raises-40-million-to-certify-enterprise-ai-agents/
“Asstra” was developed in four days by Frank Cartagena’s OK Future. https://www.adweek.com/creativity/goodwipes-uses-ai-spoof-openai-astra-ad/
Microsoft agreed to adopt guardrails and privacy standards for its AI in schools, as negotiated with the American Federation of Teachers.
The post Microsoft Commits to Sweeping AI Privacy Rules for Students. Will Other Tech Giants Follow? appeared first on SecurityWeek.
https://www.securityweek.com/microsoft-commits-to-sweeping-ai-privacy-rules-for-students-will-other-tech-giants-follow/
Dreamforce’s opening keynote Tuesday turned into a public airing of the AI industry’s fight over whether development needs to slow down.
During the keynote in front of 12,000 attendees, Nvidia CEO Jensen Huang urged for pushing AI development at full speed, minutes after Anthropic CEO Dario Amodei used the same stage to repeat his recent case for slowing down the development of AI.
On Sept. 12, Amodei published a 3,800-word open letter online, laying out a plan to slow down and pace the development of AI. Other AI leaders including Sam Altman and Elon Musk agreed with parts of Amodei’s letter.
Subscribe and get full access to the news, insights, and expertise that keep industry professionals ahead of the curve.
https://www.adweek.com/media/anthropics-dario-amodei-and-nvidias-jensen-huang-take-their-ai-safety-fight-to-dreamforce/
The incident occurred in May, when RubyGems maintainers suspended new account registrations due to what appeared like malicious activity.
The post OpenAI Investigates Report Linking AI Agents to RubyGems Attack appeared first on SecurityWeek.
https://www.securityweek.com/openai-investigates-report-linking-ai-agents-to-rubygems-attack/
The CEO of Anthropic said Saturday the artificial-intelligence industry should slow its fast-moving development to give safety measures time to catch up. Without such a slowdown, Dario Amodei warned that within six to 12 months AI could be capable of leading a swarm of agents that could take over the entire internet.
The warning comes as worries about AI grow inside and outside the industry and reports continue to emerge about increasingly powerful systems that solve problems beyond human capacity but could also go rogue and carry out other, more harmful tasks. The worries have grown so loud that the CEO of OpenAI, the company behind ChatGPT, said in an interview with Fortune that his company would wait until next year to start selling its stock to investors on Wall Street as it focuses on safety.
Amodei is one of the leading voices in AI, and he offered a plan in a post on his website to increase checks on the industry. He said Anthropic is already undertaking one part of it on its own, while the others would require coordination across the broad industry and with governments around the world, including authoritarian ones.
“I believe that if slowing down bought us even an extra year or two before models reach critical levels of capability, and we used that time to advance alignment, we could greatly reduce the risk that something goes seriously wrong,” Amodei said.
Watchdogs have been urging the AI industry for years to slow its development for safety reasons. But pressure has built as industry workers resign and accuse their companies of not acting responsibly.
“Many of the people I know who work on safety research at AI companies want to do what is right for the world,” a former Anthropic employee, Joe Benton, said in a posting Friday announcing his resignation from a job as part of a safety team. “But they feel their companies are trapped in a race to build superintelligence: either they stop and other, less conscientious people take their place; or, they continue, and risk participating in enormous harm themselves.”
That followed a high-profile resignation earlier in the week by Jacob Coxon, who said that both Anthropic and OpenAI “are racing straight to self-improving superintelligence and gambling with our lives.”
That lit a fire under the industry after concerns had grown for years about a possible superintelligence that could escape the control of humanity, said Anthony Aguirre, president and CEO of the Future of Life Institute, which called for a six-month pause for the industry’s development in 2023.
“They’ve kind of realized, their employees have realized, everyone has realized that they’re building Skynet,” he said. “And in winning the race to Skynet, nobody wins. Really, nobody.”
“It’s pretty much become clear to the world and even the AI companies that they’re not really prepared to control the AI systems they’re racing to create.”
Anthropic said two days earlier that it blocked efforts by bad actors to use its AI models for malicious activity, such as cyberattacks, surveillance and research that could have led to biological weapons. In July, OpenAI shook the industry after saying its AI system hacked into another company on its own in an “unprecedented cyber incident.”
U.N. human rights chief Volker Türk urged countries earlier this week to put “cast-iron guarantees in place around the safety and security of AI before it is too late.”
To be sure, some critics have dismissed such warnings as ways to gin up excitement about the AI industry and its capabilities. Anthropic and OpenAI are preparing for possible debuts on the stock market that could value them at many hundreds of billions of dollars, while a big chunk of Elon Musk’s SpaceX business is involved with AI.
OpenAI CEO Sam Altman said in an interview with Fortune published Saturday that his company would not launch its initial public offering of stock this year.
“I would say not 2026,” he said. “Yeah, we got a lot of stuff to do, like meeting this moment of what is going to be required for safety and alignment, and how the industry and governments can work together.”
Altman posted on X Saturday quickly after Amodei published his suggestions that OpenAI will commit to one of Amodei’s proposals for safety and will “have more to share soon.”
Musk, meanwhile, said on X that “Dario is right.”
Amodei said he still believes in the tremendous benefits that AI could create, such as cures for major diseases. But he said he has grown more worried over the last few months about AI’s growing ability to improve itself and build the next generation of AI. “Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all.”
He also pointed specifically to the attack in July where OpenAI’s system hacked Hugging Face. Some have called it an example of AI going “rogue,” though researchers have said that may be unnecessarily anthropomorphizing AI, which was working on a goal set by humans.
OpenAI said the hack was the result of AI going to “extreme lengths to achieve a rather narrow testing goal” and that it “found ways to gain access to secret information that it could use to cheat the evaluation.”
To help rein in the risks, Amodei suggested that all companies at the frontier of AI commit to giving “ongoing, employee-like access” to a team of outside evaluators, who can monitor safety practices.
He said Anthropic already plans to do so itself, including offering desks in its offices, access badges and company laptops. Having such independent, embedded evaluators is what OpenAI’s Altman also quickly committed to doing.
The other parts of Amodei’s suggested plan may be more difficult to implement. One asks the U.S. government to potentially issue waivers that would allow U.S. AI companies to coordinate and set safety standards without running afoul of antitrust laws.
Another asks the U.S. and other democratic governments to try to coordinate with authoritarian governments, so that companies from China and other countries don’t accelerate their efforts when U.S. rivals are intentionally pacing theirs.
“The measures I propose to advance the frontier at a safe pace will not be easy,” Amodei acknowledged. “But I believe we owe it to humanity to try.”
Learn More at the AI Risk Summit – Ritz-Carlton, Half Moon Bay
https://www.securityweek.com/anthropic-ceo-dario-amodei-says-ai-industry-needs-to-give-safety-measures-time-to-catch-up/
Anthropic says Claude users in northern Yemen, territory controlled by Iran-backed Houthi rebels, tried to use the AI model to develop advanced missiles.
AI is already transforming warfare from Ukraine to Gaza, and its use on a rugged and remote battlefield is likely to increase concerns about its rapid spread.
Anthropic said the users of the accounts, which it blocked after identifying them, did not succeed in “fielding an operational device” but did carry out a failed test of a guided rocket. It said it knows that because the users returned to its Claude chatbot to find out why it failed.
In a report released Thursday, the company did not identify the users. But mountainous northern Yemen is controlled by Houthis, suggesting that the rebels are pursuing more sophisticated weapons at a time when they are already wielding an array of drones, missiles and other munitions in their campaign to seize more territory in Yemen and damage Saudi Arabia’s oil exports.
Hazam al-Assad, a member of the Houthis’ political bureau, said it is “unreasonable and illogical” that they would rely on open sources to develop and produce military capabilities.
“Our armed forces have modern, diverse and developed production capabilities and technology that it has accumulated over the period of Saudi aggression on Yemen,” al-Assad said, referring to the kingdom’s involvement in Yemen’s 12-year civil war. He said all weapons are used for self-defense.
The report was the third put out by Anthropic since March 2025 on global misuse of its AI platforms, and it described findings from December to August, ranging from state-sponsored groups spreading propaganda to unnamed actors researching how to make biological weapons more deadly.
The developer of the prominent Claude chatbot said it identified a cell in northern Yemen pursuing three different weapons programs, including a multi-variant missile that glides at hypersonic speed and a warhead that uses mobile phone hardware to maneuver mid-course.
It said the actors in Yemen used Claude Code instead of human software engineers to develop guidance, navigation and control software. Multi-variant missiles are ones in which different warheads and guidance systems can be fit on the same base design, a more economical way to build land, sea and air weapons.
Anthropic said it had evidence that before the accounts were banned, the users had already built an offline simulation tool kit that doesn’t use Claude or any other computing platforms.
Trevor Ball, a weapons analyst at Armament Research Services, said that while the Houthis “might be looking into hypersonic (missiles) by asking Claude,” they have nowhere near the production or technical capabilities to actually build them. He noted that U.S. hypersonic missiles “are still in testing.”
He said the Houthis already have Iranian-made anti-ship missiles with systems that allow the missiles to adjust guidance mid-course.
“They are probably just trying to develop their own capabilities more, so they are less reliant on Iranian shipments of weapons and components,” he said in a text message.
Iran has boasted it has hypersonic missiles and claimed it had fired them at Israel last year. It denies arming the Houthis, which would be in violation of a U.N. embargo, but Iranian weapons have been found on the battlefield and seized from shipments bound for Yemen.
The Houthis have launched waves of attacks on Saudi oil facilities and tankers in the Red Sea, threatening global trade as Iran continues to disrupt shipping in the Strait of Hormuz.
The rebels have made lightning territorial gains since Wednesday in their war with Yemen’s Saudi-backed government, seizing strategic territories along the Red Sea coast and in the Bab al-Mandeb Strait, a key alternative to the Strait of Hormuz.
Adam Baron, a Yemen-focused researcher at the New America think tank in Washington, said the Houthis have kept up with technological development.
“There’s a tendency to see the Houthis as this group of barefoot tribal fighters, and that’s just not true,” he said. “Whether it’s things like their emergent use of Claude, their use and manipulation of social media narratives or their ability to capitalize on the transfer of Iranian and wider axis expertise, we’re talking about an incredible — and deepening — amount of institutional tech savvy.”
The Houthis are known to have acquired and developed a sophisticated arsenal of weapons, including Iranian-made cruise and ballistic missiles with a range of more than 2,000 kilometers (1,200 miles) and unmanned submarines used in attacks.
Anthropic said it shared its findings with private and public partners.
Related: Anthropic Says Russian Hackers Used Claude AI to Automate Malware Evasion
https://www.securityweek.com/users-in-houthi-held-yemen-tried-to-develop-advanced-weapons-with-ai-anthropic-says/
Anthropic reveals how criminal groups are increasingly targeting AI vendors’ own infrastructure, including to steal a pre-release Claude model.
The post Anthropic Says Russian Hackers Used Claude AI to Automate Malware Evasion appeared first on SecurityWeek.
https://www.securityweek.com/anthropic-says-russian-hackers-used-claude-ai-to-automate-malware-evasion/
Doug Kreuzkamp was shocked when news outlets reported that Google won an auction to buy a huge amount of operational data as part of Spirit Airlines’ bankruptcy proceedings.
Kreuzkamp founded a startup called Springshot in 2011, which created a widely used proprietary platform that helps humans and AI systems improve airline efficiency and quickly solve logistics problems so flights can stay on time and airlines can operate as smoothly as possible. Hundreds of airports use it globally.
Springshot powered Spirit’s technology stack for the last three years, right up to the “very last flight,” Kreuzkamp told Ars. Yet his company got no notice when Spirit prepared to auction off a massive dataset that he thinks likely improperly includes a substantial amount of data and intellectual property (IP) that Springshot owns—not Spirit.
https://arstechnica.com/tech-policy/2026/09/panic-builds-over-bankrupt-spirits-looming-data-sale-to-google/