Sora 2 debuted on Sept. 30 as an invite-only app, and isn’t available to the public. It had millions of downloads in its first five days, surpassing the download rate of OpenAI’s flagship product, ChatGPT, according to Bill Peebles, OpenAI’s head of Sora.
Sora 2’s feed features not only antisemitic tropes but also copyrighted characters such as SpongeBob SquarePants in a Nazi uniform, depictions of deceased public figures including Queen Elizabeth and Princess Diana, and graphic scenes of violence and racism, The Guardian previously reported.
The controversy comes amid broader concerns and debate about how effectively OpenAI enforces its guardrails across products. On Oct. 14, CEO Sam Altman said in a post on X that erotic content will be permitted in ChatGPT starting in December, a shift that could further test the company’s content moderation systems as it pivots to more commercial use-cases. On Oct. 16, OpenAI said it paused Sora’s ability to generate videos depicting the late civil rights leader Martin Luther King Jr. after some Sora users generated “disrespectful depictions” of his image.
Minda Smiley, senior analyst at Emarketer, noted that AI-generated videos can make harmful stereotypes more potent because of their realism. Many users cannot easily discern what is real, and even those who recognize AI generation may still share content that aligns with preexisting beliefs, she said.
The videos appear to violate OpenAI’s global usage policies, which prohibit content promoting “threats, intimidation, harassment, or defamation.” The company said it employs “layered defenses” to keep its feed safe, including automated tools that check prompts and video outputs for policy violations.
“We’ve red teamed to explore novel risks, and we’ve tightened policies relative to image generation given Sora’s greater realism and the addition of motion and audio,” OpenAI said in a blog post.
Emarketer’s Smiley said users have already found ways around Sora’s guardrails, and without industry-wide standards, platforms remain largely on their own when it comes to moderation.
