Ahmed said OpenAI’s assurances rang hollow. “It’s not credible for them to say they’ve red-teamed this if they didn’t predict that people would use it to produce racist content,” he said. “It’s staggering that the world’s cleverest engineers can’t engineer something to stop antisemitism from being produced by their services.”

This is not the first time OpenAI has come under scrutiny for its safety controls. Earlier this year, the company faced backlash after reports that its ChatGPT chatbot provided a suicidal teenager with information about methods of self-harm—an episode that raised broader concerns about how effectively OpenAI enforces its guardrails across products.

“Ultimately content moderation can become quite subjective when the platform itself is deciding what should be moderated,” Emarketer’s Smiley said.