With generative AI exploding everywhere, Google will start requiring that Android apps include a way to report offensive AI-generated content and follow new moderation rules around what kind of AI content is acceptable.

Apps using AI-generated content will be required to add a button to flag or report offensive material early next year to remain in Google’s Play Store.

Users should be able to report without needing to navigate away from the app, similar to already existing in-app reporting systems today. 

The company said its AI-generated content policy covers AI chatbots, AI-generated image apps, and apps that create voice or video content of real people with the use of AI. Apps that host AI-generated content, those that only use AI to summarize materials like books, and productivity apps that use AI as a feature will not be subject to the new policy.

For Google, examples of problematic AI content include nonconsensual deepfakes of sexual material, recordings of real people designed for scamming, false or deceptive election content, apps with generative AI “primarily intended to be sexually gratifying,” and malicious code creation. 

In its announcement, Google acknowledged that generative AI “is a fast-evolving app category,” giving the impression it may revisit its AI policies as the technology continues to evolve.

In addition to new rules around AI-generated content, Google is also beefing up the Play Store’s photo and video permissions policy by limiting how extensively apps can access data. 

“Photos and videos on a user’s device are considered personal and sensitive user data and should be treated with utmost privacy best practices,” the company said. “This sensitive information leaves users vulnerable to leaks or targets of exploitation, so minimizing this access helps avoid the burden on developers of handling such sensitive data.”

Only apps that need broad access to photos and videos will continue getting general permissions, while apps with a limited use of media files will be required to use a photo picker.