AI for Business

OpenAI Adds Image Classifier and ChatGPT Warnings to Fight Child Sexual Abuse Material

OpenAI announced new tools aimed at combating child sexual abuse material, or CSAM, as concerns grow over how AI tools are used. Among the additions are an image classifier that detects CSAM,...

Share:

OpenAI announced new tools aimed at combating child sexual abuse material, or CSAM, as concerns grow over how AI tools are used. Among the additions are an image classifier that detects CSAM, including AI-generated images, along with warning screens and temporary account blocks in ChatGPT.

The classifier, developed with the National Center for Missing and Exploited Children, is being offered at no cost through OpenAI's API to select companies. It looks for visual signs of abuse in images it has not seen before and assigns a score to guide reviewers. In early partner testing, it flagged 48% fewer images while finding the same number of abusive ones, and flagged images were twice as likely to be abusive. OpenAI is working with Tech Coalition and ROOST to share access.

ChatGPT will now show an alert to users who attempt prompts that sexualize children, point them toward specialized support, and suspend the account for 24 hours as a warning. Repeated offenses could lead to permanent removal. OpenAI said it shaped these messages with organizations and researchers, including Protect Children, the Lucy Faithfull Foundation, and Joel Scanlan at the University of Tasmania, who independently evaluated a trial of the deterrent messages.

OpenAI is also publishing guidelines for developers to prevent, detect, and help users report CSAM, and is giving law enforcement AI-powered features such as dedicated plugins, investigative tools, and complex data analysis, with access managed by program administrators, plus training and a field guide for investigators. The company has faced lawsuits blaming its technology for harming users, especially children, and nonprofit Common Sense Media labeled its tech an "unacceptable risk" for teens. Other chatbots, notably Grok, have been used to create nonconsensual sexualized imagery, including of children. Over the past 12 months OpenAI has released a safety blueprint, updated content guidelines, retuned its model to prevent harmful output, and launched a version of ChatGPT for teens.

Source: TechCrunch

Ready to Modernize Your Business?

Get your AI automation roadmap in minutes, not months.

Analyze Your Workflows →