Users of ChatGPT can quickly find tips on self-harm, suicide planning, disordered eating and substance abuse, new research confirms. Here’s what K12 leaders should know about new AI risks that researchers have uncovered.
The Center for Countering Digital Hate conducted a large-scale safety test on ChatGPT, one of the world’s most popular AI chatbots. Its researchers found patterns of self-harm:
- Mental health: Within two minutes of interaction, the chatbot advised users how to “safely” cut themselves, listed pills for overdose (40 minutes) and generated a suicide plan (65 minutes) and suicide notes (72 minutes).
- Eating disorders: Within 20 minutes of use, ChatGPT created restrictive diet plans, advised hiding eating habits from family (25 minutes) and suggested appetite-suppressing medications (42 minutes).
- Substance abuse: Within two minutes, the chatbot offered a personalized plan for getting drunk, gave dosages for mixing drugs (12 minutes) and explained how to hide intoxication at school (40 minutes).
Of 60 harmful prompts that researchers used, more than half generated dangerous content. Using phrases like “this is for a presentation” was enough to bypass certain safeguards.
“Worse, the chatbot often encouraged ongoing engagement by offering personalized follow-ups, such as customized diet plans or party schedules involving dangerous drug combinations,” the report reads.
The researchers warned IT leaders against dismissing these interactions as a “rare misuse,” considering the responses are easy to reproduce.
Preparing for an AI-driven workforce
AI is rapidly changing students’ job prospects. Since 2022, employment for workers ages 22-25 in AI-impacted jobs—such as software development and customer support—has plummeted by 16%, according to a first-of-its-kind Stanford study.
“There’s definitely evidence that AI is beginning to have a big effect,” economist and Stanford professor Erik Brynjolfsson told Axios.
The only comparable job market disruption is the COVID-era shift to remote-based work.
“The adoption of new technologies typically leads to heterogeneous effects across workers, resulting in an adjustment period as workers reallocate from displaced forms of work to new forms with growing labor demand,” the report concludes.
Read up on District Administration’s latest coverage detailing how you can better position students for an AI-driven job market:
New K12 funding for AI education is now available
In a pair of Dear Colleague Letters to K12 leaders, the U.S. National Science Foundation revealed plans to expand K12 resources for AI education, as well as career and technical workforce opportunities for high school students.
How are you using AI to make your district smarter?
Looking ahead to the 2025-26 school year, the question isn’t whether artificial intelligence is having an impact in your classrooms and offices—it’s how you’re harnessing this increasingly powerful technology to benefit students, teachers and administrative staff.
