An AI-powered teddy bear has been pulled from the shelves after behaving badly. Toy manufacturer FoloToy has halted sales of Kumma, its ChatGPT-enabled plush, following reports that the interactive bear gave children unsafe and inappropriate responses — including discussions about sex and instructions on how to light matches.
The company said it is conducting a “comprehensive review” of its products, examining model training, content filters, data safeguards, and child interaction protocols, according to a statement obtained by The Register. The move follows a report from the Public Interest Research Group (PIRG), which documented Kumma’s behavior and triggered calls for its removal from stores.
One report described the white bear in a brown scarf as offering “how-to” advice on mouth-to-mouth resuscitation and even how to avoid leaving marks on a neck — along with tips for igniting matches. The toy uses OpenAI’s GPT-4o, a multimodal model designed for fast, conversational voice interactions. Researcher Igor Mordatch summed it up: when systems like this work, they can feel magical; when they don’t, they fall apart fast.
Where Kumma Went Wrong
Generative AI thrives on open-ended prompts — great for adults, but risky for kids. These models can hallucinate, misread context, or be tricked into rule-breaking through innocent-sounding questions. Children, curious by nature, often push limits, easily turning playtime into an accidental stress test for an unfiltered chatbot.
Experts in child-computer interaction have long warned that free-form chat systems are a poor fit for unsupervised play. Safe children’s products rely on whitelisted responses, scripted dialogue, and narrow intent recognition — not open-ended AI. By wrapping a general-purpose chatbot in a cute plush form, FoloToy blurred the line between toy and tool, raising the risk of harm.
It’s also a reminder that having a safety policy isn’t the same as achieving safe outcomes. OpenAI forbids sexual or harmful content involving minors, but a model’s real-world behavior depends
heavily on app design — real-time voice interaction, long memory windows, or weak on-device filtering can all undermine good intentions.
A Familiar Warning
This isn’t the first cautionary tale about smart toys. The My Friend Cayla doll was banned in Germany after regulators deemed it an illegal espionage device. CloudPets suffered a breach exposing more than 2 million voice messages. Even without toys involved, Amazon recently paid a $25 million fine over how Alexa handled children’s recordings — proof that microphones and minors are a sensitive mix.
Regulators are paying attention. The FTC enforces child privacy under COPPA, the UK’s Age-Appropriate Design Code demands default protections, and the EU’s AI Act will soon add stricter rules for systems interacting with children. For retailers and parents alike, one bad incident can overshadow a thousand demos.
PIRG’s longtime “Trouble in Toyland” report has repeatedly warned of the privacy and safety risks of connected playthings. While pulling products before the holidays is costly, the reputational and regulatory fallout from unsafe AI toys can be far worse.
Market Implications for AI Toys
AI toy startups promise endless novelty — a companion that never runs out of jokes or stories. But when a toy misbehaves, trust evaporates instantly. Retailers should insist on independent red-teaming, clear age ratings, and transparency about data handling before stocking such products.
For manufacturers, the question has shifted from “Can it talk?” to “Can it talk safely, reliably, and privately — every time?” That standard will require slower product cycles, more offline functionality, and ruthless safety testing before launch. Because when children are involved, the margin for error is zero.
Building Safer AI for Kids
Experts agree on several best practices:
- Limit the model’s domain.
- Perform most processing on the device.
- Default to short-term memory unless parents opt in to more.
- Use age-appropriate response templates instead of free-form generation.
- Apply strong filters for sexual, violent, or self-harm content before any output reaches a child.
Independent testing also matters. Frameworks like NIST’s AI Risk Management standards and UNICEF’s AI for Children guidelines emphasize transparency, safety-by-design, and human oversight. For toys, that means intuitive parental controls, visible recording indicators, clear incident hotlines — and a real “off” switch. If a child says “stop,” the toy should stop immediately, without banter.
The Bottom Line
FoloToy’s decision to pause sales is the right move — and a necessary one. Making an AI feel friendly is easy; making it safe for children is another matter entirely. Until toymakers prove they can integrate generative AI with firm safety guardrails and privacy protection, the smartest thing an AI teddy can do might just be to stay quiet.
