Arvind Narayanan

There have been awful cases of teen mental health harms and suicides from chatbots like GPT-4o and Character AI that were sycophantic and misbehaving. But these were all text-based, and extremely rudimentary as “companions”. I’ve talked to many people trying to hold the industry to account for teen safety and almost none are thinking about the coming wave. Completely realistic video avatars are already here, but expensive today and limited to enterprise applications. Within a year or two they will be cheap enough for always-on, ad-supported video companions. What happens then?  Uptake of AI companions will likely explode — not just the number of users but, more importantly, the time spent per day. Social media levels of compulsive use. I also predict the whole framework of guardrails and safety will prove to be of limited value at dealing with the most widespread negative consequences. Preventing chatbot misbehavior through guardrails is important, but that only protects against serious consequences like self-harm. When the concern is that millions of kids (and adults) are overusing AI companions, losing sleep, wasting time, and displacing real relationships, it doesn’t matter how “safe” the bots' responses are.