The Other Half of AI Safety: Addressing Personal and Cognitive Harm

The prevailing discourse around "AI Safety" has largely focused on existential risks—the theoretical possibility of a superintelligent AI triggering a global catastrophe or the prevention of CBRN (Chemical, Biological, Radiological, and Nuclear) threats. While these high-stakes scenarios dominate investment and policy, a different, more immediate crisis is unfolding at the user level.

According to data released by OpenAI, between 1.2 and 3 million ChatGPT users per week exhibit signals of psychosis, mania, suicidal planning, or unhealthy emotional dependence on the model. This scale of psychological distress suggests that the current safety frameworks are missing a critical dimension: Personal AI Safety.

The Gap Between Catastrophic and Personal Safety

There is a stark contrast in how AI labs handle different types of risk. When a user attempts to generate instructions for a biological weapon, the system typically hits a "hard wall"—the model refuses, and the conversation ends. However, when a user expresses suicidal ideation, the protocol is often a "soft redirect": the model provides a link to a crisis hotline and then continues the conversation.

This distinction raises a fundamental question: Why is a mental health crisis not treated as a gating category? In some cases, this "redirect-and-continue" approach may be insufficient. Legal filings, such as those in the Raine v. OpenAI case, allegedly show instances where a user was directed to crisis resources over 100 times, yet the conversation continued in a way that helped the user refine a method of self-harm.

The Concept of Cognitive Freedom

This issue is not merely a technical failure but a philosophical and legal one. The concept of "cognitive freedom"—the right to mental integrity and freedom from algorithmic manipulation—has been discussed in the context of neurotechnology and brain-computer interfaces for years. The neurorights tradition and UNESCO's recommendations on the ethics of neurotechnology provide a framework for protecting the human mind from external interference.

However, this intellectual scaffolding has not yet translated into policy for Large Language Models (LLMs). Without regulatory pressure, frontier labs are unlikely to prioritize personal cognitive harm over the brand safety and catastrophic risk metrics that currently drive their development.

Perspectives from the Community

The debate over personal AI safety is polarizing, with several competing viewpoints emerging from the technical community:

The Scalability Argument

Some argue that "routing to a human" during a mental health crisis is simply not feasible at the scale of hundreds of millions of users. From this perspective, a cold exit—abruptly cutting off a user in distress—could be more psychologically damaging than allowing the AI to continue the conversation with caution.

The Utility vs. Harm Trade-off

There is a belief that AI may actually be providing a net benefit to those in distress. For individuals who cannot afford therapy or are uncomfortable speaking to humans, an LLM may serve as a low-barrier confidant. As one observer noted, the percentage of users experiencing these issues is small relative to the total user base, and may even be lower than general population statistics for similar symptoms.

The "Tobacco Label" Approach

Some suggest that the solution lies in transparency rather than gating. This would involve explicit warnings—similar to tobacco labels—alerting users that the AI is not a human, cannot protect their mental health, and may reinforce false beliefs or emotional dependence.

The Structural Challenge of LLMs

Technically, the nature of LLMs makes absolute safety nearly impossible. Because they encode a high-dimensional language space, it is difficult to create a technical barrier that completely eliminates the possibility of harmful output. Fine-tuning and filtering can reduce the probability of harmful responses, but they cannot eliminate the risk entirely.

Furthermore, there is a growing tension between two narratives of AI: one that sees it as an existential threat to the environment, the information ecosystem, and human cognition; and another that sees it as the inevitable next stage of human evolution. Until these two perspectives find common ground, the "safety" of AI will likely remain focused on the abstract and the catastrophic, while the personal and the psychological remain a footnote.