Policy brief “Ethical challenges of AI companions”

Existing approaches to user protection often assume that vulnerability is a pre-existing characteristic of particular groups. However, new forms of vulnerability may emerge from sustained human-AI interaction: forms of emotional reliance that become emotional dependency, social withdrawal that leads to permanent social isolation, or a reinforcement of cognitive biases.

These emergent vulnerabilities develop gradually. They are difficult to detect using conventional safeguards. In addition, applying the same safety standards to all users risks imposing unnecessary restrictions on some, while failing to adequately protect others. Governance mechanisms should focus on indicators of elevated risk that emerge through interaction, rather than proceed from fixed assumptions about users’ vulnerability.

The AI Act’s human oversight model assumes a reviewable and attributable decision point — an assumption that does not scale to the continuous, private, and high-volume interaction with AI companions. The same limitation extends to existing evaluation frameworks and technical safety measures: they rely predominantly on single-snapshot benchmarks, while content moderation and behavioural guardrails remain focused on short excerpts from interaction logs. As a result, these tools are poorly equipped to detect harms that emerge gradually through prolonged engagement.

Most infrastructure required for longitudinal oversight already exists as platforms collect and retain information about user interactions and behavioural patterns, yet they use them primarily for optimising engagement and user retention. Governance frameworks should require platforms to apply this same data infrastructure to deploy wellbeing-oriented analysis.

Well-being analysis involves identifying indicators of distress or dependency against clearly defined intervention thresholds. This reorientation of existing data practices, rather than introducing new surveillance, supplies an ethical basis of profiling users, based on the “Know Thy User” recommendation. To support accountability and independent scrutiny, AI providers should organize regular data and privacy audits, and also publish aggregated risk-related statistics on AI companions.

Many AI companions are optimised to maximize engagement through continuous availability, personalised responsiveness, and alignment with user preferences, which systematically reduce opportunities for reflection, disagreement, or disengagement. To address a regulation gap around AI-related manipulation, governance frameworks should disincentivize sycophancy and encourage calibrated forms of human-AI friction as a means of promoting human autonomy.

Developers have already begun introducing safeguards such as crisis referrals, break prompts, and restrictions on certain anthropomorphic behaviours. However, these remain largely reactive, voluntary, and rarely accompanied by mechanisms to revisit user consent as interactions become more personal.

Existing transparency requirements are largely based on the assumption that users can protect themselves from harm if they are provided with sufficient information about the system. In the context of AI systems, this assumption is valid only partially. Users may remain fully aware that they are interacting with an AI system while nevertheless developing emotional dependency, disclosing intimate information, or becoming behaviourally manipulated over time.

Going beyond disclosure, regulation should strive to provide users with the opportunity to understand how and whether these effects may arise over time. There is a need for an active and contextual transparency toolkit operating alongside the companion system to help users recognize engagement patterns, understand how personal information shapes the interaction, and identify when the relationship builds up in intensity or crosses invisible behavioural borders.