{"id":737,"date":"2026-06-09T08:09:55","date_gmt":"2026-06-09T06:09:55","guid":{"rendered":"https:\/\/aiolia.eu\/?page_id=737"},"modified":"2026-06-09T08:14:51","modified_gmt":"2026-06-09T06:14:51","slug":"open-governance-issues-for-emotional-ai","status":"publish","type":"page","link":"https:\/\/aiolia.eu\/index.php\/open-governance-issues-for-emotional-ai\/","title":{"rendered":"Open governance issues for Emotional AI"},"content":{"rendered":"\n

In the process of operationalising AI ethics principles conducted by AIOLIA industrial partners, several open issues emerged according to the specificities of the different types of AI technologies used, which correspond to the three AIOLIA research areas, namely, General-Purpose AI (GPAI), Emotional AI and Decision Support Systems. This is an overview of the key open issues in the governance of AI for Emotional AI, which acknowledges both the role and limits of AIOLIA\u2019s organisational measures in addressing these. For a complete analysis see AIOLIA Deliverable 3.3<\/a>.<\/p>\n\n\n\n

Emotional AI has traditionally been associated with the inference of emotional states from biometric data, a practice that attracted significant civil society and academic criticism on grounds of technical unreliability and ethical risk, and was eventually restricted by the AI Act (Article 5(1)(f)). However, the emergence of large language models has significantly expanded the scope of what constitutes Emotional AI, as has the experimental use of deepfakes in therapy contexts. General-purpose systems can be used for emotionally intimate interactions even without being designed for this purpose, and purpose-built AI companions such as Replika and Character.ai are developed on top of the same GPAI models. In both cases, emotional states are inferred from language rather than biometric data, and may be directly disclosed or inferred from interaction patterns. Beyond inference, LLMs are also capable of emotion emulation, imitating emotional states in ways that introduce anthropomorphism and distinct risks to user autonomy and privacy. Neither language-based emotional inference nor emotion emulation falls within the AI Act’s prohibited or high-risk categories, leaving a significant regulatory gap. Further to language-based AI systems used in the private sphere, therapy contexts represent a distinct governance challenge: these solutions are currently only used in experimental settings, with their effectiveness yet to be established.<\/p>\n\n\n\n

Given these governance gaps, organisations can pro-actively take a precautionary stance, treating Emotional AI systems as high-risk unless demonstrated otherwise. For policy, the challenge is two-fold: regulating the use of AI for emotional ends, regardless of whether they are purpose-built applications or general-purpose; and ensuring regulation protects society from the risks, without blocking the potential for increased human well-being.<\/p>\n\n\n\n

\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
Open Issue & Concerns<\/th>\n How Organisational Measures Help<\/th>\n <\/tr>\n <\/thead>\n
\n Open issue #1: How to distinguish between beneficial and harmful anthropomorphism and emotional engagement?\n <\/td>\n <\/tr>\n
\n Concern:<\/strong>\n The same features that make Emotional AI effective, such as affective rapport and human-like interaction, also carry the risk of emotional dependency, manipulation, and erosion of autonomy. Since the evidence base is nascent, there is no settled threshold for when engagement becomes harmful; what’s more, this threshold will necessarily be context-dependent and may be human-judgement-dependent.\n <\/td>\n \n Organisations can define content and behaviour restrictions and document the limits of legitimate influence, involve mental health professionals in design and monitoring, and invest in research on user impacts. Internal and independent ethics reviews can provide additional scrutiny as systems evolve.\n <\/td>\n <\/tr>\n\n
\n Open issue #2: How should subjective, dynamic, and emergent psychological factors, including vulnerability, be incorporated into risk analysis?\n <\/td>\n <\/tr>\n
\n Concern:<\/strong>\n Vulnerability in Emotional AI users is not always pre-existing or identifiable: it can emerge during interaction, shaped by the system itself. Standard risk frameworks are designed to assess discrete, observable, and largely technical failure modes, and are therefore poorly equipped to capture psychological or cognitive harms that are subjective, gradual, and may only become apparent over extended use.\n <\/td>\n \n Organisations can define vulnerability profiles and identification protocols, integrate mental health expertise into risk analysis teams, and collect user feedback on wellbeing and perceived coerciveness. Continuous monitoring and periodic review allow profiles to be updated as evidence grows.\n <\/td>\n <\/tr>\n\n
\n Open issue #3: What constitutes harmful influence, and how can governance address manipulation that is gradual or unintentional?\n <\/td>\n <\/tr>\n
\n Concern:<\/strong>\n Manipulation by Emotional AI systems may emerge as an unintended product of training dynamics rather than deliberate design, and its effects may accumulate gradually through repeated interaction. These are not well captured by the AI Act’s intent-based prohibition on manipulation.\n <\/td>\n \n Organisations can restrict and audit behavioural nudging, establish multi-level safety criteria that go beyond legal compliance, and monitor long-term interaction patterns for signs of harmful drift. User contestation mechanisms provide an additional check.\n <\/td>\n <\/tr>\n\n
\n Open issue #4: How can governance prevent Emotional AI from eroding social norms and interpersonal skills at a societal level?\n <\/td>\n <\/tr>\n
\n Concern:<\/strong>\n Widespread use of AI companions may produce spillover effects on human relationships \u2013 reducing motivation to invest in them and gradually degrading social skills. Deepfake therapy, if normalised, may also contribute to the broader normalisation of deepfake technology in contexts where it causes harm.\n <\/td>\n \n Organisations can communicate risks to individuals, next of kin, and the public, and contribute research findings to broader academic and policy communities.\n <\/td>\n <\/tr>\n\n
\n Open issue #5: Should AI companionship be treated as high-risk, and how should general-purpose AI used for emotional ends be regulated?\n <\/td>\n <\/tr>\n
\n Concern:<\/strong>\n Most Emotional AI applications, especially AI companions and general-purpose systems used for quasi-clinical emotional support, fall outside the AI Act’s high-risk and prohibited categories, leaving a significant regulatory gap. Regulation cannot apply to purpose-built applications only, as GPAI models may be used for emotional ends, or fine-tuned to become Emotional AI systems.\n <\/td>\n \n Organisations can go beyond legal compliance and adopt a precautionary stance, i.e. treating Emotional AI systems as high-risk in the absence of a regulatory requirement to do so. Governance committees and independent ethics reviews can help establish voluntary standards that inform future regulation.\n <\/td>\n <\/tr>\n\n
\n Open issue #6: Are Emotional AI systems effective in clinical therapy settings, and how should this be established?\n <\/td>\n <\/tr>\n
\n Concern:<\/strong>\n The clinical grey zone between wellness and medical products is expanding, yet evidence on the therapeutic effectiveness of Emotional AI remains limited. Without robust evidence, neither benefit nor harm can be reliably established, undermining the proportionality requirement at the heart of good clinical care.\n <\/td>\n \n Organisations can establish dedicated R&D functions or enable access for external researchers, involve mental health professionals in evaluation, and collect and share user feedback on wellbeing outcomes. Participation in academic studies and cross-sector consortia will be key in accelerating the evidence base.\n <\/td>\n <\/tr>\n\n
\n Open issue #7: Are current privacy and data protection frameworks fit for Emotional AI, particularly regarding inferred data, consent, third-party data, and “mind data”?\n <\/td>\n <\/tr>\n
\n Concern:<\/strong>\n Emotional AI systems continuously infer sensitive mental states from interaction data, yet both inferred data and mind data as a whole sit outside GDPR’s sensitive data categories. Consent mechanisms designed for one-off disclosure are ill-suited to long-term, affective interactions; moreover, users could be well aware of the privacy risks, yet have no choice but to give their consent in order to use the platform. Third-party data used in deepfake therapy remains legally unresolved in some cases, such as in grief therapy.\n <\/td>\n \n Organisations can conduct privacy impact assessments that go beyond GDPR compliance to map the full range of data risks, implement enhanced technical and procedural safeguards for certain categories of data, and document how consent mechanisms fall short of equipping users with meaningful agency.\n <\/td>\n <\/tr>\n <\/tbody>\n <\/table>\n<\/figure>\n\n\n\n

Source:\u00a0AIOLIA deliverable 3.3<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

In the process of operationalising AI ethics principles conducted by AIOLIA industrial partners, several open issues emerged according to the specificities of the different types of AI technologies used, which correspond to the three AIOLIA research areas, namely, General-Purpose AI (GPAI), Emotional AI and Decision Support Systems. This is an overview of the key open… Read More »Open governance issues for Emotional AI<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"neve_meta_sidebar":"","neve_meta_container":"","neve_meta_enable_content_width":"","neve_meta_content_width":0,"neve_meta_title_alignment":"","neve_meta_author_avatar":"","neve_post_elements_order":"","neve_meta_disable_header":"","neve_meta_disable_footer":"","neve_meta_disable_title":"","footnotes":""},"class_list":["post-737","page","type-page","status-publish","hentry"],"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/aiolia.eu\/index.php\/wp-json\/wp\/v2\/pages\/737","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/aiolia.eu\/index.php\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/aiolia.eu\/index.php\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/aiolia.eu\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/aiolia.eu\/index.php\/wp-json\/wp\/v2\/comments?post=737"}],"version-history":[{"count":2,"href":"https:\/\/aiolia.eu\/index.php\/wp-json\/wp\/v2\/pages\/737\/revisions"}],"predecessor-version":[{"id":745,"href":"https:\/\/aiolia.eu\/index.php\/wp-json\/wp\/v2\/pages\/737\/revisions\/745"}],"wp:attachment":[{"href":"https:\/\/aiolia.eu\/index.php\/wp-json\/wp\/v2\/media?parent=737"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}