Decision support systems (DSS) refer to information systems developed to assist decision making. The evolution of DSS, from traditional data modelling to neural network-based systems, has resulted in the significant expansion of domains in which these AI-based systems can operate, speed up and assist in decision-making processes. Today DSS have significant capabilities that enable the extraction of complex patterns from large amounts of data, undertaking more complex, autonomous reasoning tasks which can enhance and support human judgement. However, several challenges remain for the governance of DSS. The table below provides an overview of key open issues faced by organisations in the governance of DSS, and how AIOLIA’s organisational guidelines can help address these.
| Open Issue & Concerns | How Organisational Measures Help |
|---|---|
| Open issue #1: How can organisations prevent sustained DSS use from eroding the professional competencies on which meaningful human oversight depends? | |
| Concern: Practitioners defer too heavily to DSS outputs rather than exercising professional judgement. This risk arises not only from technical system design but from organisational practices that amplify automation bias, including productivity KPIs, heavy workflows, and a diluted safety culture. Sustained engagement with DSS risks eroding professional competencies over time, degrading the diagnostic capability and situational awareness essential for responsible decision-making. Market pressures incentivising rapid AI adoption compound this further, as automation complacency means users of reliable systems gradually reduce scrutiny. | Organisations can put in place structured training programmes to maintain AI literacy and professional competency alongside DSS use, and commit to periodically monitoring how reliance on DSS outputs is evolving over time. By tracking whether practitioners are accepting, questioning, or overriding AI recommendations, organisations can identify deteriorating oversight practices before they become entrenched. These measures recognise that deskilling is a long-term structural risk and that without active intervention, commercial and workflow pressures will consistently favour deference to the system over professional judgement. |
| Open issue #2: How should responsibility for AI-assisted decisions be allocated when liability frameworks remain legally unresolved? | |
| Concern: Complex workflows, task interdependencies, and professional relationships create accountability gaps unless adequate mapping is in place. Every consequential decision must be attributable, explainable, and traceable throughout the system lifecycle. This is particularly acute in high-risk contexts such as healthcare, where liability is distributed across multiple layers of service providers. Liability in AI-assisted decision-making remains a legal grey area, with no established European standard for a fully compliant high-risk medical AI device, and patients’ right to recourse in the event of harm not yet adequately operationalised. | Organisations can implement responsibility assignment frameworks that clearly define who is accountable for each stage of an AI-assisted decision-making process, and establish escalation paths for when something goes wrong. By embedding accountability structures into HR policies, workflow design, and governance processes, organisations ensure that in the absence of legal certainty, responsibility is at least clearly structured and traceable internally. These measures are important precisely because legal clarity does not yet exist: robust internal accountability frameworks are the primary safeguard currently available. |
| Open issue #3: How can transparency obligations be meaningfully upheld when individual-level justifiability is technically unresolved? | |
| Concern: All actors involved in or affected by AI-supported decisions need adequate information to oversee, question, and override DSS recommendations. Transparency obligations are multi-directional: clinicians require explanatory interfaces, whilst patients require plain-language summaries and clear disclosure of AI involvement consistent with informed consent. However, individual-level justifiability is technically unresolved: neural networks cannot explain results for a single patient case, meaning patients cannot contest recommendations whose reasoning is inaccessible to them. In security contexts, operational secrecy places additional hard limits on transparency. | Organisations can design explanation interfaces that require users to form their own judgement before seeing AI outputs, and put in place accessible mechanisms for users and those affected to report, question, or contest AI-assisted decisions. In security contexts, measures on transparency obligations establish what responsible disclosure should look like where it is permissible. These measures recognise that full technical justifiability cannot yet be achieved, and focus on creating the organisational conditions for challenge and review so that the gap between what can be explained and what is required for accountability is actively managed rather than ignored. |
| Open issue #4: How can meaningful human oversight be institutionalised when commercial pressures and fatigue systematically undermine it? | |
| Concern: Human-in-the-loop mechanisms are resource-intensive and fatigue-inducing. Automation bias and alarm fatigue reduce oversight quality in high-pressure environments. A critical distinction exists between formal oversight and meaningful oversight: the former may satisfy procedural requirements whilst failing to deliver genuine scrutiny of DSS outputs. Responsibility for meaningful oversight lies principally at the organisational level, but is shaped by commercial pressures and market incentives favouring rapid AI adoption that require intervention beyond the organisational level. | Organisations can embed a safety-first culture, establish representative AI governance committees that provide collective oversight independent of operational pressures, and define explicit policies requiring human judgement in high-risk or sensitive decisions. These measures recognise that formal compliance with oversight requirements is not sufficient, and that the conditions making oversight genuinely effective must be actively built and maintained. In doing so, they also surface the limits of what organisations can achieve alone, highlighting where policy and regulatory intervention is still needed to address the market-level incentives that undermine meaningful oversight. |
Source: AIOLIA deliverable 3.3