“Human in the loop” is not enough

The phrase has become commonplace in discussions about healthcare AI.

But simply placing a clinician somewhere in a workflow does not automatically make an AI system clinically safe or useful.

Meaningful clinical oversight requires much more. The clinician needs appropriate information. The purpose of the AI output needs to be clear. The output must arrive at an appropriate point in the clinical workflow. The professional needs to be able to review it critically rather than simply accept it. And responsibility for the clinical decision needs to remain unambiguous.

That is why we prefer to think about clinician-led clinical intelligence, rather than AI making clinical decisions.

Start with structured information

AI outputs are shaped by the information available to them.

If information is incomplete, fragmented or poorly structured, introducing a sophisticated model does not remove the underlying problem.

This is particularly important in complex paediatric pathways, where relevant information may originate from families, schools, questionnaires, clinical records and multiple professionals.

Before asking AI to interpret information, we should therefore ask: Are we collecting the right information, from the right people, at the right point in the pathway?

Structure first. Intelligence second.

AI as decision support

Within EnrichMyCare, clinician-facing AI functionality has been developed around a simple safety principle: AI supports clinical work. The clinician makes the clinical decision.

The useful workflow therefore looks more like: Structured information → AI-enabled clinical intelligence → Clinician review → Clinical decision, rather than Patient data → AI decision.

The distinction is fundamental.

Governance is part of the product

Clinical AI governance cannot be something added after the algorithm has been developed. It has to influence how the system itself is designed.

That includes intended purpose, risk management, clinical safety, software lifecycle controls, data protection, monitoring and the ability to understand how AI is being used within the pathway.

For EnrichMyCare, these controls sit within the wider medical-device and NHS assurance framework, including clinical risk management, software lifecycle controls, DCB0129 clinical safety and data-protection requirements.

Governance therefore should not sit outside innovation. Good governance enables responsible innovation.

The purpose is not to remove clinicians

In many pressured pathways, the scarce resource is clinical expertise.

The objective of AI should not be to remove that expertise from the pathway. It should be to help clinicians use it where it creates the greatest value.

If technology can organise information before an assessment, reduce repetitive documentation, surface relevant patterns or prepare information for review, clinicians can spend more of their time interpreting, communicating and making decisions.

Those are precisely the activities where professional judgement matters most.

A better test for clinical AI

Rather than asking “Can AI do this?” we should ask: “Does using AI here strengthen the pathway while preserving appropriate clinical judgement, safety and accountability?”

Sometimes the answer will be yes. Sometimes it will not.

Knowing the difference is part of responsible clinical innovation.

Because the future of clinical AI should not be about choosing between clinicians and technology. It should be about designing systems in which technology makes clinical expertise more effective.