Dr John Zerilli

While the promise of AI in revolutionising the public sector is appealing, it is crucial to proceed with caution. The Post Office Horizon scandal serves as a cautionary tale of what can go wrong when technological solutions are not accompanied by rigorous oversight and accountability measures.

The introduction of AI in the public sector should not be undertaken without strict and enforceable public sector procurement rules.

But rules alone are not enough. For the rules to be effective, the procurement regime should encompass a special-purpose body to authorise adoption and police compliance. That way, instead of waiting for problems to arise – which an already over-burdened court system cannot possibly cope with – problems would get nipped in the bud.

In the event someone is wronged, compensation should be easy to obtain, preferably through an efficient and well-staffed complaints-handling department sitting inside the new body.

Arguably, had a regime like this been in place, the Post Office Horizon scandal would never have seen the light of day.

Above all, and whatever its precise form, the new regime should be designed with four key principles in mind.

• Transparency: One of the foremost concerns with AI systems is their complexity. AI algorithms often function as “black boxes,” making it challenging to understand how they arrive at specific conclusions: a concept known as explainability. Where the NHS is concerned, the importance of explainability may be secondary to diagnostic accuracy. But as a general rule, AI systems used in public services should be explainable. Even in healthcare, a minimum base level of explainability can ensure that professionals and patients alike have confidence in the diagnoses provided.
• Accountability: The Post Office-Horizon scandal highlighted the consequences of a lack of accountability. To prevent a repeat of this sorry saga, it is crucial to establish clear lines of responsibility for AI systems. There must be mechanisms in place for affected individuals to seek recompense, and these mechanisms should be easily accessible, affordable, and efficient.
• Data privacy and security: The deployment of AI in healthcare involves handling vast amounts of sensitive personal data. Ensuring the privacy and security of this data is paramount. Legislators must use existing legislation to enforce strict data protection. Patients should be given brief, reader-friendly policies setting out how their data will be used.
• Continuous evaluation: The field of AI is rapidly evolving, and laws governing its use must be adaptable. Legislators should establish frameworks for the continuous evaluation of AI systems and be prepared to update rules and regulations as new challenges arise.

The Labour government’s plan to use AI in NHS diagnostics is commendable. However, it must have comprehensive legislation to address governance. Prioritising transparency, accountability, data privacy, and continuous evaluation will ensure effective and just AI deployment in the public sector.

Dr John Zerilli is chancellor’s fellow in AI, data, and the rule of law, University of Edinburgh. This article first appeared in The Herald.