Artificial intelligence (AI) systems are revolutionizing sectors such as healthcare and customer service, offering promises of speed and efficiency. However, a recent analysis reveals that these systems can accumulate hidden risks when operating without active supervision. These risks can manifest in brand damage, cybersecurity issues, and ethical concerns, which often remain hidden until a public crisis occurs.
The problematic machine
A notable example is the GP at Hand application from Babylon Health, launched in 2017. Although it promised 24-hour digital triage, external audits found that the system underestimated the severity of chest pain and produced gender-biased results. This led regulators to warn about its methodology, highlighting the consequences of treating governance as a post-launch remedy.
In a more recent case, the British company DPD experienced a slip with its chatbot. After a routine update, the chatbot began to interact inappropriately, severely affecting customer trust and damaging the brand’s reputation. This situation arose because governance was not integrated into the system design.
On the contrary, the virtual assistant Erica, from Bank of America, illustrates the success that can be achieved with proper governance. Since its creation, its architecture was designed with a clear focus on governance and limitation of its scope, which allowed it to handle interactions in a highly regulated sector without jeopardizing the entity’s credibility.
To prevent similar crises, mechanisms such as the Agent broker are suggested, which verifies permissions and ensures alignment with policies, and the Evidence latency budget, which establishes the speed of availability of evidence for any AI action. The key to the success of AI lies in effective governance, which must be established from the beginning to prevent technology from becoming an imminent risk.