Welsh businesses could be exposing themselves to costly mistakes and poor decision-making by relying on artificial intelligence to produce information without ensuring employees have the skills, knowledge and time needed to check its accuracy.
This was a topic highlighted at a recent Business News Wales AI, Tech and Innovation editorial Board meeting
As more organisations introduce AI into everyday business operations, concerns are emerging about the extent to which employees can confidently assess the information generated by increasingly sophisticated systems.
While the technology can produce detailed reports, customer communications and business recommendations within seconds, the speed and apparent quality of those responses do not necessarily guarantee that the information is accurate or suitable for its intended purpose.
For employers investing in AI to improve productivity, the challenge is ensuring that greater efficiency does not come at the expense of professional judgement, particularly where generated information influences commercial decisions, customer relationships or important operational activities.
One of the risks is that AI-generated material can appear convincing even when it contains errors, overlooks important details or fails to account for the specific circumstances of a business. Employees may be presented with polished documents and apparently authoritative answers without having sufficient knowledge of the subject to identify weaknesses or challenge the conclusions.
This raises questions about whether businesses are doing enough to prepare their workforce for the responsibilities that come with using AI, especially where organisations introduce new tools before establishing clear processes for reviewing and approving their outputs.
Many businesses are exploring approaches that retain human involvement in AI-supported activities, allowing employees to assess generated information before it is used. However, simply requiring a person to approve an answer does not automatically provide effective oversight, particularly if the individual lacks the relevant expertise or is expected to review large volumes of material within limited timeframes.
Laura Newman, Chief Commercial Officer at ModelProp, describes her company’s approach as “AI with a human eye overlay”, highlighting the importance of retaining human involvement when technology is used to support business activities.
For human oversight to be meaningful, employees need to understand what they are checking, why particular information matters and what action to take when an AI-generated response is incomplete, inaccurate or inappropriate.
The issue becomes more complicated when the technology produces lengthy reports or detailed analysis that require significant effort to verify. Although AI may reduce the time needed to prepare an initial document, the work involved in checking its contents can transfer that burden to another employee, potentially reducing the productivity benefits the organisation expected to achieve.
This creates a need for businesses to assess AI investment across the entire working process, rather than measuring success solely through the speed at which information can be generated.
Management teams will also need to create an environment in which employees feel able to challenge AI-generated results and report problems without being seen as resistant to innovation. Feedback about inaccurate responses, excessive checking requirements or unsuitable applications can provide valuable evidence about whether the technology is delivering the intended benefits.
For Welsh companies seeking to build AI into their operations, the priority should therefore be developing the knowledge, processes and accountability needed to use the technology responsibly while retaining the productivity advantages it can offer.
