Brief: AI Adoption Mishaps
AI advancement and adoption are moving quickly. That makes it hard to know what these tools can do, and more importantly, what they cannot. It also creates challenges for organizations that mandate staff adoption without years of case studies showing what works, what fails, and why. Fortunately for us, though not for the companies involved, enough cases are starting to emerge to give us an early sense of how AI tools should not be employed. Here are a few of them.
Case: Air Canada Chatbot
What happened: Air Canada deployed a customer service chatbot trained on the company’s policy and guidance documents to give customers quick, accurate answers. In one instance, the chatbot hallucinated a policy. It incorrectly told a customer he could apply for a bereavement fare refund within 90 days of travel, so he bought a full-price ticket to a family member’s funeral. When he later applied for the refund, Air Canada pointed to its policy manual, which said the application had to be filed before the flight. Air Canada later argued in court that the chatbot was responsible for its own actions.
What we learned: Legal liability questions aside, AI tools do not take responsibility for their own actions. They act within goals, permissions, and systems that humans create, even when their specific outputs are not scripted in advance. If AI users abdicate human responsibility for outcomes, those outcomes will disappoint at best and cause harm at worst. Any AI workflow should be designed with clear human responsibility for decisions, especially high-stakes ones. Otherwise, adoption becomes a liability rather than a gain.
Case: Amazon Resume Screening
What happened: Amazon had been developing automated tools to sort through large numbers of resumes in the hope of speeding up the process of identifying and onboarding promising candidates. Despite early enthusiasm for the tool, hiring managers soon realized it showed systematic bias against women. Developers traced the problem to the tool’s training data: the previous 10 years of hiring decisions used to inform its ranking system. Those decisions had been biased, for many reasons, and the tool reflected that bias.
What we learned: AI tools reflect the human-created data they are trained on, along with all its human flaws. AI tools are often trained on very large datasets, and the assumption is that as dataset size increases, minor errors are canceled out by more numerous non-errors. That principle can hold when errors are random. It does not hold when the error is systematic. Training AI on large datasets does not make it neutral, fair, or flawless.
Case: CNET Financial Articles
What happened: The news outlet CNET used AI tools to publish articles on various topics, including financial news. Some were intended to be financial explainers and backgrounders. A CNET reader identified inaccuracies in one article. When CNET audited the AI tool’s performance, it found other articles that required corrections. Some articles published inaccurate financial advice; others appeared to include plagiarized material.
What we learned: One strength of AI is its ability to produce vast amounts of seemingly correct information faster than people can verify it. AI is often oversold as a way to cut labor costs. It can reduce some labor, but other labor is required to validate the increased output. Reckless adoption in pursuit of an overpromised benefit will be more expensive in the long run.
Photo by Matt Cornock, no alterationAI Adoption Mishaps