Beyond Intelligence: Why AI Needs the Wisdom of Humans and Smart Data
Introduction
Artificial intelligence is transforming numerous industries, from healthcare to finance, by enhancing efficiency and supporting better decision-making. However, like any powerful tool, AI is not without its flaws. It can produce biased results, generate misinformation, or behave unpredictably. Here are a few recent examples of publicly available AI (large language model) failures:
- Air Canada refund AI mishap – February 2024
Air Canada’s AI chatbot misinterpreted a refund request, issuing over-refunds and displaying how flawed AI can cause financial damage.
- Chevrolet chatbot exploited – December 2023
A customer manipulated a dealer’s AI chatbot into offering a $76,000 Chevy Tahoe for just $1 revealing how easily low-security chatbots can be tricked.
- UK High Court fake case-law citations – June 2025
The UK High Court warned lawyers after discovering AI generated bogus citations (18 fake cases in one filing), cautioning misuse could result in contempt proceedings.
- DPD swearing chatbot – Jan 2024
A disgruntled customer managed to persuade their chatbot to swear and write a poem all criticizing the company’s delivery service.
Occasionally AI systems can also confidently produce factually incorrect, nonsensical, or even fabricated outputs; this exhibited behaviour is referred as AI hallucinations. Errors such as these erode customer trust and may lead to financial and reputational damage.
The problem is that whilst AI based tools are powerful tools they lack intent, understanding, or accountability and are incapable of independent thought and so cannot validate the integrity of their responses. Therefore, human involvement and oversight throughout the entire AI system lifecycle is of paramount importance. This responsibility falls on those who design, train, and integrate them into decision making systems. This is where IT vendors play a critical role, by providing tools and frameworks for delivering robust governance, metadata design, testing, and quality control tools for AI-powered products. From development to production, suitable tool usage can greatly help organizations deploy AI more responsibly and effectively mitigate risks while maximizing the technology’s potential.
Whilst there are many causes of AI failures, such as:
- Unclear business objectives and ill-defined business problem.
- Hyped expectation and excitement surrounding AI leading to inflated expectations and failure.
- Some techniques, such as Large Language Models and Neural networks, have great product complexity with many interconnected components and dependencies. This coupled with highly complex data preparation pipelines can lead to unexpected behaviors and outcomes.
- AI models are only as good as the data they are trained on. Using low-quality, inaccurate, or biased data will lead to poor outcomes, therefore effective data governance is required.
This blog does not aim to provide an exhaustive analysis of why AI systems fail. Rather, it focuses specifically on the data-centric perspective highlighting key practices to adopt that help ensure high-quality AI outputs and reduce the risk of hallucinations. These include, but are not limited to consideration of:
- Sources of data used for AI training
- Quality of data used
- Governance of data and Validation of AI models
Data Sources
The old adage “Garbage In, Garbage Out” (GIGO) applies equally to the training data that is used for AI development. Ensuring high-quality data sources starts with a clear understanding of the business objectives and a well-defined problem statement. These essential basics guide the design of metadata through data modeling: its structure, definitions (both at the technical and business level), and relationships that align with the intended usage purpose. This ensures that the data is well defined, understood, suitable for governance, and is fit for purpose. Without robust data modeling as a foundation, AI systems are highly likely to have built-in failure from the start.
Data Quality
Having a well-defined AI data and lineage model is the foundation on which to develop high quality data. It is used to:
- Develop automated data validation rules for testing to find invalid formats, values, and ranges.
- Profile and clean data using automated tools to detect missing values, duplicates, inconsistencies, outliers, and logical inconsistencies.
- Develop data quality metrics, for example, for completeness, accuracy, freshness and monitor them continuously.
- Shift-left to test early and often in the development lifecycle of the AI model
- Validate source data, AI results, and relevance to business outcomes.
Governance of Data and AI Models
Governance applies to both the data and the AI model and must span both technical matters and organizational levels and align with legal, ethical, and business objectives. An effective governance program will:
- Assign ownership and accountability to data stewards to ensure continuing quality and compliance.
- Keep a centralized catalog describing data meaning, source, structure, and usage for effective data management.
- Use the defined metadata to track the origin, movement, and transformation of data, data products, and datasets across systems for transparency and root-cause analysis.
- Ensure compliance with industry applicable and regulatory requirements.
- Use automated tools as part of the governance process to ensure, scale, and effectively govern assets.
- Set policy management rules for access, permitted usage, retention, privacy, security, and classification of both the data and the AI model.
Human involvement is essential across all stages of an AI system’s lifecycle from development, training, and through to operation. Maintaining a human-in-the-loop approach ensures ongoing oversight and the practical application of human wisdom. This not only improves the accuracy, reliability, and adaptability of AI systems but also helps optimize the data and processes behind them. This human guidance is key to minimizing errors, preserving trust, protecting reputations, and avoiding costly missteps.
Conclusion
We started off in this blog citing some examples of AI failures caused by AI hallucinations, part of the solution to avoid this from occurring is to:
- Use advanced modelling tools to continuously update the metadata architecture in response to evolving business objectives and feedback from AI system responses.
- Use automated tools to assist to ensure that the data that you have is high quality and well defined.
- Review your governance practices to ensure that they are fit for purpose and that there is always human oversight in the AI system development lifecycle.
Sandhill Wants You to Take Control of Your Data Before AI Does
erwin Data Intelligence by Quest brings together data cataloguing, quality management, governance, and data literacy to help you build a reliable foundation for AI success. With erwin DI, your data—and your models—are easier to find, understand, govern, score, and use, reducing the risk of AI hallucinations, misinformation, and failure.
Don’t just collect data—make it intelligent.
Explore erwin Data Intelligence or reach out to us to learn how it can support your AI initiatives
Contact Us Now
Ready to turn your data into a strategic asset?
With erwin Data Intelligence by Quest, you can unify data cataloging, quality, governance, and literacy—all in one platform. Make it easier for your teams to find, trust, and use data that fuels accurate insights and AI outcomes.
Discover how erwin DI helps you unlock the full value of your data—securely, intelligently, and at scale.
