# How to Check AI Output Before You Trust It: A Risk-Tiered Method > Sort AI output into low, medium or high stakes, then run the matching checks for facts, numbers, code, summaries and translations, with a log template. - Canonical page: https://learnhubplay.nl/blog/checking-ai-output-before-you-trust-it - Published: 2026-10-12 - By: LearnHubPlay, an independent software studio in Amsterdam - Topic: AI at Work - Tags: Verifying AI output, Checklists, AI policy, Privacy & security Say a colleague drops an AI-written competitor summary into a client proposal. Every sentence reads well. One figure comes from a 2022 report, one "quote" is the model's paraphrase wearing quotation marks, and a package named in the appendix script has never existed. Nobody checks, because the text looks finished. Checking everything at full depth is not realistic. Checking nothing has a record: lawyers sanctioned for citing cases that did not exist, an airline ordered to pay a customer who relied on its chatbot's wrong answer, and [2,184 court and tribunal decisions](https://www.damiencharlotin.com/hallucinations/) listed in a public database of hallucinated AI content (a few only alleged) as of 10 October 2026. This guide gives you a three-tier method that sets how hard to look, checks for five kinds of output, and a short log that turns checks into a team habit. You will be able to tier any output in under a minute. ## Why a polished answer gets waved through Automation bias is the tendency to follow an automated aid instead of checking it. In a 2010 review in *Human Factors*, [Parasuraman and Manzey](https://doi.org/10.1177/0018720810376055) found that with imperfect aids it produces omission errors (missing a problem the aid did not flag) and commission errors (acting on a wrong recommendation). It occurs in novices and experts, their abstract says, "cannot be prevented by training or instructions", and affects teams. In our view, "be more careful" is not a control; the fix has to be structural: a tier, a checklist, a second person. Generative AI adds a fluency problem: NIST's generative AI profile defines confabulation as "confidently stated but erroneous or false content" and warns that confabulated citations or logic may further mislead people into trusting an answer. Overtrust has been measured. In a 47-person study, [Perry and colleagues](https://arxiv.org/abs/2211.03622) found that participants with an early Codex-based assistant wrote significantly less secure code and were more likely to believe it was secure. A 58-student study by [Sandoval and colleagues](https://arxiv.org/abs/2208.09727) found critical security bugs at most 10% more frequent than in a control group, so the effect size is uncertain. ## Two failures, and the check that would have caught each | Case | What went wrong | The check that would have caught it | |---|---|---| | *Mata v. Avianca* (S.D.N.Y., June 2023) | A brief cited cases ChatGPT had invented. The lawyer who did the research testified that he entered the Varghese citation in a lookup site and "couldn't find it"; the brief went in anyway. He also asked ChatGPT whether the case was real and was told it was. Penalty: $5,000, jointly and severally on two lawyers and their firm | Treat a failed lookup as a finding. Verify in a source the model does not control; asking the model to confirm itself is not a check | | *Moffatt v. Air Canada* (BC Civil Resolution Tribunal, February 2024) | A website chatbot said a bereavement fare could be claimed retroactively; the linked policy page said otherwise. The tribunal held the airline responsible for all the information on its website and ordered it to pay C$812.02 | Test a tool's answers against published policy before customers see them. Whoever publishes the output owns it | The judge in the [*Mata* sanctions opinion](https://storage.courtlistener.com/recap/gov.uscourts.nysd.575368/gov.uscourts.nysd.575368.54.0.pdf) wrote that there is "nothing inherently improper" about using a reliable AI tool, but that rules impose "a gatekeeping role on attorneys to ensure the accuracy of their filings." The [*Moffatt* decision](https://decisions.civilresolutionbc.ca/crt/crtd/en/525448/1/document.do) is a single small-claims ruling. Read both as statements of who answers for the output, not as rulings on a technology. ## Tier the output before you read it Tiers are our rule of thumb, not a standard; NIST's profile says organizations may apply existing risk tiering to generative AI. The cost of being wrong sets the tier, not how confident the output sounds. | Tier | Test | Minimum check | Sign-off | |---|---|---|---| | 1. Low | All of: only you will use it; a mistake would show at once; nothing is sent, published, merged or run; no money, law, health, safety, security or personal data | Read once for fit. Verify the one or two facts you will reuse | You | | 2. Medium | Any one of: colleagues will rely on it; it feeds a decision someone else reviews; it goes in an internal doc or pull request | Verify every claim a decision depends on, outside the chat. Run code and tests. Sample the rest | You, plus a reviewer who sees the source | | 3. High | Any one of: it leaves the organization; it touches money, law, health, safety, security or personal data; it is hard to undo | Verify every checkable claim. Independent reviewer. Logged | A named owner and a reviewer who did not write the prompt | The highest tier any test matches wins. Then move up a tier if any answer is yes: Will anyone outside your team act on this? Is it hard to undo once sent, merged or published? Would a mistake stay invisible until it hurts? Same assistant, three tasks: ten project names (tier 1), a customer-interview summary for the product team (tier 2), your support bot's refund answers (tier 3, the *Moffatt* pattern). ## What to check for each kind of output ### Facts and citations Check three things in order: the source exists, it says what the output claims, and it is current. Look it up in a catalogue you control (library search, Crossref, a court database, the vendor's docs), never by asking the model. Then find the sentence and compare its date or version with the claim. ```bash # Is the DOI registered? "responseCode":1 means yes, 100 means not found curl -s https://doi.org/api/handles/ ``` A registered DOI proves little. Compare title, first author and year with the citation: a real DOI on the wrong paper is still a fabricated reference. Crossref's API (`api.crossref.org/works/`) returns 404 for DataCite DOIs such as arXiv's, so a 404 there is not proof of a fake. Retrieval does not remove the need. Magesh and colleagues [found](https://doi.org/10.1111/jels.12413) that commercial legal research tools built on retrieval hallucinated 17% to 33% of the time (*Journal of Empirical Legal Studies*, 2025), less than a general chatbot but far from never. ### Numbers: recompute from the original data Accept a number only when you can state its value, unit, base and period, and recompute it from the original data. Say churn went from 4.0% to 3.2%: a 0.8 percentage-point fall, or a 20% relative reduction (0.8 ÷ 4.0). "Down 0.8%" is wrong, and "down 20%" needs the word relative. Recomputing is not enough. In a [2023 high-school maths study](https://doi.org/10.1073/pnas.2422633122), GPT-4 answered each of 57 practice problems ten times and was correct 51% of the time on average. Most errors were wrong steps (42% of attempts), not wrong arithmetic (8%). The authors note newer models have improved, so read it as where errors hide: check the method, not only the final sum. ### Code and dependencies Check what it imports or installs before you run anything. Then run it somewhere disposable, such as a container or a VM with no credentials or personal files in it. [Spracklen and colleagues](https://arxiv.org/abs/2406.10279) generated 576,000 code samples with 16 models: on average, at least 5.2% of packages from commercial models and 21.7% from open-source models did not exist, across 205,474 unique invented names. An attacker can register an invented name, which the authors flag as a supply-chain risk. ```bash # npm: when was it first published, and where is its source? npm view time.created repository.url # npm: does it run code on install? Look for preinstall, install, postinstall npm view scripts # PyPI: 200 means the project exists, 404 means it does not curl -s -o /dev/null -w "%{http_code}\n" https://pypi.org/pypi//json ``` Existence is the floor, not proof of safety. A package first published last week, or with no linked repository, deserves suspicion (our heuristic); the repository field is self-declared, so open it and confirm it is the same project. Then: - Check every function, flag and config key against the version you run (`--help`, the type checker or that version's docs); packages are not the only thing a model invents. - Write a test that fails without the change, and watch it pass. - Read the diff as a stranger's pull request: input handling, authentication, SQL, file paths, secrets. Run your usual linters and security scanners. ### Summaries: check what was left out A summary can fail by omission and misplaced emphasis as well as by wrong facts, and the first two are harder to see. In a [five-week 2024 trial for the Australian regulator ASIC](https://ia.acs.org.au/article/2024/humans-outperform-ai-in-australian-govt-trial.html), five assessors scored human summaries 81% and a Llama2-70B summariser 47%, citing incorrect information, missed key points and odd emphasis. ASIC said a week of tuning likely hindered the model, so read it as a failure pattern, not a current rate. 1. Pick five sentences, preferably ones you will act on, and find the passage each came from. 2. Scan the source for headings, numbers and every "not", "except" and "unless"; confirm each is in the summary or was dropped on purpose. 3. Ask for a supporting quote per claim, then search the source for that exact text. A quote that is not there removes the claim. ### Translations Errors hide in fluent text, so a reader who cannot read the target language cannot check it. Verify names, numbers with their dates and units (watch decimal separators), negations and conditionals, and terms with legal or technical meaning. Have a fluent reader review any tier 2 translation. A back-translation by a different tool is only a tripwire: a mismatch is a finding, and a match proves little. Tier 3 text (contracts, medical or safety instructions) goes to a professional translator. ### Output nobody reviews first: bots and automations When a bot answers customers directly, no person reads each answer, so check upstream. Write 20 to 30 real customer questions, each with the expected answer and its policy page (our starting size), and rerun them whenever the model, prompt or policy changes, as in the fixtures routine of our [team prompt library guide](https://learnhubplay.nl/blog/team-prompt-library). Read a sample of live conversations weekly and add every wrong answer to the set. A link to the policy page is no defence: in *Moffatt* the chatbot's answer linked to the very page that contradicted it. ## A ten-minute routine, shown on one summary For a tier 2 output: 1. **Tier it** with the three questions. 2. **Mark every claim:** names, numbers, dates, quotes, citations, URLs, imports. 3. **Rank by consequence:** which claims, if wrong, would change the decision? 4. **Verify the top claims outside the chat:** open the source, recompute, run the code. 5. **Fix or cut.** Delete what you could not verify rather than hedging it. If one citation, quote or package proved invented, treat every item of that kind as unverified until opened (our rule). 6. **Log it**, for tier 2 and 3. Say the competitor summary from the opening has five checkable claims (numbers invented for illustration): | Claim | If wrong | Check | Result | |---|---|---|---| | `pip install acme-sdk` | Runs on a colleague's machine | PyPI JSON check | 404: use the real package | | "Revenue grew 18%" | Changes the pricing argument | Recompute from the annual report | Report gives 11%: fix | | "31% market share" | Anchors the proposal | Find the report; check year and who measured | 2022 figure: date it or cut it | | A CEO "quote" | Reputational | Search the transcript | Not found: delete | | "Founded in 2014" | Low | Company website | Matches | Work top to bottom; the founding year can wait. ## A log entry and a sampling rule for teams ```yaml date: 2026-10-23 tier: 3 output: "Competitor summary for the Acme proposal" tool_and_model: "" prompt_or_template: "