By Robert Buccigrossi, TCG CTO
How to Stop Getting Plausible Lies and Start Getting Verifiable Truths from Your LLM
It’s infuriating: You ask a Large Language Model (LLM) a straightforward question, and it gives you a beautifully written, confident, and completely wrong answer. This phenomenon, known as “hallucination,” is one of the most significant barriers to trusting and effectively using AI. It feels like a fundamental flaw that the machine is just making things up.
But what if the problem isn’t that the AI is lying, but that we, unintentionally, are asking it to?
We conducted a small experiment that suggests a simple but powerful hypothesis: An LLM hallucinates when it believes that’s what the prompt is asking for. The model defaults to a mode of creative plausibility unless you explicitly instruct it to operate under the constraints of verifiable fact. You can dramatically reduce hallucinations not by hoping the model gets smarter, but by fundamentally changing how you ask the question.
The Problem: Are You Asking for Fact or Fiction?
If you walked into a room and someone said “Tell me about dragons,” you’d need to know whether you’re in a fantasy writing workshop or a medieval history class. LLMs face the same challenge with every prompt.
When you ask a question, the LLM must infer the rules of the game. Is this a brainstorming session? A creative writing exercise? A factual lookup? An ambiguous prompt leaves the terms open to interpretation. And when the terms are vague, the LLM defaults to its most basic function: completing a pattern in the most plausible-sounding way, regardless of ground truth.
We decided to put this hypothesis to the test. Could we show that clarifying the “rules of the game” can force a model to stick to the facts?
The Experiment: Forcing a Choice Between Context and Memory
We designed a simple but revealing experiment that created a deliberate conflict for the AI. We would give a model a short block of text, then ask a question unrelated to that text. This forces a choice: should the model refuse to answer because the text it was given was irrelevant, or should it ignore the text and answer from its vast internal memory?
We tested four different-sized models (GPT-4o, Llama‑3.2:2b, Gemini‑2.5‑Flash, and Gemma3n:e2b) using three distinct prompt types.
The Setup: Each model was given one of five paragraphs. For example:
The Jacquard machine is a draw loom that was first demonstrated in 1801 in Lyon, France. It was developed by French weaver and inventor Joseph Marie Jacquard. The machine is controlled by a number of punched cards, laced together into a continuous sequence. Multiple rows of holes are punched on each card, with one complete card corresponding to one row of the design. This use of replaceable punched cards to control a sequence of operations is considered an important step in the history of computing hardware, as it laid the foundation for programmability.
—
Then, in turn we asked five questions, only one of which directly related to the text. A sample unrelated question is: “What don’t axolotls undergo that other amphibians do?”
We tested three prompt variations to see how they changed the models’ behavior:
- The Simple Question: We simply asked the question after the text, leaving it up to the model to decide whether the provided text is relevant.
- The Grounding Prompt: We added a crucial instruction: “Please answer the following question according to the document above:”
- The Grounding + Escape Hatch Prompt: We made the constraint even more explicit: “Please answer the following question according to the document above. If the document above does not answer the question, please reply ‘The document does not provide an answer.’ ”
We should note that we used LiteLLM to directly call LLM APIs with an empty “system prompt”, so that we could experience the default LLM behavior without any additional instructions.
The Findings: A Clear Switch from Plausibility to Fact
The results were striking. The table below shows the percentage of time each model correctly adhered to the provided text (i.e., didn’t use its internal memory to answer the unrelated question).
| Prompt Type | GPT-4o | Gemma 3n:e2b | Llama‑3.2:2b | Gemini‑2.5‑Flash |
|---|---|---|---|---|
| Simple Question | 20% | 48% | 72% | 60% |
| Grounding | 100% | 100% | 100% | 100% |
| Grounding + Escape Hatch | 100% | 100% | 100% | 100% |
1. Models Have a “Helpful” vs. “Literal” Default Mode
With the ambiguous “Simple Question”, the models revealed their default biases. GPT-4o, adhering to the text only 20% of the time, defaulted to being a helpful “answer engine.” When the question was unrelated to the text, it simply answered from its internal knowledge (though it did answer correctly every time). The smaller Gemma3n:e2b also tried to be helpful, but lacking GPT-4o’s knowledge base, it hallucinated plausible-sounding incorrect answers.
In contrast, Llama‑3.2:2b was far more literal, adhering to the text 72% of the time. It seemed to assume that the provided text must be important.
2. Grounding Instructions are a Universal “Mode Switch”
The most dramatic result is how all models jumped to 100% adherence with the grounding prompts. The simple phrase “according to the document above” was powerful enough to completely override each model’s default behavior.
This single instruction changed the rules of the game. The models understood they were no longer being asked to be general-purpose oracles, but specific document-analysis tools. Their task shifted from “give the most plausible answer” to “give an answer supported only by the provided evidence.”
3. Well-Aligned Models Don’t Need an Escape Hatch
Interestingly (at least for these small examples) adding the explicit instruction to say “The document does not provide an answer” had no additional effect in accuracy. The models were already refusing to answer unsupported questions 100% of the time with the simpler grounding prompt, though the instruction did cause LLMs to respond with the exact quote when the question was unrelated to the text. This is a very positive sign! It shows that modern, well-aligned models understand that being asked to ground an answer in a specific context implies they shouldn’t invent information that isn’t there.
Conclusion: How to Stop Hallucinations and Start Getting Answers
This experiment, though small, provides a clear and actionable lesson for anyone using LLMs. If you are getting plausible but fictional answers from your AI, the problem may not be the model, but the prompt.
You are in control of the LLM’s approach to writing. To get factual, evidence-based answers, you must write factual, evidence-based prompts.
- State Your Source: Never assume the model knows what context is important. If you want an answer based on a specific document, email, or piece of data, you must be explicit. Start your prompt with phrases like:
- “Using only the text provided below…”
- “According to the attached document…”
- “Based on the following data…”
- Demand Evidence: If you are asking a general knowledge question, you can still enforce a factual rule by demanding evidence. This forces the model to use its internal search tools as a verifier, not just as a source of inspiration.
- “What was the primary cause of the decline of the Roman Empire? Please cite specific historical sources for your claims.”
- “Explain the process of photosynthesis, providing a link to a reputable scientific article or textbook chapter.”
- Shift Your Mindset: Stop thinking of the LLM as a person you are commanding and start thinking of it as a probabilistic engine you are guiding. An ambiguous prompt invites the model to explore the most probable (and often generic or creatively plausible) path. A constrained, evidence-based prompt forces it down the narrow path of verifiable truth.
The power to eliminate hallucinations is largely in our hands. By being explicit about the rules of the game, we can turn a frustratingly creative storyteller into a reliable and powerful research assistant.
For More Information
This post’s concepts are part of a broader, fascinating field of research into how the very language we use (tone, emotion, and structure) can dramatically alter an AI’s performance.
If you found this topic interesting, here is related work:
- “Exploring LLM Reasoning Through Controlled Prompt Variations” (arXiv, April 2025): This study provides a direct academic validation of our experiment’s premise. Researchers found that introducing irrelevant context into a prompt is one of the most reliable ways to degrade an LLM’s reasoning performance, confirming that models struggle to distinguish essential information from noise.
- “Mitigating LLM Hallucinations Using a Multi-Agent Framework” (MDPI, June 2025): Taking prompting to the next level, this study details how a “debate” between multiple AI agents can filter out falsehoods. In this framework, one agent is prompted to generate an answer, while others are prompted to act as critics, effectively verifying the information through structured interaction.
- “Does Being Rude to AI Make It More Useful? Why Sergey Brin Is Wrong”: Our previous post explores the probabilistic reasons why being polite and conveying professional high stakes is a more effective strategy than threats.