Why do agents hallucinate?

Companies are getting real value out of ChatGPT and Claude today. That part is not in question. What breaks down is trying to solve everything, from sales copy to financial planning to customer support, inside one long conversation with one general-purpose agent.

If you run a small business you have probably watched this happen. You opened a chat months ago, named it something like "Business", and kept it going. It learned your prices, your tone, the names of your regulars. It drafted quotes, answered a payroll question, rewrote your service descriptions. Then one day it told you a client had paid when they had not, or quoted a price you retired in the spring, or wrote a reminder to someone whose job you had cancelled. The chat did not get worse at writing. It ran out of room to know.

What actually happens inside a long chat

A model does not remember the way a person remembers. It reads. Every time you send a message, it reads a window of text: your latest message, as much of the conversation as the window holds, and whatever instructions sit at the top. It answers from that window and from its general training. Nothing else. Four things follow, and together they are what you experience as hallucination.

The window has an edge

The window is large, but it is finite, and the product decides what happens when the conversation outgrows it. Some cut the oldest turns. Some compress them into a summary. Either way, the decision you made in March about payment terms for repeat clients is not in front of the model in June. The model did not forget in the human sense. The text is simply no longer there to be read.

It predicts a plausible continuation

The model's core skill is producing the text most likely to follow what it has just read. When the fact it needs is inside the window, the most likely continuation is that fact. When the fact is gone, the most likely continuation is something shaped like the fact: a date that looks like your dates, a price that looks like your prices, a client name that appeared earlier in a different context. It has no built-in sense of "I have not seen this." Unless it is told to say so, and given a way to check, it fills the gap. That fill is the hallucination.

Your instructions start to fight each other

Over months, one thread accumulates instructions from unrelated jobs. Be brief with customers. Be thorough with the numbers. Formal tone for proposals. Casual on social. Never mention the discount unless asked. Always mention the referral program. Each rule made sense for the job it was written for. Read all at once, they contradict, and the model resolves the contradiction the only way it can: it picks one, or it blends them. You get a thorough, casual, formal proposal that mentions the discount.

There is nothing solid to check against

In a chat, the source of truth is the chat. The only record of what your business knows is text that you and the model typed. Nothing in that thread updates when an invoice gets paid or a job gets moved to Thursday. So the model cannot verify. It can only recall what is still in view, and recall degrades as the thread grows. A person in the same position would open the accounting software. The chat has no software to open.

A chat is not an agent

The word gets used loosely, so here is the working distinction. A chat answers from its window. An agent is a model given a job and given tools.

The job is narrow on purpose: follow up on unanswered quotes, and nothing else. Its instructions cover that job and only that job, so they never contradict the instructions for invoice reminders, because those live in a different agent.

The tools are defined actions the agent can take against real systems. Read this customer's open invoices in the accounting software. List tomorrow's bookings in the calendar. Open the written follow-up procedure. Send this email. A tool returns current data at the moment the agent asks. The model still does the reasoning and the writing. The facts come from the tool, not from recall.

The difference shows up when something goes wrong. When a chat invents a paid invoice, there is nothing to inspect. When an agent gets it wrong, you can see which tool it called, what came back, and what it did with it. Sometimes the tool returned stale data. Sometimes the agent misread a correct result. Both are fixable, because both leave a trace. A scoped agent with tools still makes mistakes, and anyone who tells you otherwise is selling something. What changes is that the mistakes become visible and specific instead of invisible and plausible.

What a person has that a single chat doesn't

This is the same principle every company already applies to people, which is why the parallel holds.

One overloaded thread, split into three agents

Take the "Business" thread from the top of this piece and list what it was doing: drafting and chasing quotes, confirming bookings, reminding late payers, answering the occasional question about the month's numbers, writing the odd social post, and once, a question about how to handle an employee's leave. Six jobs, one window, no tools. Here is how that splits.

Quote follow-up. One agent, one job: find quotes that went out and got no answer past the cadence you set, and send the follow-up your written procedure describes. Its tools are the job or CRM tool where quotes live, that procedure, and email. It never invents a price, because it reads the quote that was actually sent. Its owner is whoever runs sales, and for the first weeks every message goes out only after that person approves it.

Invoice reminders. A second agent reads the accounting software for what is overdue and the bank feed for what has landed, so it never chases someone who paid yesterday. Its tool set is those two plus your payment-terms document. It drafts and you send, until you are ready to let it send within limits you set. Money never moves on an agent's say-so. That line does not move.

Scheduling. A third agent reads the calendar and the job board and proposes slots or confirmations. Its scope is the calendar. It has no idea what anyone owes you, and it does not need to.

What is left: the monthly numbers question is a real job and a good fourth agent later, once the accounting connection exists and has earned some trust. The social posts get their own thread. The leave question stays with a human, because it is judgment about a person, and judgment is the last thing you delegate.

Each agent now has one scope, a couple of tools, one owner. When one is wrong, you know which one and what it read. The thread that did everything could never tell you that.

Model limit or overload? How to tell

Two different problems produce the same wrong answer, and they have different fixes, so it is worth telling them apart. The test costs you a few minutes.

Rerun it clean. Open a fresh chat. Paste only the facts the question needs, current as of today, and ask again. If the answer is now right, the model was never the problem. The thread was. That is overload, and the fix is scope and tools, not a different model.

Trace the wrong fact. Ask where the wrong detail could have come from. If it was never in the conversation and never in any tool the model could reach, it could not have known it, and it filled the gap. The fix is a source: connect the tool that holds the truth. If it was in the thread but far back, the window moved past it. Same fix. If it was recent, correct, in plain view, and the model still got it wrong, you are closer to a real limit, or to an instruction that told it two things at once. Reread your own instructions before you blame the model.

Know the real limits. Some failures are the model's, and no amount of scoping fixes them. Arithmetic across a long list of numbers. Events after its training ended. Specialized legal or tax judgment. A blurry photo of a handwritten note. For these the fix is a tool built for the task, or a person, not a better prompt. A calculator does not hallucinate. Neither does your accountant, at least not about your books.

What to write down before building anything

Every fix above assumes there is something to point the agent at. In most small businesses, part of it does not exist yet. It lives in the owner's head. So the first work is not technical. It is writing.

This is where our own engagement starts, and it is why the first thing Holybiz does with a company is not build an agent. We map the business as it actually operates: the processes, who runs them, where the information lives, where the rework is. Then agents get built by scope, one job each, wired into the tools the business already runs, so the facts stay in the systems that own them instead of inside one agent's shifting sense of a conversation. We run our own companies this way, including the site you are reading. The agents will still be wrong sometimes. The difference is that you will be able to see what they read, fix the source, and move on.