Almost every "the chatbot is wrong" complaint is a retrieval failure wearing a generation costume. I start by measuring the two separately: take twenty to thirty real failing queries, log the top-k chunks for each, and check by hand whether the correct passage was ever retrieved. That single number splits the problem in half.

From there the work is usually chunking that severs meaning, a missing metadata filter, an embedding model mismatched to the domain, or the absence of a reranker. Then the output schema changes: a verbatim quote and a source identifier for every claim, and a real refusal path below a retrieval-score threshold.

You keep the eval set. That is the part that stops the problem coming back.

A balance should be derived from entries, never stored. Every movement is two rows, and "pending" is a first-class state with its own ageing report rather than an error nobody looks at.

Around that sits the reliability layer: idempotency keys required on every write so a retried request cannot pay twice, webhook verification and replay handling, a poller that chases unresolved transactions to a terminal state, and a daily job comparing your position against the provider's statement that alerts on any non-zero delta.

Finding a small discrepancy the day it appears is ten minutes. Finding it at quarter end is a week.

Agents run long and processes restart - a deploy, a timeout, an OOM. If state lives in memory, the user sees a silent failure and a half-finished task with side effects already committed.

The fix is unglamorous: checkpoint every state transition to Postgres, make every tool call idempotent so a replay cannot double-charge or double-send, and make the final answer structurally depend on every tool result being present. When something is missing, the agent surfaces an explicit incomplete state rather than writing around the hole.

That last part matters more than any prompt change. A tool erroring is easy - it is loud. The expensive failure is three of five tool calls landing and the summary reading as though all five did.

The auth, audit and encryption layer is most of the real work in a compliance-sensitive build, and it is shared by every part of the product. Built once and correctly, each additional module is cheap. Built fast to hit a deadline, the exposure is in the foundation.

What that means concretely: permissions enforced at the data-access layer so no view can forget them, with a written authorisation matrix as the spec. An append-only audit log recording actor, action, object, timestamp and IP, never deletable from the application. TLS in transit and AES-256 at rest. E-signatures captured with a hash of the exact document version signed, so you can prove later what was signed and not merely that something was.

Infrastructure goes in your cloud account, under your control, with the BAA signed by you. I build against synthetic data and migrate real records only at cutover.