AI harness engineering: what makes an AI agent dependable

A good demo of an AI agent takes an afternoon. An agent that handles real work every day, without someone watching every step, takes engineering. We call that engineering the harness: everything around the model that turns a capable but unpredictable component into a dependable system.

What the harness includes

• Tools: well-defined functions the agent can call, with strict inputs and clear errors, instead of free-form access to everything.
• Context and memory: what the agent sees on each step, what it remembers between steps, and what it is never shown.
• Guardrails: checks before and after each action, such as validating outputs against a schema, blocking unsafe operations, and requiring approval for irreversible steps.
• Evaluations: a test suite of real tasks with known good outcomes, run on every change to prompts, tools or models.
• Observability: traces of every run, so you can see what the agent saw, decided and did.

Start with evaluations, not prompts

The most common mistake is to tune prompts by feel. A change that fixes one example quietly breaks three others. Before optimising anything, collect 30 to 100 real tasks with expected results and score the agent against them. That set becomes your safety net for every later change, including switching to a newer model.

Make tools narrow and honest

Agents fail most often at the boundaries: a tool that accepts vague input, returns an unhelpful error, or does too much in one call. Narrow tools with descriptive names and explicit error messages give the model fewer ways to go wrong and give you clearer logs when it does.

Put humans where the risk is

Not every step needs a person, but some do. Sending money, emailing customers or changing records in a core system are good candidates for an approval step. Reading, summarising and drafting usually are not. Designing these checkpoints deliberately is cheaper than cleaning up after an agent that acted alone.

Watch it in production

Behaviour drifts as inputs change and models are updated. Monitoring success rates, costs, latency and escalations, and sampling runs for review, catches problems before your users do.

If you already have an agent that works in a demo but not reliably in practice, a harness review is often the fastest way to find out why.