Operating agentic systems: what maintenance actually means

The project ends, the system does not

Traditional software degrades slowly and visibly: dependencies age, tickets accumulate. An agentic system degrades differently. It keeps returning confident, well-formatted answers while the ground underneath it moves: a supplier changes an invoice layout, a model provider ships a new version, the finance team quietly raises an approval threshold. Nothing throws an exception. The output is simply wrong more often than it was last quarter.

Maintenance is the work of noticing that before anyone else does.

Four things that change

The process. The most common source of drift, and the least technical. Rules change, new exceptions appear, a step moves to another team. The system is still faithfully executing last year's procedure.

The models. Providers release new versions and retire old ones. A newer model is usually better on average and still different in specifics: it may format an extraction differently, call a tool the old one avoided, or become more cautious in a way that raises the delegation rate.

The systems around it. APIs change fields, tighten rate limits, deprecate endpoints. A CRM upgrade on a Saturday is an agent incident on Monday.

The data. Document layouts, naming conventions, volume and seasonality all shift. Retrieval that worked against last year's corpus drifts as the corpus grows.

Evaluations are the load-bearing part

Everything else in maintenance depends on having a test set of real cases with known correct outcomes. Without it, every change is a guess and every rollback is an argument.

A working evaluation set has a few properties:

• Built from production, not invented. Real documents, real tickets, real edge cases.
• Includes failures. Every incident adds a case. This is how a system stops repeating mistakes.
• Covers each skill and each typed error, including the cases that should escalate.
• Runs cheaply enough to run often: before every prompt, skill, tool or model change, and on a schedule regardless.
• Reports per category, not one aggregate score. An average hides the one class of case that broke.

Keep it growing. An evaluation set that has not changed in six months is no longer describing the system you are running.

What to monitor in production

Four families of signal, all of which should be visible on one page:

Signal	Watch for
Outcomes: completed, escalated, failed	Shift in the mix, week over week
Delegation rate by reason	A rising reason means a fixable gap
Human overrides on approval gates	A rejected recommendation is a wrong answer caught in time
Cost and latency per run	Creeping token spend, retries, slow upstreams
Tool error rates by type	Upstream change, credential expiry, new validation rule

The override rate deserves special attention: it is the closest thing to a ground-truth accuracy measure a live system produces for free. If reviewers start rejecting more of what the system proposes, something changed, and the evaluation set has not caught it yet.

Handling a model update

Model changes are routine, not emergencies, if there is a procedure:

1. Run the full evaluation set against the new model, per category.
2. Compare cost and latency, not just quality: the better model may not be worth it for every role.
3. Check tool-calling behaviour specifically, since that is where differences usually show.
4. Roll out per agent role rather than everywhere at once. The orchestrator and a narrow specialist do not need the same decision.
5. Watch the override and delegation rates for two weeks before calling it done.

Keep the previous model configuration available for a rollback. Model routing decided per role, against the evaluation set, is what makes this a controlled change instead of a leap.

A cadence that works

Continuously: monitoring and alerts on failure rates, cost ceilings and stalled runs.

Weekly: review escalations and overrides from the week; cluster them; turn recurring clusters into a skill change, a tool change or a threshold change.

Monthly: full evaluation run; review cost per case against the benchmark the system was built to beat; check for upstream API deprecation notices; add the month's incidents to the test set.

Quarterly: review the process itself with its owner. This is where "we changed that rule in March" surfaces. Review model options, permissions, and whether any part of the system is now doing enough volume to deserve a deterministic pipeline instead of an agent.

Runbooks and ownership

Before a system goes live, write down who is called when it misbehaves and what they do first. At minimum: how to pause the agent team without stopping the business process, how to drain the queue of in-flight cases, how to fall back to manual handling, and how to roll back a skill, tool or model change. Agentic systems fail in ways that are unfamiliar to a standard on-call rotation, so the runbook should include examples of what wrongness looks like, not just how to restart a service.

What maintenance is not

It is not a support contract that waits for tickets. By the time a user reports that an agent team is getting things wrong, the wrong results have been flowing into a system of record for weeks. Maintenance is the ongoing, scheduled work of measuring a system against reality and adjusting it, which is also the work that makes an agent team an asset rather than an expensive pilot.

Frequently asked questions

How much maintenance does a live agent team need?

Less than building it, and more than zero. Expect a recurring weekly review of escalations and overrides, a monthly evaluation run, and a quarterly session with the process owner, plus reactive work when a model or an upstream API changes. The load falls as the evaluation set matures.

Can our own team maintain the system?

Usually yes, and it is the right goal. What they need is the evaluation set, the runbook, documented skills with named owners, and monitoring that surfaces the signals above. The part teams most often lack at handover is the habit of adding every incident back into the test set.

What is the first sign that an agentic system is drifting?

A rising human override rate on approval gates, or a rising delegation rate for one specific reason. Both usually appear before anyone files a complaint, which is why they belong on a dashboard rather than in a monthly report.