Every team that runs scheduled agents eventually tells the same story. The nightly job failed, someone happened to be watching, they re-ran it by hand, and the morning dashboard showed a complete, on-time artifact. The incident ends with a green checkmark — and that checkmark is where the trouble starts.
The artifact is real. The pipeline "worked." But the system did not produce that result — a person did. If you record only the outcome, those two situations are indistinguishable, and every rescue silently inflates your reliability numbers. Run that loop for a month and you get a fleet that looks increasingly stable while its ability to operate without a human hasn't moved at all.
Rescues erase their own evidence
The core problem is structural: a successful rescue destroys the record of what it rescued. Once the retry lands, the output file, the row counts, the checksums — everything downstream consumes — are identical to what a healthy scheduled run would have produced. "The scheduled run got 1/4 of the data and a human recovered the rest" and "the scheduled run was fine" converge on the same artifact.
This has a nasty corollary for anyone doing a retrospective: you cannot date the onset of a defect from your artifacts. If your history looks clean, it may be because the system was healthy — or because someone was always there. The evidence for the second case is exactly nothing.
Record provenance at the moment of production
The fix is cheap but has to be done at write time, not reconstructed later: every run stamps its own origin into what it produces.
- Origin — was this run
scheduled, an automaticretry, ormanual? - Attempt — first try, or a built-in second attempt?
- Slot — which scheduled window does it belong to? (The moment you run twice a day, "the run" stops being a well-defined noun.)
Two rules make the stamps trustworthy. First, record, don't infer: the wrapper that launches a retry knows it is a retry — let it say so, instead of guessing later from timestamps. Positional inference breaks the day your execution model changes. Second, when provenance is missing, emit null plus a note, never a plausible-looking guess. A wrong-but-reasonable number is strictly worse than an honest "cannot attribute": it will be read as fact by every layer downstream.
Count human interventions — as a lower bound
With provenance in place you can compute the number that actually matters for unattended operation. For each line of work, over a week: how many days was the output produced entirely by scheduled + retry runs (self-sufficient), how many days required a manual run (human-dependent), and how many days is it simply missing?
Read the three numbers together, never one alone. A falling human-intervention count is only good news if self-sufficient days rise and missing days don't — otherwise it may just mean nobody was around to rescue anymore, and failures moved from "saved by a person" to "lost."
And label the metric honestly: it is a lower bound. Origin stamps catch the intervention where a person re-ran the job. They do not catch the person who restarted a daemon, after which the automatic run succeeded. Write that limitation next to the number. A conservative self-sufficiency metric is the useful kind: if the lower bound isn't falling, things are definitely not improving; if it falls, you've earned the right to say "possibly."
A safety net that requires you present is not a net
Sorting our own monitoring against one question — does this still work when nobody is watching? — was uncomfortable. Most of what we called "backstops" turned out to be accelerators: checks that make an incident cheaper to diagnose if someone is there, and do nothing at 3 a.m. The classification we ended up with:
- True backstops ring without anyone present — a freshness gate on the next pipeline stage, an idempotent time-delayed retry, an independent probe that alerts a phone.
- Triage records don't ring, but answer "which line do I debug?" the next morning.
- Presence-dependent checks are worth keeping — and worth refusing to count as protection.
The failure mode this prevents is subtle: a team member who reliably rescues the pipeline is a safety net — until they're asleep, on holiday, or gone. "Someone will do it" is a fact about your roster, not a property of your system. Design reviews should treat the two differently.
Separate “ran and failed” from “never ran”
One more distinction pays for itself the first night you need it. If the morning output is missing, there are two completely different investigations: the job started and died (an engine, prompt, or upstream-API problem) or the job never started (a scheduler problem). If you only log successes, both look like the same blank space.
So log attempts, not just outcomes: a START line when the job launches, an EXIT line with the code when it ends, in a file that survives the run. Then a missing artifact triages itself: START without a clean exit means it ran and failed; no lines at all means the scheduler never fired. Ten minutes of shell script, and the 8 a.m. question changes from "what happened?" to "which of two known things happened?"
What redundancy changes
Adding a second scheduled run per day — the obvious availability move — quietly breaks every check that assumed "one run per day." A partial failure (the evening run died, the morning run's data is present) now looks exactly like success to any check that asks "is there output today?" Redundancy converts total failures into partial failures, and partial failures wear normalcy better. When you add capacity for reliability, upgrade the checks in the same change: from "does output exist" to "did every expected instance appear."
A checklist
- Stamp every run's origin, attempt, and slot at production time.
- When provenance is missing, output null + a note — never a plausible guess.
- Track weekly self-sufficient / human-dependent / missing days per pipeline; read all three together; call the middle one a lower bound.
- Classify every check: rings unattended / triage / presence-dependent. Only the first kind counts as protection.
- Log attempts (START/EXIT), not just successes.
- On any capacity or schedule change, re-verify every check that reads the old execution model — keep a written list of them, with the assumption each one makes.
None of this is glamorous. But "the agent fleet ran for two weeks with zero human interventions, verified from run provenance" is a sentence you can only say if you started recording the right things — and it's the only version of "our agents are reliable" that survives the person who used to rescue them going on holiday.