On 25 July my publishing automation skipped a day and told nobody. I wrote about it the next day, in a post about how a schedule is not a system. I said that every automation needs three parts and not two: the work, the guardrails, and a watcher that knows whether the work happened at all.
I meant it. I built the watcher that week. It writes one line per run to a log, so that a missing line is detectable, and it runs on its own schedule to check the last several days for gaps.
On 29 July the automation skipped another day.
What I got wrong, precisely
The interesting part is not that it failed twice. Things fail twice. The interesting part is that the second failure had a different cause than the first, and my fix was built for the first one.
On 25 July the machine was closed. Nothing ran, nothing errored, nothing was there to notice. That is a genuine blind spot and the heartbeat log closes it.
On 29 July the machine was open all day. I can prove that, which is the one benefit of having built the logging: another scheduled job on the same machine wrote eight entries across that day, from early morning to late evening. So the environment was available. The publishing run simply did not happen, in the middle of a day when other things did.
And the watcher? The watcher ran. Its own scheduler record shows it executed on 29 July, in the afternoon, hours after the publishing run should have gone out.
It wrote nothing to the log. Not a pass, not a failure, not a line. Its instructions say to always append one line recording the outcome either way. It ran and left no trace of having run.
So the state of things on 29 July was: a job did not run, and the monitor built specifically to catch that ran, and the day passed with no signal to anyone.
What actually caught it was the publishing run this morning, doing its own catch-up check as the first thing it does. The recovery mechanism worked. The detection mechanism, which is the thing I wrote a whole post about, did not.
Two design mistakes, both mine
The monitor reports into a place nobody reads. I built a watcher whose output is a line in a log file. A log file is where you look after you already know something is wrong. Detection has to arrive somewhere a person will encounter it without going looking. Writing to a log and calling it monitoring is the same category of error as writing a note to myself and calling it a reminder.
The monitor shares a failure mode with the thing it monitors. Both live on the same machine, both depend on the same scheduler, both need the same app to be running. A monitor that dies of the same disease as its patient is not redundancy. It is one system that happens to be described in two files. I knew this in the abstract. I did not apply it to my own setup, because when I built it I was thinking about the failure I had just had, not the class of failures it belonged to.
There is a third thing that is smaller but sharper. The watcher deliberately excludes the current day from the window it checks, so that a run still in progress is not reported as missing. Which is reasonable, and it also means a missed day is structurally invisible on the day it happens. The earliest it can notice is tomorrow. I designed a same-day detector that cannot detect anything on the same day, and I did not see it because I was reading my own logic for correctness rather than for what it would do.
The thing I actually want to write down
I published a post about this failure four days before repeating it. There is a version of that which is just embarrassing. The version I think is true is more uncomfortable.
Writing about a problem carefully feels a great deal like solving it. It has the same shape. You examine the failure, you find the root cause, you articulate the lesson, and you finish with a sense of resolution. Everything about the process feels like closure, and the artifact you produce is genuinely good. But an article is an explanation, and an explanation changes nothing on its own.
I did not stop at the article. I built the thing. That is the part I want to be fair to myself about. What I did not do was test whether the thing worked, or ask where its output would land, or check it against the next failure. I treated shipping the monitor as the end of the task. It was the beginning of it.
This is the same mistake, one level up. On 25 July I had a schedule and assumed it ran. On 29 July I had a monitor and assumed it reported. Both times the assumption was that building a mechanism is the same as having a working mechanism. The lesson from the first post applies to the fix for the first post, which is a slightly humbling thing to discover in public.
What changes now
Detection has to leave the building. If a run is missed, the signal has to arrive somewhere I will see without deciding to look, and it has to come from something that is not sitting on the same machine as the job it watches. I would rather have a crude external check that reaches me than a well-written internal one that does not.
The watcher's own silence has to be a detectable event too, which is where this obviously starts to recurse, and the honest answer is that you stop the recursion by escalating outward rather than by adding another layer inward. One external nudge that reaches a human beats three internal layers that all report into the same file.
And I am going to stop treating a written explanation as a completed action. I will keep publishing the failures, because not publishing them makes me worse at seeing them. But the post is the record, not the repair.
The engine published four pieces today instead of two, and yesterday's gap is filled. That part works. The part that is supposed to tell me it did not work is what I am rebuilding.
More build notes in the Journal.