Yesterday my publishing system did not run. Nobody noticed until the afternoon, and the thing that noticed was not me.
This is a post about what I automated instead of making a first hire, so it would be dishonest to write it on a day the automation worked. It did not work yesterday. That is the more useful version of the story anyway.
What I handed over
When it became clear I could not do everything myself, the normal move was to hire someone. I did not. I wrote systems for the parts that were mechanical and kept the parts that were not.
What actually went to software, in the order I did it:
The publishing. Two posts a day, written, checked against a list of rules that must never be broken, rendered with their own images, published, and shared. This is the biggest one. It is genuinely a job.
The checking. Everything published goes through a gate that refuses copy containing certain things: a project that is not public, a suburb name, dashes I do not use, our own prices, a link promised without a real destination behind it. The gate blocks. It does not warn and continue.
The remembering. Follow-ups, the invoice numbering sequence, which client is waiting on what. Not the deciding, the remembering.
The watching. A separate job that looks at whether the other jobs ran, because a system that fails by not starting leaves no error behind it.
That last one exists because of a specific day in July when the run simply did not happen. Nothing broke. The machine was closed through the morning, the scheduled slot passed, and nothing was there to catch it. I wrote about it at the time in the day my own automation missed a day.
And then it happened again
Yesterday, 17 August. No run. No log file, no draft folder, no line in the heartbeat. The newest thing on either site was still the previous day's pair.
The watchdog did its job exactly as designed. It fired in the afternoon, checked the window, and reported that the marketing run had not fired that day, about nine and a half hours past its usual slot, with nothing published. It was right, it was specific, and it was clear.
Then it stopped. Because that is what I built it to do.
I even wrote the reasoning down when I built it: the watchdog alerts, it does not republish. Content recovery is handled by the daily run's own catch-up step. Which is true. The catch-up step works. It ran this morning and backfilled yesterday's two posts alongside today's two.
But look at the actual sequence. The system failed at dawn. It was detected mid-afternoon. It was recovered the following morning. In between, the recovery depended entirely on the next scheduled run happening, which is the exact thing that had just demonstrated it does not always happen.
The distinction I had not made
Here is what I think I got wrong, and it is not a bug.
I had been treating "automate it" and "hire someone for it" as two routes to the same outcome. They are not, and the difference is not reliability. My systems are more reliable than a person at the mechanical parts. They do not get tired, they do not skip the boring check, and they have never once forgotten the invoice numbering rule.
The difference is what happens at the edge of the job.
A person who cannot do the work tells you. They do not simply not arrive and leave no trace. And when something adjacent to their job is broken, they notice, because they have context that extends past the task description. The context is the thing I did not automate and mostly cannot.
So what I actually automated was the work. What I did not automate, and had somehow assumed was included, was the responsibility for the work happening at all. That stayed with me the entire time. I just stopped looking at it, because for weeks the thing kept running and I got used to it running.
I built a watchdog and then treated the watchdog as if it had closed the loop. It closed half of it. It converts a silent failure into a visible one, which is genuinely valuable and is the harder half. It does not convert a visible failure into a fixed one. I knew that. I wrote it down. I still behaved as though the loop was closed.
What I am changing, and what I am not
Not changing: the decision itself. I would make it again. The mechanical work genuinely is mechanical, and handing it to a system rather than a person was the right call for where the business is. On the days it runs, it does the job well, and it does it identically every time, which a person would not.
Not changing either: the watchdog staying an alerter rather than a fixer. A watchdog that republishes is a watchdog that can double publish, and I would rather have a gap than two of everything.
What I am changing is smaller and more specific. The gap yesterday was not detection and it was not recovery. Both of those existed and both worked. The gap was that recovery waited for the next scheduled run, so a failure to start could only be repaired by the same mechanism that had just failed to start. The watchdog already knows the run is missing, in the afternoon, hours before the next slot. That is the moment the catch-up should be triggered from, rather than the following dawn.
The honest summary
I did not replace a hire. I moved the boring parts of a job into software and kept the whole of the responsibility, and then quietly forgot that I had kept it.
That is fine. It is workable, and for a business this size it is probably correct. But the story I had been telling myself, that the system handles the publishing now, was never quite true. The system does the publishing. Making sure the publishing happens is still my job, and yesterday is what that job looks like when I am not doing it.
The automation did not fail me. It did exactly what it was written to do, including the part where it stopped.