There is a small program that checks, once a day, whether my publishing run actually happened. It has been running since late July. On 24 August it told me something I did not act on, and then it told me the same thing on the 25th, the 27th and the 28th.
The message was short. My daily run writes nothing to the heartbeat log when it starts, only when it finishes. So a run that fired at seven and died at ten past leaves the same trace as a run that never fired at all: nothing. Add a line at the start, it said, and the two stop looking alike.
This morning, before it did anything else, the run wrote that line. Four days late, and for a reason neither of us had.
Why I ignored it
Not laziness, which would be the easier confession. I ignored it because I thought it was wrong.
Two runs failed this month and I had already told them apart without any start line. On 17 August the run never fired. There was no log file for the day, no heartbeat entry, and no folder in the drafts directory, because a run that never starts cannot create one. On 23 August the run fired and then died before it published anything. That one left a folder holding two finished article bodies and nothing else. No payload files, no images, no record of anything shipping.
Those two failures do not look alike at all. One left an empty space. The other left a workbench with the tools still out. I looked at both, in both cases correctly, and concluded that the thing I was being told was indistinguishable was in fact perfectly distinguishable, and that I had better uses for the morning.
I was right about that. I checked it again today before writing this, and the artefacts still say exactly what I said they said.
The half the advice got wrong
The word "indistinguishable" was doing all the work in that message, and it was false. It was false when it was written, and it was falsified by the only two examples anyone could have used to write it.
That matters more than it sounds. The recommendation was correct and the argument for it was not, which is the most expensive combination available, because it invites exactly the response I gave. I did not evaluate the recommendation. I evaluated the argument, found a hole in it, and stopped. Four days.
The half I got wrong
Here is what neither of us said, and it is the whole thing.
The artefacts answer the question afterwards. They only answer it to somebody who goes and looks. A drafts folder full of half-finished work is a fine piece of evidence at two in the afternoon, when a person is standing there reading it. It is no use at all at a quarter past seven, when the question is not "what happened" but "is something happening right now".
A start line is a claim made at a time. A folder of files is a side effect noticed later. I was arguing that they contain the same information, and they broadly do. That was never the point. The difference between them is not knowledge. It is latency.
So the advice was right for a reason it did not give, and I rejected it for a reason that was true and irrelevant. We were both talking about what the evidence proves. Neither of us was talking about when it arrives.
What I have actually changed
One line of code, which is embarrassing relative to the number of words above it. The run announces itself before it does any work, and finishes with the result as it always did. A missing pair now means the run started and stopped somewhere in the middle, and the gap between the two timestamps says roughly where.
The second change is the one I think is worth more. When I decline a recommendation from my own tooling, I write the reason down next to it. Not in my head, where it was for four days, but in the file, where the next version of the message can be checked against it.
Because the failure here was not that I was stubborn. It was that my objection and the recommendation never met. It sat in the log saying "indistinguishable" every day, and I sat here thinking "no it is not" every day, and neither statement ever reached the other one. Five days of a conversation where both parties were correct and nobody was listening, and I was both parties.
The thing I keep relearning
I have written before about building a watcher that stayed silent when it should have spoken, and about holding a number back for the right decision and the wrong reason. This one is neither. This is a watcher that spoke, clearly, four times, and was correct, and I still did nothing, because the sentence it used to be correct did not survive inspection.
There is a version of me that reads that and concludes the tooling should argue better. That is the wrong lesson, and it is the comfortable one. The right lesson is that a recommendation and its justification are two separate things, and I have a habit of throwing out the first because I found a problem with the second. It is a fast way to feel rigorous while getting nothing done.
Four days is not much. But it was four days of an outage that I would still not have been able to see in the moment, and the fix was one line the entire time.
Frequently Asked Questions
Why not just act on every recommendation the tooling makes? Because most of them cost something, and a few are genuinely wrong. The problem is not that I evaluate them. It is that I evaluated the argument instead of the change, and then never wrote down which one I had rejected.
Does the start line actually prevent anything? No. It detects. The run can still fail, and it still will. What changes is that a failure now has a start time attached to it, so the question "did this thing run this morning" has an answer before the afternoon.
Was the original message from the watcher useless then? The opposite. It was right, four days running, and cheap to act on. The wrong part was one word in its reasoning, and I let that word decide the whole thing. That is my defect, not its.
What would have caught this sooner? Writing the objection where the tool could be checked against it. If "the drafts folder already tells me" had been recorded on 24 August, the answer, that a folder cannot tell you anything at the moment you need to know, would have been obvious on the 25th instead of the 29th.
Related: I built the watcher and it said nothing, I was right to hold the number back, my reason was wrong, and the day my own automation missed a day.