Yesterday my publishing run wrote itself a warning. Two blog posts appeared to have left the sitemap. The count it measured was lower than the count in an earlier record, and the gap was two.
That is exactly the kind of thing you want a system to notice. Posts disappearing silently is a real failure mode, I have written about worse, and it went onto the list as something to check.
I checked it this morning. Nothing was missing. Nothing had ever been missing.
What actually happened
The earlier record was not a measurement. It was a heading.
When the engine planned this coming month, it pulled the live inventory and wrote it into the plan, in the form of a sentence saying how many posts were live followed by the list of them. The sentence said one number. The list underneath it contained a different, smaller one. Both were written in the same file, in the same run, one line apart.
From that point on, the two numbers led separate lives. The list stayed correct, because it was the actual pull. The sentence was wrong, and being a sentence, it read exactly like a fact. A later run compared today's real count against that sentence, found a shortfall, and reported a loss.
This morning I stopped comparing counts and compared the sets instead. Every slug in the list is live right now. Not one is missing. The only difference between then and now is the posts published since, which is what you would hope.
So the alarm was real, the process worked as designed, and the finding was false. The number it was defending was never true.
The part I want to keep
I have spent two weeks writing about monitoring, and this is the third distinct way it has gone wrong for me. The first was no signal at all: a job that did not run and told nobody. The second was a silent monitor: the watcher ran and wrote nothing. This one is different again, and I think it is the one people underrate. The signal fired correctly and pointed at nothing.
A false alarm is not free. It cost me a real check this morning, and it sat on a list for a day looking like a live problem. Worse than the time, it teaches you slowly that the alarm might be wrong, and that lesson is very hard to unlearn once a system has taught it to you a few times. A monitor that cries wolf ends up with the same practical value as one that stays silent, and it takes longer to find out.
The specific mistake underneath it is small and completely generic. I compared a number against a number, where one of them was written by hand and the other was measured. Once a hand written number is sitting in a file, nothing about it looks hand written. It has the same shape as data. The engine had no way to know that one side of its comparison was a claim and the other was an observation.
What I changed
Two things, both narrow.
Compare things, not counts. The check that actually resolved this took a second: two sets, one subtraction each way, and the answer was unambiguous in a way that no count difference ever is. A count tells you that something changed. A set tells you what. If you can get the set, there is no reason to be storing the count at all.
Never write a total by hand next to the list it describes. That is what created the whole problem. A heading that says how many items follow is a second copy of information that already exists, and second copies drift. If a number can be derived from something else in the same file, deriving it is not the fancy option, it is the only one that stays true.
There is a broader version of this that I am less comfortable with. Most of what I record about my own business is written by hand and then read back later as though it were measured. Notes, summaries, the top line of a document, the number I quote in conversation because I remember writing it somewhere. Every one of those is a claim wearing the clothes of a fact. The difference only shows up when something checks it, and almost nothing ever does.
This one got checked because a machine happened to compare it against reality and complain. I do not have that for most of the rest.
The honest scorecard
Three failures, three different lessons, over about two weeks of paying attention: a schedule is not a system, building a monitor is not having one, and now a signal is only worth what its inputs are worth.
None of these are sophisticated problems. That is the uncomfortable pattern. They are all the ordinary consequence of building something quickly, believing it works because it ran, and not going back to check the assumptions it was built on. I keep publishing them because writing this one down took twenty minutes and finding it took a year of not looking.
Today's is at least the good version. The system flagged something, I checked it, and the answer was that everything was fine. I would take that outcome every time over the alternative, which is the same message never arriving.
More build notes in the Journal.