Six days ago I wrote a note to myself telling me not to publish a statistic. This morning I found out the note was wrong. Not the instruction, which was right. The reason, which was rubbish, and which I had written down as though it were a finding.
This is a small thing that I think is actually a large thing.
The note
I keep a running file of which numbers in our marketing copy have been checked and which have not. It exists because in July I found four statistics on our own site that I could not trace to anything, and I did not enjoy that afternoon.
One entry in that file covers a claim you see everywhere: that 77 percent of leads never get a response. On 16 August I added a line to it. The line said, roughly: this one is real but treat it with suspicion, because an older and well-verified study found that 23 percent of companies never responded, and 77 is exactly the complement of 23. That pattern is the signature of somebody inverting a figure and restating it as a new finding.
I was quite pleased with that. It felt like the kind of thing a careful person notices.
This morning
Today's business article is about response times, so I went back to that entry to settle it properly rather than leaving it flagged for another month. That meant finding the actual source instead of reasoning about it.
It took about ten minutes. The figure comes from a company announcement published in February 2021, covering a dataset of 14,000 companies and 55 million sales interactions. It is real research. It has nothing whatsoever to do with the older study I had connected it to.
And the arithmetic coincidence I had been so pleased with is a coincidence. Both studies produce a 23. One means 23 percent of companies never replied. The other means 23 percent of leads get picked up by a salesperson. Same digits, opposite meanings, fourteen years apart, unrelated datasets. My clever pattern was two unrelated things landing on the same number, which happens, because there are only a hundred of them.
The part that bothers me
The conclusion in my note was correct. That statistic should not go in our copy as it is normally written, and today's article says so.
But it is correct for a reason I had not found. The real problem is that the source states the figure conditionally, for teams who lead with marketing automation, and the version circulating online drops the condition and applies it to everyone. That is a much more ordinary failure than the one I invented, and it is also much easier to check, because it is right there in the first sentence of the announcement.
So for six days I was holding the right position for the wrong reason. If anyone had pushed back on it, I would have defended it with an argument that does not survive ten minutes of looking. And I would have defended it confidently, because it was written down in my own file, in my own handwriting, in the section marked verified.
What I actually got wrong
Not the fact. The category.
There is a difference between "I checked this and here is what the source says" and "I reasoned about this and it seems off". Both of those are useful. Only one of them is a finding. I had filed the second as the first, and once something is in the file it stops being re-examined, because the entire point of the file is that I do not have to keep re-deciding things.
That is the actual defect. A note that records a suspicion is doing its job. A note that records a suspicion in the voice of a conclusion quietly turns into a fact by sitting still. Six days is nothing. Six months and I would have been repeating it in meetings.
I have gone back through the file and split it. Every entry now says either what the source says, with a link, or what I suspect and have not yet checked. There are more in the second category than I would like.
The bit I am not going to pretend about
I have written before about a good source being wrong and not feeling wrong at all. That post was about somebody else's document being stale.
This one is worse and I would rather say so plainly. The unreliable document was mine. It was written specifically to protect me from this class of error, six weeks after I had built the habit of checking primary sources, by a person who at that moment was doing exactly the thing he was building the file to prevent.
The system worked anyway, but not because of the note. It worked because today's article needed a citation and a citation forces you to open the document. If I had needed a summary instead of a source, I would never have found out.
So the honest lesson is not "be more careful". I was being careful. It is that the only thing that reliably catches this is having to produce the link. Everything else is confidence.
Today's article is the version with the links in it, including the two studies that hold up and the one that has no source at all.
Frequently Asked Questions
Why not just delete the note and move on?
Because the note was doing something useful, badly. Deleting it loses the flag. Keeping it as written keeps a bad inference in the record. Splitting it into what is verified and what is suspected keeps both the caution and the honesty.
Does this mean the statistic is fine to use?
No. The conclusion has not changed. The figure is real but conditional, and the version people repeat has dropped the condition. It stays out of our copy, now for a reason that traces to the source rather than to my arithmetic.
How long did the actual check take?
About ten minutes, including finding the original announcement and reading it. That is the uncomfortable part. It was not a hard verification I had been putting off. It was a cheap one I never started, because I thought I had already resolved it.