This is a build-log, the kind I write while I am still annoyed enough to remember the detail. It is about one bug that hid our own website from Google for longer than I want to admit, and about why it was almost impossible for me to see. I already wrote the wider story of the SEO mess in fixing the SEO bugs that hid our site. This is the one that took me the longest to understand, on its own, because it is a genuinely sneaky one.
The symptom
We run two sites. A South African one and a global one. Different domains, on purpose, because a business owner searching from here should land somewhere that speaks to them, and someone abroad should land on the global version.
So I put in a geo-redirect. Simple idea. If a visitor is outside South Africa, send them to the global site. If they are inside South Africa, they stay on the local one. It worked perfectly. I tested it myself, from my desk, over and over. Local every time. Ship it.
Then the rankings for the South African site would not move. Not slowly climbing. Flat. And when I pulled the site up in Google's own inspection tools, the picture made no sense. It kept showing me the wrong site, as if the local domain were pointing at the global one. I stared at that for days convinced it was a caching ghost.
The thing I could not see
Here is the trap, and it is a good one.
Google does not crawl your site from your city. Its crawler mostly comes from outside South Africa. So every time Googlebot arrived at the local domain, my clever geo-redirect looked at it, saw a visitor from outside the country, and did exactly what I built it to do. It bounced the crawler to the global site.
Google was never being shown the South African site. It was being redirected away from it every single time, and then reporting back to me what it actually saw, which was the other domain. The tool was not lying. I was.
And I could not reproduce it, because I was sitting in the exact country that was the one exception to the rule. The redirect was invisible to me by design. I had built a door that opened for everyone except the person standing in the room testing it.
Why the crawler check did not save me
My first instinct, once the penny dropped, was: fine, I will just let search-engine crawlers through. Detect the bot, skip the redirect, done.
That is the right idea and it is also where I got bitten a second time. Crawler detection works off the user-agent string, the little label a visitor's browser sends to identify itself. Real search bots announce themselves. But the list of bots is long, it changes, and every testing tool, preview fetcher and link checker sends its own label. My detection caught the obvious ones and missed others. So some legitimate crawlers got waved through and some got bounced, which produced the maddening result where the site looked fine one day and broken the next depending on which tool I used to check.
The lesson landed hard: a rule that branches on where a visitor is or what they claim to be is a rule you personally will almost never trigger. That makes it nearly untestable from your own chair. You are the least representative visitor your own site has.
How I actually found it
Not by being clever. By stopping trying to reproduce it as me.
I made myself request the site the way the crawler does: from outside the country, wearing the crawler's identity instead of my own. The moment I did that, the redirect fired in front of me, plain as day, and the whole thing became obvious in about ten seconds. The bug was never hard. Seeing it was hard, because every default I had put me on the one path where it did not exist.
The fix itself was small. Let verified search crawlers reach the local site regardless of where they appear to come from, and be far more careful about which visitor actually gets redirected. Small fix, days of blindness in front of it.
What I would tell someone facing the same thing
If you build anything that behaves differently based on who is visiting, geography, device, user-agent, login state, then you are, by definition, not a fair test of it. Your normal, everyday view of your own product is the one view guaranteed to hide the bug. So test as the visitor who is affected, not as yourself. Request it from where they are. Wear the label they wear.
And when a trustworthy tool keeps showing you something that makes no sense, sit with the possibility that it is telling the truth and you are the one with the wrong vantage point. Google was reporting exactly what it saw. It just was never allowed to see what I thought it was seeing.
The wider clean-up, and what it did to our visibility, is in the SEO bugs that hid our site. And if you want the moment I decided to point the tools at my own business in the first place, that is in the morning I audited my own business.
Frequently Asked Questions
What is a geo-redirect? A rule on a website that sends visitors to different pages or domains based on the country they are browsing from. It is useful for serving local content, but it treats search-engine crawlers as visitors too, which is where it can go wrong.
Why did the geo-redirect hurt Google rankings? Google's crawler usually visits from outside the site's home country. The redirect saw it as a foreign visitor and bounced it to a different domain, so Google was never shown the local site it was supposed to index.
Why was the bug so hard to find? The redirect only triggered for visitors outside South Africa, and I was testing from inside it. My own view was the single exception to the rule, so the broken behaviour never appeared for me.
How do you test a redirect that depends on location? Request the site the way the affected visitor or crawler does: from outside the region and using the crawler's own identity, rather than testing from your own browser in your own country.
More of the build in public at heinoux.nexbdm.co.za.