Three things our own SEO data told us that weren’t true
In one week of diagnosing our own site we wrote down three confident conclusions. Each was built on a real number, and each was wrong — and so was one of our corrections. Here are the numbers, why they misled us, and the check that would have caught each one.
We make Clew, a tool for product tours and onboarding. In early November we sat down to find out why our two sites get so little search traffic, and in the first week we wrote three conclusions into our notes with full confidence. Within days each one turned out to be wrong.
None of them came from a bad tool or a typo. Each started from a number that was real and correctly reported. What went wrong was the step from the number to the conclusion, and each time it went wrong the same way: we looked at the cases that agreed with us and not at the ones that could have disagreed.
Each of the three would have sent us to do the wrong work. So here they are, with our own numbers.
1. “Our pages rank in the top 10”
Search Console reports an average position for every page. Sort our English site by that number and 149 pages sat at positions 4 to 10. That reads like a site one push away from traffic.
It was the opposite. A per-page average gives a query someone searched once the same weight as a query searched a hundred times. We had 229 queries with one or two impressions each, many of them at decent positions for phrases nobody else types, and they pulled the average up. Weight the same data by impressions — where people actually search — and the picture changes:
weighted position = Σ(position × impressions) / Σ(impressions)That gave us 24.3: page three, not the top 10. And it was still wrong, because of where most of the impressions came from.
Sixty percent of our impressions were for searches containing “clew”, mostly “clew.io alternatives” (601 impressions, position 3) and “clew.io” (119, position 5.8). Clew.io is a different company, a risk and compliance platform. These were people looking for alternatives to someone else’s product, and Google showed them ours because the name matched. We sat near the top for them, and not one of the 752 clicked. Nothing we could change on our site would make them click.
Take those out and what is left is 505 impressions, a weighted position of 54.2, and zero clicks in 28 days. Page six. That is the number we should have been steering by from the start, and it calls for a different plan from “one push from the top 10”.
2. “We have no backlinks”
Yandex Webmaster lists sample external links for a site. For our two domains it showed 96 and 84 of them, and every sample was one of our two sites linking to the other. Yandex’s own site quality score was zero. We wrote it down: no third-party links at all. The plan that followed was to get our first ones by submitting to software directories, as many as possible.
Before submitting anything we pulled the full list of referring domains from Semrush, to have a baseline. It listed 100 domains for the English site. Two were ours. Ninety-eight were not, and we had asked for none of them.
Two of the 98 are real: Product Hunt, where we launched, and ProductRadar, a Russian launch directory. The rest are generic “tech news” blogs with names like freedom251, laptop251 and techbloat, plus domains such as ku3933net22.click. Most have a Semrush authority score under 10. A new domain does not start with zero links. It starts with whatever link farms and scrapers find it first, and in our case that was 66 domains in a single month.
This changed the plan more than it changed the count. A profile that is mostly link-farm domains is the worst possible place to add a hundred more low-quality directory listings, which is exactly what the mass “submit to 140 directories” packages sell. The directories still worth doing are the dozen that people buying software actually read. And the two tools disagreeing is itself the finding: Yandex showed us only links it apparently counts, while Semrush listed everything that links. A quality score of zero next to 98 referring domains suggests Yandex is ignoring them, though nothing it publishes confirms that.
3. “Yandex recrawls our pages a minute after IndexNow” — and then “it doesn’t”
This one we got wrong twice, in opposite directions.
Yandex’s recrawl API was returning a 504 for every request, and its index statistics had not updated in a month. Then our server log showed YandexBot fetching nine pages at 23:47 one night, about a minute after a deploy finished. Our app submits changed pages to IndexNow a minute after each start, and the log for that start said 35 pages had gone out. Deploy, submit, crawl within the minute. We wrote it down as the recrawl path that works.
The next day we did the right thing and checked every deploy instead of the one that fit. Two more Yandex visits had arrived just after deploys that changed nothing but documentation, and a deploy like that, we reasoned, submits nothing. So the visits had nothing to do with IndexNow, and we wrote a correction: coincidence.
The correction rested on an assumption we had not checked. The log line that says what IndexNow sent is lost at every redeploy, but our app rewrites its IndexNow state file only after a successful submission, so the file’s timestamp survives. It read 09:24:55 UTC — the docs-only deploy had submitted after all.
The reason was a bug. Our IndexNow hashed each page including the “updated” date the page displays. An edit changes a page at one deploy; from then on the page shows a new date; and at the next deploy, of anything at all, the date makes the hash differ and the same pages go out again. Our SEO registry had always ignored that date when deciding whether a page changed. A comment in its code even says it does so “as IndexNow does”. IndexNow didn’t. It does now.
With the echo understood, the record reads differently:
- A deploy that submitted 35 pages: Yandex within a minute.
- A docs-only deploy that resubmitted them (the echo): Yandex within a minute.
- A docs-only deploy whose log says “173 urls, nothing changed”: no Yandex.
- Four deploys in five minutes one morning, submissions unknown: Yandex twice inside that window.
- No deploy nearby: Yandex comes sometimes anyway.
That is consistent with IndexNow bringing Yandex within a minute: two out of two where we know something was sent, and no visit in the one case where we know nothing was. It is also five data points. We announced a discovery on the strength of one event, then reversed it on the strength of an assumption, and the honest position is in between: plausible, not tested. The test is cheap. Deploy a change to one page, watch the log for an hour, and repeat it three times.
Postscript, the same night. Fixing the echo gave us a prediction to test. The fix changes how every page with a date is hashed, so on the first boot after it IndexNow would resubmit all of those pages once. We wrote that down before deploying. The deploy finished at 23:24; the state file was rewritten at 23:25:05; in the same minute YandexBot fetched 21 pages, and one more at 23:26. In the ten minutes before the submission, while the server was restarting, it fetched none.
Publishing this postscript was the next test: it changes only the English site, so the Russian one should submit nothing and Yandex should stay away. It submitted nothing, and Yandex stayed away. That makes three submissions out of three followed by Yandex within a minute, and two restarts without a submission followed by nothing. It is still a small sample. But this time each prediction came before its result, which is the one thing the first two rounds never had.
What the three have in common
Every number above was correct. The average position was the average; the samples were the samples; YandexBot did arrive at 23:47. The conclusions failed because each was built from the cases that agreed with it: the pages whose average looked good, the sample that showed only our own links, the deploy that was followed by a crawl. The fourth, the correction, failed differently: it was built on something we assumed the log would say, and never looked up.
This is the same mistake as calling an A/B test after one good day, and we know that because guarding against it is part of what we build: when you A/B test a tour in Clew, it plans the sample size up front and shows whether the lift is significant before you make B the main version. We had not applied that habit to our own search data until it had cost us a week of planning in the wrong direction.
One week of diagnosis, three confident conclusions, three wrong, and one correction wrong as well. The fixes were all cheap: weight by impressions, read the whole list, count the cases that disagree, look up the thing you are assuming. None takes more than an hour. Skipping them nearly cost us a directory campaign that would have made our link profile worse, and a recrawl plan that swung from “proven” to “coincidence” and back on five data points. It also turned up a real bug, which is the most useful thing a wrong conclusion has done for us so far.