A Redirect Engine
for a Catalog
Behind Cloudflare

Every discontinued product on an ecommerce catalog leaves a dead URL behind — a page that used to rank, used to carry backlinks, and now returns a 404. On a catalog running somewhere north of a hundred thousand live products, that's not a handful of broken links. It's thousands of URLs quietly bleeding equity, and at scale enough of them start to look to Google like a soft-404 pattern rather than a one-off.

The fix is obvious in principle: redirect each dead URL to its closest living replacement. The hard part is that a wrong redirect is worse than no redirect at all — a 410 tells Google and the customer "this is actually gone," while a bad 301 tells them "here's what you wanted" when it isn't. Get that wrong at scale and you're not fixing 404s, you're manufacturing soft-404s with extra steps.

This is the story of building the tool that solved that, and the two things about it that turned out to matter more than the matching algorithm itself.

The First Wall: The Site Is Behind Cloudflare

My first instinct was the obvious one — scrape the sitemap, scrape the dead-product list, diff them, done in an afternoon. That plan died the first time I actually tried it against the live site. Cloudflare doesn't block a scraper immediately, which is exactly what makes it dangerous: a plain curl with a normal browser User-Agent sails through with clean 200s for the first few thousand requests. Then, once the traffic pattern reads as non-human, it escalates — 429s start showing up on sensitive paths first (/search*, /sitemap*), and if you keep going, the whole site goes behind a 403 with cf-mitigated: challenge. At that point there's no more scraping to be done from that environment, full stop, and anything not already saved is gone.

So the environment doing the fetching had to be a normal browser session, not impersonate one. That meant the whole thing had to run as a script pasted into a DevTools console on a tab the user is already logged into — same origin, same session, same IP a human is sitting behind. No separate scraping infrastructure at all.

That decision had a consequence I didn't fully appreciate until I hit it: a script that has to be pasted whole into a console can't import anything. No bundler, no npm install, no fuzzy-matching library sitting one require() away. Whatever did the actual string matching had to be implementable in plain JavaScript, fast enough to run synchronously in a tab someone's actively using, with zero external dependencies.

Matching Titles Without a Matching Library

That constraint ruled out the "proper" answer — something like a TF-IDF cosine similarity or an edit-distance metric such as Levenshtein or Jaro-Winkler — because none of those ship in a browser for free, and re-implementing one from scratch for a console-paste script felt like solving the wrong problem. What did fit was Sørensen–Dice token overlap: tokenize both titles, treat them as sets, score on overlap. It's a handful of lines, it's order-insensitive (so a title that got reworded slightly between the dead listing and the live one still matches), and it needs nothing but the language's built-in Set:

function score(aTok, bTok) {
  if (!aTok.length || !bTok.length) return 0;
  const bSet = new Set(bTok);
  let hits = 0;
  for (const t of new Set(aTok)) if (bSet.has(t)) hits++;
  return Math.round((2 * hits) / (new Set(aTok).size + bSet.size) * 100);
}

Scores get bucketed into HIGH (≥90), MEDIUM (≥70), LOW (≥50), or NO_MATCH — no redirect target emitted below that floor, because a guess with no real signal behind it is worse than an honest dead end.

Worth noting: a second implementation exists in the same toolkit — a Python script for cases where both the dead list and a catalog export already sit on disk, so there's no live-site fetching and the console-paste constraint doesn't apply. That one uses rapidfuzz's token_sort_ratio, a real edit-distance-based metric, because there's nothing stopping it from pulling in a proper library. Same thresholds, same downstream logic, genuinely different matching math underneath — a fairly clean illustration of how the deployment constraint, not a preference for one algorithm over another, is what actually decided this.

Making It Fast Enough to Run in a Tab Someone's Watching

Comparing every dead product against every live product directly is O(D × C) — with a catalog in the tens of thousands, that's tens of millions of comparisons run synchronously in a browser tab, which is not a real option. The fix is a standard inverted index, built once: tokenize every catalog title, map each token to the list of products containing it.

const index = new Map();
catalog.forEach((c, i) => {
  c._tok = toks(c.title);
  for (const t of new Set(c._tok)) {
    if (!index.has(t)) index.set(t, []);
    index.get(t).push(i);
  }
});

Then, for each dead product, only candidates sharing at least one token get scored at all — pulled straight out of the index rather than scanned for:

const counts = new Map();
for (const t of new Set(qTok)) {
  const posting = index.get(t);
  if (!posting || posting.length > 5000) continue;   // skip ultra-common tokens
  for (const i of posting) counts.set(i, (counts.get(i) || 0) + 1);
}

The > 5000 cutoff exists because without it, a token like a common brand name would pull in a huge fraction of the catalog as "candidates" for every single query, and the whole point of the index — avoiding brute force — quietly erodes. The tradeoff is real: a legitimate match sharing only a high-frequency token with its correct target could get missed. It didn't show up as a problem in review, because product titles carry enough distinguishing detail (brand plus product line plus descriptor) that this rarely comes down to one common word — but it's the kind of tradeoff that would need revisiting on a catalog with thinner, more generic titles.

The Bug That Would Have Shipped a Genuinely Bad Redirect

Title similarity alone is blind to quantity, and this is where it actually bit. "Kenco Smooth Instant Coffee Granules 750g" and "Kenco Smooth Instant Coffee Granules 6 x 750g" share every word but one. Pure token overlap scores that at 92 — comfortably HIGH confidence, same brand, same product line. It's also wrong: one's a single jar, the other's a six-pack at six times the price. Redirecting someone from a discontinued single jar into that listing isn't a fixed link, it's a bait-and-switch with a green checkmark next to it.

The guard against this extracts size tokens from both titles and drops confidence a tier whenever both sides carry size info and none of it overlaps:

const SIZE_RE = /\d+(?:\.\d+)?\s*x\s*\d+(?:\.\d+)?\s*(?:g|kg|ml|l)\b|\d+(?:\.\d+)?\s*(?:g|kg|ml|l|cl|ct|pack)\b/gi;
const sizeClash = (a, b) => {
  const A = sizes(a), B = sizes(b);
  if (!A.size || !B.size) return false;
  for (const s of A) if (B.has(s)) return false;
  return true;
};

The Kenco pair got caught exactly as intended — downgraded from HIGH to MEDIUM, flagged for review rather than auto-approved. On review, most of the size-mismatch flags this guard threw were genuine — a real pack/quantity clash — but a chunk of them weren't, and that's the more interesting finding. A dead-product name like "Robinsons … Squash 1 75l" — de-slugified from a URL, so the decimal point is gone — tokenizes as two separate values, 1 and 75l. The live title, "…Squash 1.75L," tokenizes as one: 1.75l. Same product, same size, flagged as a clash anyway, because nothing in the regex reunifies a stray space where a decimal point belongs. That's not a matching-algorithm failure so much as a data-cleanliness gap upstream of it — and it's the kind of thing you only find by actually reading the flagged rows rather than trusting the tally.

Why Nothing Gets Imported Without a Human Looking First

The tool emits two files, on purpose, for two different audiences. shopify_redirects.csv is exactly the two columns Shopify's bulk importer wants — Redirect from, Redirect to — nothing else. redirect_detail_review.csv carries the same rows plus confidence, score, the size-mismatch flag, and both titles, sorted worst-confidence-first, so review time goes straight to the risky rows instead of re-confirming matches that were already obviously right.

That second file is the actual design decision, not a nice-to-have. HIGH-confidence matches went in with light spot-checking. Everything below HIGH — the majority of what matched — got looked at by a person before it touched the importer. The alternative, auto-importing everything the matcher produced, trades that review time for risk sitting silently in production, the kind that only surfaces when a customer lands somewhere wrong or a page's rankings quietly drop for no obvious reason. Dead URLs with no candidate clearing even the LOW floor didn't get a guessed redirect at all — they got routed to an honest 410 instead, which is a materially different outcome from a 301 and shouldn't get described as the same kind of fix.

Two Things That Changed After This First Version

The tool evolved past what's described above in two ways worth naming separately, rather than folding them into the story as if they were there from the start. It originally only accepted a JSONL dead-list; a later pass added a hand-rolled CSV parser handling quoted fields, embedded commas, and escaped quotes, with case-insensitive header detection, so a dead list doesn't need reformatting before it can be used. And matching was originally products-only; a later revision auto-detects /products/ vs /collections/ per row from the path prefix, so a mixed list of discontinued products and merged collections can run in one pass without ever matching a dead product against a live collection by mistake.

What I'd Still Fix

The size-token false positives are a known, specific gap — a normalization pass that turns "1 75l" into "1.75l" before extraction would close most of it, and it isn't hard to write. The bigger gap is structural rather than algorithmic: this was built and run without any before/after instrumentation planned ahead of time — no snapshot of 404 volume or affected URLs' traffic taken before the redirects went live. That's a bigger miss than anything in the matching logic itself, and the fix is the boring, obvious one — which should be standard practice for the next batch rather than a lesson that needs re-learning each time this runs.