Ask This Article,
No LLM
Inside It

Every article on this site started getting the same kind of comment: a specific, technical question about a specific decision — why Sørensen–Dice instead of a real library, how the size-mismatch guard actually works, what the campaign's real numbers were. Good questions. The only answer mechanism was an email link at the bottom of the page, which meant every one of them either got asked and waited days for a reply, or didn't get asked at all.

The obvious fix is a chatbot. The obvious fix is also the wrong one, for a reason specific to a portfolio: a real LLM behind the widget would happily answer "what do you think about geopolitics" with something plausible-sounding, and the person asking would have no way to tell that the answer came from a model's general knowledge rather than from anything Francis actually wrote. That's not a chat widget. That's a liability wearing a chat widget's clothes.

The actual requirement, stated plainly: answers must come from the article, verbatim or close to it. Questions outside the article's scope must not get an invented answer — they must get an honest "not covered" and a way to reach Francis directly. And the whole thing had to make its own confidence visible, so a reader treats an answer as a scored match rather than as a fact.

Retrieval, Not Generation

The system that shipped has no model in it anywhere. It is a search engine with a threshold. Every case study's content becomes the entire knowledge base for its own widget — read straight out of the rendered page at the moment someone opens the chat, not from some separate file that has to be kept in sync:

// Reads the article's own rendered content straight from the DOM — the
// deck + TL;DR + stat row become a "Summary" chunk, each <h2> section
// becomes its own chunk. This is the entire knowledge base: there is no
// separate authored dataset that can drift from what's actually published.
function extractChunks(articleId) {
  const root = document.getElementById(articleId)
  let current = null
  const chunks = []
  for (const el of root.children) {
    if (el.tagName === "H2") {
      if (current?.parts.length) chunks.push(current)
      current = { heading: el.textContent.trim(), parts: [], anchorId: el.id }
    } else if (current) {
      const text = el.textContent.trim()
      if (text) current.parts.push({ text, displayable: isProse(el) })
    }
  }
  if (current?.parts.length) chunks.push(current)
  return chunks
}

That last part matters more than it looks: the widget has no content to maintain. Edit a paragraph in the article, and the widget's answers change on the next page load, automatically, because there was never a second copy of the text to forget to update. The article is the database.

Scoring: Coverage, Weighted by Rarity

A question gets tokenized, stripped of stopwords, lightly stemmed. Each section of the article gets scored on how much of the question's vocabulary it contains — but not every word counts equally. A word that shows up in every section of an article (the article's own title terms, usually) says almost nothing about which section answers a specific question. A word that shows up in exactly one section is a much stronger signal. That's inverse document frequency, computed fresh for each article from its own sections — no external corpus, no pretrained anything:

// A token found in one section out of nine is a far stronger signal than
// one found in all nine. A query token found in NO section gets the
// maximum weight, so unknown words drag coverage down hard — that is what
// keeps "what database did you use" out when the article never says
// "database".
const weight = (token) => Math.log(1 + sectionCount / (documentFrequency(token) || 0.5))

function scoreChunk(queryTokens, chunk) {
  let unionHits = 0, unionWeight = 0, headingWeight = 0, totalWeight = 0
  for (const t of queryTokens) {
    const w = weight(t)
    totalWeight += w
    const inBody = chunk.bodyTokens.has(t)
    const inHeading = chunk.headingTokens.has(t)
    if (inBody || inHeading) { unionHits++; unionWeight += w }
    if (inHeading) headingWeight += w
  }
  if (unionHits < Math.min(2, queryTokens.size)) return 0
  return Math.min(100, Math.round((unionWeight / totalWeight) * 100 + (headingWeight / totalWeight) * 25))
}

Below a score of 38, the widget doesn't answer — it shows the fallback with Francis's email, pre-filled with the question. Above that floor, three tiers (High, Medium, Low) get surfaced with the answer, in the same HIGH/MEDIUM/LOW language the redirect-matching-engine case study uses for its own confidence tiers — a small, deliberate echo, since it's the same underlying idea: score the overlap, name the confidence, don't pretend a threshold decision is a certainty.

The Bug That Made a Good TL;DR a Liability

The first real failure showed up on this article's own sibling case study, the redirect-matching one. Asking "how does the size mismatch guard work?" returned the Summary chunk — the deck and TL;DR — instead of the section that actually explains the guard. The scoring math wasn't wrong. The TL;DR had been written to preview every mechanism in the article, which meant it legitimately shared vocabulary with the specific section it was previewing. Both chunks scored close to identically, and the Summary chunk, having been inserted first in the array, won every tie.

A well-written summary competing with the section it summarizes is not an edge case — it's what a good summary does. The fix wasn't to make the TL;DR worse. It was to change the order sections get evaluated in, so specific sections get first claim on a tie and the generic overview only wins when it strictly outscores everything else:

// Pushed last, deliberately: a well-written TL;DR previews the whole
// article, so it legitimately shares vocabulary with specific sections it
// summarizes. Scoring breaks ties by first-seen, so Summary must come
// after every specific section, or generic answers beat specific ones.
chunks.push(summaryChunk)

The Bug That Made Literal Questions Fail

The second failure was reported directly: short, literal questions like "What was the CTR?" or "How many visitors?" were falling back to email on articles that plainly answered them. Three separate defects, stacked:

The anti-coincidence guard — the rule that stops one incidental shared word from producing a false match — required two overlapping tokens, flat. A one-word question like "CTR" can only ever produce one hit, so it was mathematically unable to clear its own floor. Digits were being discarded by the minimum-token-length filter, so "phase 2" lost its "2" before scoring ever started. And heading text only added a small bonus rather than counting as real coverage, so a question matching a section's exact title — "what happened in phase 2" against a section literally called "What Phase 2 Did" — still scored below threshold because the body text never repeated the word "phase."

Each fix was small. The guard now scales with question length instead of demanding a flat two. Digits survive the token filter regardless of length. Heading words count toward coverage, not just a bonus on top of it. None of the three would have been caught by reasoning about the code — they only surfaced by generating a deliberately literal question set and checking, per question, whether the article's own words were enough to answer it.

Semantics, Without Crossing Into a Model

The honest ceiling of a pure keyword matcher is paraphrase: a visitor who writes "why not just use a real library" instead of the article's own phrase gets nothing, even though the article answers exactly that. Closing part of that gap without hiring a language model meant staying on the same side of one specific line: every match has to be explainable by pointing at literal, overlapping text. The moment a match can only be explained by an opinion about what two words mean, the system has quietly become a small language model, just a worse one.

Two additions stayed on the right side of that line. A small, hand-curated synonym table — cost/spend/price, visitor/traffic, error/bug, fix/solution — closes specific, known wording gaps without opening a general one; each group is a real same-thing cluster, not a loose thesaurus that could drag an off-topic word into scope. And character-bigram matching lets a typo or a variant spelling still reach the article's vocabulary: "clodflare" resolves to "cloudflare" because most of its letter-pairs literally match, using the same Sørensen–Dice overlap the redirect-matching engine applies to product titles, just one level down, on letters instead of words.

Showing the Arithmetic Instead of Performing Confidence

The last requirement was the most unusual one to build for a chat interface: it should not feel like it's thinking. A typing indicator or a delayed response implies computation happening somewhere hidden, which is exactly the impression a deterministic keyword matcher shouldn't give. What it built into instead is a live preview strip above the input, re-scoring the cached index on every keystroke — cheap enough to run synchronously, so there's no debounce and no spinner:

Type "size," and the strip reads High match · Matching Titles Without a Matching Library. Keep typing toward "size mismatch guard," and it settles on the section that actually covers it. Type something the article never discusses, and it reads No match yet — send anyway to get Francis's email. Watching the tier shift while typing is the honest version of a typing indicator: not "the assistant is thinking," but "here is what your current words would score, right now, against the actual text."

Every answer carries the same number, permanently: a small badge reading something like 58% · Medium match. The badge is the entire point. It reframes the excerpt from "here is the fact" to "here is the best-scoring passage, and here is exactly how well it scored" — which is the honest description of what happened, stated in the interface instead of left implicit.

What Falls Back, and Why That's the Feature

Every regression check that matters is really one check, run against two opposite failure modes. On one article, ten literal, on-topic questions — ad spend, CTR, visitor counts, named people from the text, specific decisions — all have to match their correct section. On another, seven adversarial off-topic questions — geopolitics, an election, cryptocurrency, licensing, a request for a joke — all have to fall through to the email fallback. Loosening the matcher to catch more paraphrase risk breaking the second list. Tightening it to reject more nonsense risks breaking the first. Every fix in this build was checked against both lists together, because a matcher that only gets tested on the questions it's supposed to answer will quietly learn to answer questions it shouldn't.

The widget you're reading this on runs the exact system described above. Ask it something the article covers, and watch the preview strip find it before you finish typing. Ask it something the article doesn't, and it will tell you so, and hand you an email address instead of a guess.