Drift sentinels — detecting when an LLM stops citing you

An LLM's answer for the same query, run a week apart, is rarely identical. The model has not changed (between training refreshes), but the retrieval layer has, the temperature jitter has, the prompt sampling has. The answer drifts. Most of the drift is noise. A small fraction is signal — your firm has slipped out of the answer set, your sentiment has flipped, a competitor has displaced you.

Drift sentinels are the math that turn continuous answer streams into actionable alerts without drowning the user in noise. This is how we think about that problem.

The naive approach (and why it fails)

The naive approach: hash the answer text, alert when the hash changes. This produces an alert on every probe — the model rephrases something every single time.

Slightly less naive: compute string similarity (Levenshtein or Jaccard) between consecutive answers, alert when similarity drops below a threshold. This produces alerts on every non-trivial rewording. Still useless.

The right unit of analysis is not the answer text. It is the firm's representation within the answer.

The drift sentinel math

For each (engine, prompt, timestamp), we compute a firm representation vector:

• included ∈ {0, 1} — whether the firm is named in the answer.
• position ∈ [0, 1] — normalized rank within named firms.
• sentiment ∈ [-1, +1] — classifier output.
• citation_count ∈ ℤ ≥ 0 — citation references to firm sources.

Drift is the L1 distance between consecutive vectors (or, more practically, between the current vector and a rolling 7-snapshot baseline median). Drift alerts fire when this distance exceeds a per-workspace threshold.

Severity gradient

We classify each drift event into three severity levels:

• Informational — small position shift, no sentiment change. Logged but not alerted.
• Watch — meaningful position change or sentiment change in the watch range. Surfaced in the next weekly digest.
• Critical — firm dropped out of the answer, sentiment flipped to negative, or a competitor displaced you in the lead position. Real-time alert.

Each level has a different alerting policy: informational goes nowhere, watch goes to email digest, critical goes to Slack + email + (Counsel+) MS Teams + (Enterprise) PagerDuty.

The false-positive problem

The biggest engineering challenge with drift sentinels is false-positive suppression. Two patterns dominate the false-positive surface:

Temperature noise

Same engine, same prompt, two probes 30 seconds apart can produce different answers due to sampling temperature. We suppress this with a 3-of-5 confirmation policy: a drift event only escalates to alert level after the new state is seen in at least 3 of 5 consecutive probes.

Synonym drift

The model rewords your firm description without changing meaning. "leading IP boutique" becomes "prominent intellectual property specialist." This trips naive sentiment classifiers. We use a contextual sentiment classifier that scores against a baseline of acceptable rewordings, so synonym drift inside the acceptable envelope does not fire.

The time-to-alert target

For Counsel-tier engagements, we target a sub-10-minute median time-to-alert from drift event to user notification. The budget breakdown:

• Probe cadence: 30 minutes per (engine, prompt). Expected detection latency: ~15 minutes (mean of uniform distribution).
• Confirmation policy: 3-of-5 probes. Expected confirmation latency: ~5 minutes (on the 30-min cadence, but probes are spread across engines).
• Classification + alert routing: under 30 seconds.

Composite: ~20 minutes from drift event to alert in the worst case, with a target of 8–10 minutes for the typical case. We hit the target ~85% of the time across active Counsel-tier workspaces.

What the alerts are good for

Drift sentinels do two things well: catch reputation incidents in the first hour (before they cascade across engines), and validate causation when content is shipped. When a firm ships a new attorney bio and the sentinel reports a sentiment lift on the next probe round, the causal chain is documented. That documentation is valuable.

What drift sentinels do not do: predict drift before it happens. The math is reactive. The leading-indicator work — sentiment heatmaps, citation source changes — is a separate product surface we'll cover in a future post.

Implementation note

If you are building this without Cognoverge, the minimum viable version is: a cron probing one engine on one prompt every day, with a manual review of any non-trivial difference. You will miss the 90% of incidents that happen between probes, but you will catch the slowest-moving ones — which is more than most firms have today.