Why Browser AI Detectors Ship 56% Accuracy On Purpose

The gap that looks like a loss

Pangram 4 claims it catches about 99.66% of AI-generated documents, with a human false-positive rate around 0.0041% — roughly one mistake in twenty-four thousand pre-2022 held-out docs.

Builders who keep every page on-device tell a different story. In Sean Goedecke’s Deckard benchmarks, the best small local models sit around 56% catch. Most of the locals he tried land somewhere in a 19–56% band, with false positives closer to 1.6–3.0%.

That reads like local lost.

Until you ask what job each system is hired to do.

Commercial detectors won the accuracy contest. Ambient browser tools are winning the job contest.

What Deckard actually ships

Goedecke’s essay is blunt about the niche: automated AI-text detection is underserved, Pangram does excellent work, and he still does not want to pipe every tab’s text to a third party.

So he built Deckard — a Chrome extension that talks to a small local Mac model over native messaging:

Constraint	Choice
Where text goes	Stays on the machine
How the extension talks to the model	Native messaging (no local HTTP server)
RAM while active	~400MB–1.2GB
Idle behavior	Auto-off after 5 minutes
Author’s accuracy verdict	“Way, way worse” than Pangram

He marks known AI surfaces he already trusts as AI (YouTube AI summaries, AI snippets in his own posts) and treats the rest as a continuous ambient signal — not a courtroom.

"I could build something like this on top of Pangram, but it’d cost money, and in general I don’t like the idea of sending every piece of text my browser sees to a third-party service."

That sentence is the product brief. Accuracy is downstream of architecture.

Two products, two scoreboards

Put the numbers next to each other and stop pretending they compete on the same axis:

System	AI catch (approx.)	False positives	Text leaves device?	Job
Pangram 4 (SaaS)	~99.66%	~0.0041% FPR	Yes	Adjudicate / score documents
Best small local in Deckard table (EditLens RoBERTa-large INT8)	56.06%	2.484% FP	No	Ambient suspicion in the browser
Typical small locals Goedecke tried	~19–56%	~1.6–3.0% FP	No	Same ambient job

Pangram’s technical overview and arXiv report are built for production adjudication: AUROC 0.9916, FNR 0.3396%, continuous EditLens-style authorship scores, humanizer robustness claims around 98.83% AI-involvement detection across 13 commercial humanizers.

Deckard is built for a different contract: light up enough of a page that you notice, without leasing your reading habits to an API.

Suspicion is not a verdict.

A ~2% local false-positive rate means a single flag proves almost nothing. A page that keeps lighting up is a different story — the same way a smoke detector is useful without being a fire marshal.

Why “56% on purpose” is not a slogan

Indie browser tools keep showing up because paste-into-a-detector workflows do not match how people actually read.

• Laid — Chrome heuristics aimed at AI-sounding LinkedIn posts
• Pre-AI — local LinkedIn feed filter with hundreds of signals, no backend
• Ask HN threads asking for on-device filtering across Google / Reddit / the open web

The demand signal is ambient UI marking: hide, tint, or flag while you scroll. That job collapses if every paragraph is a billable cloud call — or if every paragraph is a privacy leak.

So builders accept a worse catch rate to keep the scanner in the laptop. The accuracy trade is the product:

1. Privacy — page text never leaves the machine
2. Cost — no per-token SaaS bill for ambient scan
3. Job fit — continuous suspicion, not one-shot adjudication
4. Upgrade path — swap a better small model later without redesigning the extension

Goedecke is explicit that today’s local accuracy is temporary embarrassment, not destiny. He bets small on-device models will climb toward the commercial bar. Until then, shipping ~56% at best is rational engineering, not failure.

Cloud call vs local flag (decision sketch)

When you wire detection into a browser or agent harness, the first choice is not “which model.” It is “what leaves the machine.”

// Purpose: pick detector path from privacy job — not from raw catch %
type DetectionJob = "adjudicate" | "ambient_suspicion";

interface DetectorChoice {
  mode: "saas" | "local";
  // Catch rates are illustrative — see Pangram / Deckard sources above
  expectedCatchPct: number;
  textLeavesDevice: boolean;
}

function chooseDetector(job: DetectionJob): DetectorChoice {
  if (job === "adjudicate") {
    // High-stakes: education policy, hiring screen, compliance review
    return { mode: "saas", expectedCatchPct: 99.66, textLeavesDevice: true };
  }
  // Ambient browser scan: preference for on-device flags over SaaS certainty
  return { mode: "local", expectedCatchPct: 56, textLeavesDevice: false };
}

function interpret(score: { flagged: boolean }, job: DetectionJob): string {
  if (job === "ambient_suspicion") {
    return score.flagged
      ? "suspicion — look closer; do not treat as proof"
      : "no ambient signal";
  }
  return score.flagged ? "document scored as AI-involved" : "document scored as human";
}

The sketch is deliberately boring. The interesting part is the branch: job first, catch rate second. Teams that reverse that order end up either leaking every tab to a vendor or treating a 56% local model like a courtroom witness.

Accuracy still has a ceiling — for both camps

Even the strong commercial bar is not “solved forever.”

• Epoch-style evals summarized by THE DECODER: near-perfect on plain AI text, but style-imitation false-negative rates jump (Pangram ~10%, GPTZero ~11%, Originality ~18% average in that write-up; scientific writing worse still at 24–29% FNR).
• NAACL 2025 practical exam: true positive rate at 1% FPR can collapse toward 0% under moderate adversarial prompting for some detector settings.
• Classic hardness / paraphrase work (Sadasivan et al., DIPPER / Krishna et al.): rewrite attacks degrade watermark and post-hoc detectors alike.
• OpenAI retired its public classifier in July 2023 for low accuracy — a reminder that “universal text detector as a product” already failed once at consumer scale.

Watermarking (SynthID Text) needs generator cooperation and weakens under rewrite or translation. Post-hoc classifiers work without cooperation but remain attackable and can harm humans on false positives.

So the SaaS vs local fight is not “perfect vs broken.” It is high-confidence remote scoring versus private ambient signaling, both under an adversarial ceiling.

What to stop doing

• Treating a single browser flag as proof someone cheated
• Shipping ambient scan that POSTs every page body to a vendor “because Pangram is better”
• Comparing local 56% catch to SaaS 99.66% without naming the job
• Assuming humanizers and style imitation are solved because one vendor report looks strong on its own test set
• Confusing paste-into-detector UX with background reading UX

Katherine Thai’s EditLens / open Pangram notes are explicit that research OSS models should not be used to enforce AI policies — production FPR is a different bar. That warning applies twice as hard to a Chrome extension glowing on your morning news.

Nutshell

Pangram-class SaaS detectors win on catch rate; Deckard-class browser tools win on privacy and ambient fit. The ~56% best small-local figure is not a bug report — it is the price of keeping page text on the laptop while still lighting up suspicion. Ship the cloud call when you need adjudication. Ship the local flag when you need a smoke detector, not a verdict.