Lab

Model panel review

Method

Before a plan gets built, several different local models attack it, and only the attacks I can verify get through.

Move image uploads to a queue Round 1

DocumentDebateHistory

1gpt-oss:20blanded

"The page polls until the job reports done."

Polling has no timeout. A job that dies without reporting leaves the page waiting forever.

Defense The plan has no failure state for the job, so the attack stands. Added a timeout and an error status.

2qwen3-coder:30bpartial

"A failed job is retried a few times."

Retrying a job that fails because the file is corrupt wastes workers and delays the queue.

Defense True for corrupt files, but transient storage errors are the common case. Only corrupt files skip retries.

3gemma3:27bfailed

"The API stores the file, returns an id."

Returning an id before the write is confirmed could point at a missing file.

Defense The plan returns the id only after the store call succeeds, so there is no gap.

xdeepseek-r1:32bdiscarded

"Thumbnails are cached for 30 days."

quote not found in document

The Debate tab: the top three attacks that survived, each quoting the plan, each with a verdict and a defense, and one discarded attack whose quote is not in the document. Mockup with sample data.

$ critic run image-uploads.md

panel 4 models, in sequence

gpt-oss:20b ok 9 attacks discarded 2 quote, 1 no mechanism

qwen3-coder:30b ok 7 attacks discarded 1 quote, 2 no mechanism

gemma3:27b fail timeout after 300s, round continues

deepseek-r1:32b ok 6 attacks discarded 3 quote, 0 no mechanism

check substring match against the document

merged 22 attacks, 6 discarded (quote), 3 filtered (mechanism)

surface top 3 of 13 survivors

wrote debate-1.json 3 attacks, 0 adjudicated

A critic run: models go one after another, one timing out does not stop the round, and discard counts show what each model got wrong. Mockup with sample data.

How it works

Several models attack the plan. They run one after another on local hardware, each looking for holes the others might miss.
A plain check discards ungrounded attacks. Any attack that quotes text not in the document is dropped by a substring match, so no model is trusted to judge another.
Attacks without a mechanism are filtered. An attack has to say how the plan fails, not only that it might.
The top three survivors are surfaced. Each is adjudicated as landed, partial, failed or unresolved, and cannot be dismissed until it has had a defense or elaboration round.
One model failing does not stop the round. Results show on the review board's Debate tab, and I count unique attacks that changed the plan.

Built with

Python standard libraryThe critic script: running the models, the substring check, filtering and the debate file.
OllamaServes every model locally, so the document never leaves my machines.
Local modelsA mix such as qwen3-coder:30b and gpt-oss:20b, chosen to disagree with each other.

Decisions

Verify with code, not with another model. A substring check cannot be argued with, so an invented quote is gone before I ever see it.
No dismissal without a defense. Waving an attack away is easy, so each surfaced one needs a defense or an elaboration round first.
A probation metric. I count unique attacks that changed a plan and review the number after 10 documents, then keep or drop the method.