A summary you can audit, one sentence at a time.
A summary can sound right and still be wrong: the wrong speaker, a dropped "maybe", a number that was never said. Gavel does not grade how a summary sounds. It checks whether each sentence is actually in the recording, and shows you the passage.
Transcript
Captions are fetched from YouTube (or pasted) and cut into short passages of 20 to 45 seconds, ending at sentence boundaries.
Claims
A model writes 6 to 20 standalone sentences that cover the whole recording, keeping names, numbers and hedges exactly as spoken.
Evidence
For each claim, a search over the passages picks the five most likely sources. Only those five are shown to the judge.
Verdict
A judge model reads the claim and its five passages and answers: supported, partly, not supported, or contradicted, plus who said it and a quote.
Flag
A fixed rule decides which verdicts to flag. You can see the trade-off on the toggle: the rule matters more than the model.
A worked example
From Andrej Karpathy's "Let's build GPT". The claim on the right was typed into the "check a claim of your own" box. Gavel found the passages on the left and returned the verdict.
Passages the search picked
The claim and the verdict
Karpathy says the model is trained on the complete works of Shakespeare and reaches a validation loss of 0.9.
Why: Karpathy does state it is trained on all the works of Shakespeare, but the evidence shows the validation loss is around 2.5, not 0.9.
Half true, half invented. The judge split it, named the real number, and pointed at the passage. With "also flag partial support" on, this is flagged; with it off, it is not. That is the whole point of the toggle.
Eight ways a sentence can be wrong
These are the error types the judge was measured on. Gavel's benchmark took 383 sentences that were true by construction, planted exactly one of these errors into 681 copies, and checked whether the judge caught them. The one that matters most for spoken recordings is the first: text alone cannot tell you who said something.
Wrong speaker
Justice Kagan asked whether...
Justice Alito asked whether...
Flipped negation
...the statute does not apply.
...the statute applies.
Changed number
...reached a loss of 1.48.
...reached a loss of 0.9.
Swapped parties
The petitioner argued...
The respondent argued...
Dropped hedge
...this might reduce costs.
...this reduces costs.
Reworded quote
"we need a smaller data set"
"we want less data"
Flipped outcome
The court affirmed.
The court reversed.
Made up
(nothing in the recording)
The speaker cites a 2019 study showing...
The threshold matters more than the model
The judge gives every claim one of four verdicts. Which of those you treat as a flag is a choice, and it changes the numbers more than switching models did. Both settings below use the same verdicts, measured on the 1,064-claim benchmark.
Default: flag only clear problems
Flags "not supported", "contradicted", and "wrong speaker". Almost never flags a correct claim.
"Also flag partial support"
Adds "partly supported" to the flag set. Catches almost everything, and flags most correct claims too.
Choices that came from the benchmark
Gavel Live is the product end of a benchmark that was built first. A few decisions only make sense in that light.
Keyword search, no embeddings
The evidence search is BM25, a classic keyword ranker, and nothing else. In the benchmark, adding a dense vector search did not improve results (F1 0.83 vs 0.81), so there is no vector index to run or pay for.
A forgiving parser
The first version of the benchmark silently threw away 75% of the judge's answers because of a formatting quirk, and defaulted their severity. The parser now repairs and salvages responses, and every verdict records whether it needed repair. You see that count on each result.
The judge only sees five passages
That keeps verdicts cheap and traceable: every "why" points at a passage you can play. It also means the judge cannot see what a summary left out. Omissions are the next thing to measure.
What it cannot do yet
Speakers
Auto-generated captions have no speaker names, so "who said it" is only checked when the transcript names them (pasted transcripts can).
Omissions
A claim can only be judged if it exists. If the report never mentions something important, nothing flags that. A coverage check is planned.
Not advice
Verdicts are about whether the recording supports a sentence, not whether the sentence is true in the world. Not medical, legal or financial advice.