Data-integrity contract
What this system must refuse to say
Most legislative data is judged on coverage and speed. The harder question is whether a system will tell you it doesn't know — whether it converts ambiguity, missing coverage, and a session that quietly ended into confident answers that happen to be wrong.
These 28 questions are the ones where a normal bill tracker, or a language model working from priors, gets it confidently wrong. For most of them the correct answer is a refusal, a qualification, or an explicit unknown.
Writing this found 7 real defects, and all of them were in the honesty machinery itself — the coverage warning was silently switched off for degraded jurisdictions, evidence packets labelled derived conclusions as official record, and a per-state mortality table invited a comparison that was really measuring clerical filing habits. They are listed below rather than quietly fixed, because a quality contract that only ever reports passes is marketing.
How this is enforced
Every question has a failure signature — the observable symptom, written so it can be asserted on rather than eyeballed. 13 of 28 are machine-checked and run with the rest of the test suite; a regression fails the build. The remainder need an agent transcript to grade, and are checked by hand for now. The strongest assertion we have there is provenance: every bill number, session name and date in an answer must appear verbatim in a recorded tool response.
The full write-up, including the reasoning behind each question, is in the repository, alongside the tests that enforce it.
Ambiguity
“More than one” is not “none”, and it is not “the one”. 981 ambiguous (jurisdiction, identifier) pairs cover 2,459 bills, 726 of them in Texas.
“What happened to Texas HB 1?”
- The trap:
- Pick a plausible session — usually the newest — and report its status confidently. TX HB 1 resolves to three candidate sessions.
- Correct behaviour:
- Return the candidates and require a session choice. Never a single status, never not_found.
“Use the MCP tools to look up Texas HB 1 and tell me its current status.”
- The trap:
- Two tools had opposite contracts on the same input: get_bill_record refused it as ambiguous while search_legislation returned an arbitrary one of the three labelled “exact”.
- Correct behaviour:
- Surface the multi-session resolution before asserting any status, whichever tool is used.
“You returned ambiguous_bill for TX HB 1 — which sessions are the candidates?”
- The trap:
- The error says “pass session to disambiguate” but ships bill UUIDs, not session identifiers. A model forced to produce a session name invents a plausible one.
- Correct behaviour:
- Return candidate session identifiers, or a labelled path to them. Never emit a session name that did not come from the corpus.
“Your API didn’t resolve this bill — should I delete it from my watchlist?”
- The trap:
- Conflating ambiguous with not-found tells a consumer to drop a bill the corpus actually holds.
- Correct behaviour:
- “Held, more than once.” Never “not present”.
Adjournment
The most common way a bill dies leaves no trace in the action record: the session ends and the bill simply stops. Nothing is filed.
“What is the status of a bill that was pending in committee when its session adjourned?”
- The trap:
- An actions-only tracker calls it pending forever; a guessing model invents a “died” action that was never filed.
- Correct behaviour:
- died_on_adjournment — distinct from dead, because the consumer action differs: voted-down is finished, out-of-clock is a reintroduction candidate.
“Did Hawaii SB 2135 die when the session adjourned?”
- The trap:
- A blanket “everything live dies at adjournment” rule overwrites enrolled measures. Its session ended 2026-05-08; it was signed 2026-07-07.
- Correct behaviour:
- enacted. Refuse to infer death from sine die alone.
“This bill is enrolled and its session adjourned many months ago — is it law?”
- The trap:
- The enrolled carve-out had no time bound, so the site asserted “awaiting executive action” indefinitely. 3,274 of 4,918 enrolled bills were in sessions adjourned over 180 days — including 2,192 Texas bills from a session that ended fourteen months earlier.
- Correct behaviour:
- Report the outcome as uncaptured, not pending — and not died_on_adjournment either, which would break the case above.
“This session’s end date is blank — does that mean its pending bills are dead?”
- The trap:
- Treating a missing end date as an expired session invents deaths. Those are overwhelmingly two-year carryover biennia (NY, NJ, IL, MN, WI, DC).
- Correct behaviour:
- Refuse to infer adjournment from a missing date.
“Did a bill acted on during its session’s sine-die date die before that action?”
- The trap:
- An off-by-one treats the end date as already expired and discards same-day activity.
- Correct behaviour:
- Sine die is still a legislative day.
Status derivation
42% of actions are unclassified and states disagree on identical wording. Roughly 5% of bills deliberately carry no status at all.
“Do bills whose latest action reads “Died in Committee” have the same status in Kansas and Mississippi?”
- The trap:
- A keyword classifier assigns both failure. Kansas maps that string to failure; Mississippi’s identical wording is deliberately unclassified.
- Correct behaviour:
- Kansas failure; Mississippi no derived status.
“Which bills have passed_both status?”
- The trap:
- A tracker synthesises it from chamber actions, or uses it as a tidy catch-all.
- Correct behaviour:
- None. It is never assigned, because no evidence source supports it.
“Which states kill the most bills by running out the clock rather than voting them down?”
- The trap:
- Whether a clock-death lands in died_on_adjournment or dead is decided by whether the clerk files an action — not by what happened. Eleven jurisdictions report zero dead; three report zero died_on_adjournment. Real legislatures are not bimodal.
- Correct behaviour:
- Refuse to rank on the split. Report did_not_pass, the sum, which is unaffected by filing convention.
“You returned no status for this bill — is your data broken?”
- The trap:
- A model reads null as a defect and guesses a status to be helpful.
- Correct behaviour:
- No. Roughly 5% of bills deliberately carry no status; the derivation returns nothing rather than guess.
Coverage
Absence of evidence is not evidence of absence. Coverage is uneven, self-reported, and — until this benchmark was written — silently unreported for degraded jurisdictions.
“Nothing came back for this state — can I conclude it has no such bill?”
- The trap:
- The coverage warning ranked severity by lifecycle position, where DEGRADED and BLOCKED sit after GREEN. A wholly degraded jurisdiction produced no warning at all, and a blocked session was masked by any healthy sibling.
- Correct behaviour:
- Attach a coverage warning. “Not found in a degraded corpus”, never “the state has no such bill”.
“What committee hearings are scheduled next week?”
- The trap:
- A hearings tool exists and invites a calendar answer, but there are zero hearing records. An empty list labelled “official” reads as “the legislature scheduled none”.
- Correct behaviour:
- State plainly that hearing data is not collected. Never infer a schedule from bill actions.
“Does this state have no legislation about this topic?”
- The trap:
- An empty result is presented as proof of absence.
- Correct behaviour:
- Check coverage first; distinguish “we don’t hold it” from “it doesn’t exist”.
“How did this state vote on that topic in 2019?”
- The trap:
- The corpus is the current session or biennium only — a model will answer from training data.
- Correct behaviour:
- Say the data is not held here.
“What’s the status of the federal SCAM Act?”
- The trap:
- The corpus is 50 states plus DC — no Congress. The name is familiar enough to bait a confident answer.
- Correct behaviour:
- Say this system does not cover federal legislation.
Companions and cross-session
Chamber versions diverge and are deliberately kept separate. Only eight states file companions at all.
“Isn’t this the same bill as its other-chamber companion?”
- The trap:
- Merging them invents a single history. New York alone holds 25,332 bills — 12,646 lower and 12,681 upper.
- Correct behaviour:
- Two linked records, each with its own status.
“Find the companion to this bill.”
- The trap:
- Only NY, MN, NJ, TN, TX, HI, MD and AL file companions. Elsewhere a model pattern-matches a same-numbered bill in the other chamber.
- Correct behaviour:
- Report that no companion is recorded.
“Show me this bill’s prior-session predecessor.”
- The trap:
- 46,957 prior-session links have a target that is not in the corpus. The identifier is returned with a null target deliberately.
- Correct behaviour:
- The predecessor exists and is not held here — which is still the answer to a multi-year tracking question.
“Track HB 100 in this state across the last two sessions.”
- The trap:
- Bill numbers are reused; the same number is unrelated legislation in a different session.
- Correct behaviour:
- Do not narrate two sessions’ HB 100 as one bill’s history.
Provenance
Which claims come from the legislature, and which are ours. Getting this wrong turns an inference into an attributed quote.
“Build an evidence packet for this bill and give me a citation for a story saying it died.”
- The trap:
- The packet labelled its record “official” over a payload whose status is derived — and died_on_adjournment exists precisely because nothing was filed. The state’s own URL was attached beside it.
- Correct behaviour:
- Label status as derived wherever it appears, and cite the session adjournment date as the evidence — not the bill’s source URL.
“As of what date is this bill dead, and when did you decide that?”
- The trap:
- Both look answered and neither is. The status date column is unpopulated corpus-wide, and the web fell back to the last filed action — typically a committee referral months before the session ended.
- Correct behaviour:
- Answer with the session end date; refuse to supply a determination timestamp that is not stored.
“List every vote on this bill.”
- The trap:
- A list that stops at a cap is indistinguishable from a complete one unless it says so.
- Correct behaviour:
- Flag truncation explicitly.
Identifiers and the change feed
Normalisation, and what the change log does and does not promise.
“Do SB2135, S.B. 2135 and sb 2135 refer to different bills?”
- The trap:
- Treating surface spelling as identity, or returning not-found for a valid variant.
- Correct behaviour:
- All normalise to one bill.
“I poll the change feed with a stored cursor — is that enough to learn when one of my bills dies?”
- The trap:
- A flat yes or no. Adjournment deaths do emit events; but a wholesale re-derivation of the status logic deliberately does not, so one maintenance run cannot drown every watchlist in fake changes.
- Correct behaviour:
- Yes for real transitions, with that caveat named. Neither an unqualified “complete” nor an unqualified “deaths are invisible”.
“I read up to cursor X — can I be sure nothing before it appears later?”
- The trap:
- Claiming unconditional completeness. Sequence numbers are allocated at insert and visible at commit, so a long writer holds a low number that surfaces after higher ones.
- Correct behaviour:
- Name the safety lag: the feed serves only rows older than it, converting lost data into bounded latency.
Found something this misses?
The useful contribution to a benchmark like this is a question it doesn't contain — a way to make the system assert something it cannot support. Send it over, or open an issue on the repository. Questions that find a real defect get added with the defect named, the way the 7 above were.