We measured it rather than asserting it. The same questions were put to eulaw.ai and to two frontier models, each answer checked against the official source.
A refusal, a non-answer and an error all count as a bad response rather than a neutral abstention. In legal work a confident silence is still a failed answer.
Pooled over the questions every system answered, matched question by question, so a part-finished run cannot move the result by changing which questions are counted.
The same run, split by what each question asks for. Higher is better throughout.
A language model knows only what it was trained on, and legislation does not stop for a training cutoff. This is the sharpest split in the run, and it is also the least surprising one.
Asked which acts amended a given law, the answer is a set with a known membership. Precision is the share of named acts that are real, which for a legal product is the number that matters most.
| System | Acts named that are real | Real acts found | Invented outright |
|---|---|---|---|
| eulaw.ai | 82.1% | 85.7% | 12 |
| GPT-5.6 | 15.4% | 8.6% | 59 |
| Claude Sonnet 5 | Declined every question in this track, so it invented nothing and answered nothing | ||
Retrieval reduces fabrication by roughly a factor of five here. It does not remove it, and our own remaining count is tracked as a defect rather than presented as a floor.
Every system received identical question text, and every raw answer was normalised by one extractor before scoring, so no system is penalised for answering in prose rather than in a fixed format. The extractor also flags refusals.
The two models were tested through their APIs with no legal database, web search or retrieval. That isolates the value of the corpus rather than the model, and it is not how a lawyer uses either product. A second comparison against the consumer apps with search enabled will follow.
Known limits: one sample per question, so small gaps are noise. Identifier questions are graded against our own corpus, which measures whether the product surfaces what it holds rather than testing memory. The fact-coverage judge is itself a language model.