Building a benchmark with the people who do the work
Before we train anything, domain experts write the questions. What that process looks like inside a Dutch law firm.
The benchmark in brief
Written by practitioners. Graded on sources.
01 · Why practitioners
A test that looks like the job
Most legal benchmarks are built from exam questions. A practising lawyer does not work from exam questions: they work from a contract, a set of facts and a client who needs an answer.
So we asked the lawyers of a Dutch law firm to write the questions themselves, from the kind of cases they handle, anonymised.
Each question comes with the answer they expect and the text it must rest on.
How it works
Two lawyers per question. One way to pass.
Diagram of the process described in this note.
One question, as written
Illustrative example.
02 · How a question is made
Four steps, two lawyers.
- IA real caseTaken from practice, anonymised.
- IIQuestion and answerThe expected answer and its source, written by the lawyer.
- IIISecond readingAnother lawyer checks the question and the answer.
- IVGradingIs it correct, and does the citation hold?
Exam vs practice
Not the question a student gets. The one a client brings.
- Abstract
- No facts
- Answerable from a textbook
- A real clause
- Specific facts
- An answer that must cite its source
Illustrative contrast between the two kinds of question.
In our words
“The citations hold. That is the only test that counts here.”
03 · Why it matters
The test that decides what we ship
Every version of Le Lion is graded on this benchmark before it reaches a lawyer.
If a citation does not hold, the version does not ship.