Yesterday we gave Jev a real-world test: we used ten years of Apple annual reports as the corpus and ran it head-to-head against the recall-judgment LLM we use in production. The conclusion up front — accuracy was tied, latency differed by 11×, and each judgment costs about $0.00018.
This post explains what we tested, how we tested it, and gives the full data.
What we tested: a real financial corpus
The test corpus is Apple's 10-K annual reports, FY2016 through FY2025 — ten years. The original HTML comes from SEC EDGAR (AnnualReports.com rate-limited us midway, so we went straight to the official source). From each year we extracted two documents: MD&A (Management's Discussion and Analysis, including the liquidity section) and Business — 20 documents in total, 8.5k–16k characters each.
This corpus wasn't chosen at random. 10-K filings have several properties that make them great for stress-testing recall judgment: long documents, similar structure, year sensitivity, and heavy overlap between adjacent years — same template, same line items, only the numbers change. This is exactly where retrieval systems most often fail: the documents that come back all look right, but the year is wrong.
How we tested: test logic
We designed 25 year-anchored questions, mixing Chinese and English — for example, "In the FY2019 report, which line item did Apple reclassify a certain revenue into?" Each question anchors to a directly verifiable evidence substring in the document for its year. Right or wrong isn't judged by feel — it's judged by string match.
The sample set contains 79 (question, doc) pairs:
- 25 gold: a question paired with the document from the year it actually belongs to;
- 52 hard negatives: the same question paired with an adjacent year's same-type document — designed to test "looks right but isn't";
- 2 unrelated documents: non-financial content, just for good measure.
One annotation detail is worth noting: the 10-K MD&A tables carry three-year comparison data, so 13 nominal "hard negatives" actually do contain the fact being asked. Instead of labeling them all false, we marked them expected_relevant=true based on whether the evidence string literally appears. Without this fix, both sides' accuracy would be unfairly penalized.
The judgment contract was identical on both sides: output yes/<0-10> or no/<0-10>, threshold 5. Jev went through TypeSafe's live /v1/systemone endpoint (jev-1.13.0); the baseline was deepseek/deepseek-v4-flash via LLMManager/SimpleByzerLLM, gen.timeout=60s, 6 concurrent workers — identical to the production configuration.
The data
After all 79 samples ran, the core results:
| Metric | Baseline LLM | Jev |
|---|---|---|
| Judged / errors | 77 / 2 | 77 / 2 |
| Accuracy vs expected | 92.2% | 92.2% |
| Gold kept (of 25) | 24 | 25 |
| Negatives rejected (of 54 non-gold) | 40 | 37 |
| Average latency | 11.3 s | 1.0 s |
| p50 latency | 3.0 s | 0.9 s |
| Total input tokens | 280,856 | 333,103 |
| Total cost (estimated) | — | $0.0140 (about $0.000177/judgment) |
Each side erred exactly twice — and both were timeouts, not wrong judgments: Jev had two JevRequestError read timeouts; the baseline had two 60-second LLMRequestTimeoutErrors. Neither side produced a single contract violation — every returned string parsed as yes/<n>/no/<n>.
The two judges' decision agreement rate was 94.7% (only 4 disagreements among the 75 samples both sides judged), with score MAE of 0.95. All 4 disagreements were on hard negatives:
| id | expected | LLM | Jev | Reading |
|---|---|---|---|---|
| q09-neg2 | T | no/3 | yes/5 | Jev was right — the FY2020 document does contain the FY2019 reclassification note |
| q14-neg2 | T | no/2 | yes/6 | Jev was right — the FY2021 document discusses the continuing impact of COVID-19 |
| q11-neg1 | F | no/3 | yes/7 | LLM was right — Jev false-positived on an adjacent-year dividend document |
| q17-neg2 | T | yes/6 | no/2 | LLM was right — the FY2018 document kept the $250B program figure, and Jev missed it |
Jev won 2, the baseline won 2 — the disagreements are symmetric, not a systematic bias.
While we're at it: cost
The baseline's cost column in the table above is empty; let's fill it in. On DeepSeek's official pricing page, the deepseek-v4-flash we used is now carried by V4.1-Flash: cache-miss input is $0.15/million tokens (off-peak) and $0.30 (peak); output is $0.6/$1.2; cache-hit input is nearly free ($0.003/$0.006).
Our eval ran on a Sunday, so everything counts as off-peak: 281k input tokens all at miss prices is $0.042; output tokens weren't logged separately, but it's a reasoning model, so at a few hundred to a thousand tokens per call, add $0.01–0.04 — total roughly $0.05–0.08, or $0.0006–0.001 per judgment.
Against Jev's $0.0140 total and $0.000177 per judgment: Jev is 3.5–5.6× cheaper; at peak baseline pricing the gap widens to about 8×. One thing worth noting: Jev actually consumed more input tokens (333k vs 281k) yet cost less overall — it does a single forward pass that outputs a probability distribution, with no reasoning output to bill.
How to read this data
First, the accuracy tie itself is worth money. Jev doesn't generate text — a single forward pass produces a structured yes/no + score judgment. On this corpus it matched the production LLM's 92.2% at 1/11 the average latency and about $0.00018 per call — for recall judgment, a high-frequency call path, cost and latency are the product experience.
Second, its weakness is the same as the LLM's: boundary cases in adjacent years and comparison-data documents trip up both sides, and neither has a systematic edge. That suggests the bottleneck is the task's intrinsic difficulty, not Jev's capability ceiling.
Third, Jev's gold recall is a perfect 25/25. The baseline's only missed gold was a timeout, not a misjudgment — but in production, a timeout feels about the same to the user as a wrong answer.
The data, samples, corpus extractor, and eval scripts all live in the repo: evaluation_financial_results.json, samples_financial.jsonl (79 entries), evaluate_financial_recall.py — feel free to rerun.