After the last field test went out, we didn't stop: same Apple 10-K corpus, same 79 samples, but this time the judge lineup grew from two to six. Alongside the official Jev and the DeepSeek Flash we use in production, we brought in DeepSeek Pro, vector search, and two open-source System One reproductions — openjev and semif.
The conclusion up front: official Jev ties Flash for first place at 92.2% accuracy; the two open-source reproductions manage only 75.9% and 79.7%, and even with the decision threshold turned to its strictest they only reach about 85%. The "crushing victory" in the title refers to this 12-to-16-point gap between the original and the reproductions on the same exam.
The new contenders
A quick introduction to the new faces.
openjev and semif are community open-source reproductions built on the System One idea — based on Qwen3.6-35B-A3B (NVFP4) and Qwen3.5-4B (FP8) respectively — deployed on our own LLM Gate, speaking exactly the same System One contract as TypeSafe's official endpoint: one forward pass returns a probability distribution, yielding yes/no + score. To plug them in, we added a custom provider to the recall chain this round: fill in endpoint, model, and API key, and any System One-compatible endpoint is in — verified end to end.
The third newcomer is vector search: doubao embeddings with cosine similarity, documents chunked at 10k characters and scored by the max chunk. It has no decision model — it's essentially a control group, there to answer the question "can raw vector similarity do recall judgment at all?"
Corpus and samples are identical to the last round: Apple 10-K filings FY2016–FY2025, 20 documents, 25 year-anchored questions, 79 (question, doc) pairs, 52 of which are "looks right but wrong year" hard negatives. The decision contract and threshold (5) are aligned across all six; openjev / semif / vector were live API calls this round, while Jev / Flash / Pro reuse last round's cache.
The results table
79 samples through six judges — the core numbers:
VectorSearch has no native yes/no contract; the 64.6% in the table is an upper bound under an oracle threshold fitted on this very set — real deployments would do worse. It's the only one of the six that can't serve as a filter at all.
Two numbers deserve a separate look: openjev's average latency is 0.71s — faster than official Jev's 1.0s, so the reproduction didn't lose on speed; but its accuracy is 75.9%, sixteen points behind. It's "fast but can't judge" versus "fast and accurate."
Where the gap is: probability calibration
More telling than accuracy is probability quality. Mean noul probability from each judge, gold set versus negative set:
Jev's probabilities are genuinely calibrated: gold sits near 0.99, most hard negatives get pushed below 0.5. Slide the decision threshold anywhere from 5 to 9 and accuracy stays at 92.2% with all 25 gold retained — complete insensitivity to threshold choice, which in engineering terms means no tuning.
openjev's problem is a global upward bias: the negative-group mean is 0.567 — it tends to let things through. Raising the threshold to 8–9 lifts accuracy to 84.8% with gold still fully retained, but it never reaches 92%. semif sits in between: 82.3% at thresholds 7–8 — acceptable for a 4B model, but equally permissive.
Vector search loses differently: gold 0.758 versus neg 0.746 — a separation of only 0.012. Adjacent-year same-section documents are twins in embedding space. Last round we said "looks right but wrong year" is where recall filters most easily fail; this run proves it: vector similarity is structurally helpless in that scenario, which is exactly why you need a decision model.
Decision agreement corroborates the gap: Jev agrees with Flash 94.7% of the time and with Pro 96.1%, but only 80.3% with openjev and 80.5% with semif — the reproductions and the original clearly are not the same model at the decision boundary.
Response time: fast and stable are different things
Last time we only reported average latency; this round adds a dedicated measurement. Two scopes: the full 79-sample distribution (collected with 4 concurrent workers, so it includes queueing noise), and a fresh 10-sample serial benchmark run this afternoon — one request in flight at a time, each timed individually.
In the full-set distribution, all three System One judges keep p99 under 3.3 seconds; the generative LLMs are a different story — Flash and Pro hit p90 of 64.1s and 65.1s, samples that ran into the 60-second timeout and retried. "Fine on average, falling apart at the tail" — for real-time recall filtering, that tail is not something you can design around.
More noteworthy is what the serial benchmark exposed: the official Jev endpoint was flaky this afternoon. Between 15:15 and 15:26, only 4 of 10 calls succeeded — 1 read timeout, 2 remote disconnects, 2 HTTP 503s, and 1 HTTP 529 system_overloaded, with a response body that read "high traffic, please try again later". The 4 successful calls took 0.64–1.61s, consistent with yesterday's full-set numbers — still fast; it's availability that dropped. As for why it slowed down, our guess is that its recent traffic has grown — the 529 overload message is about as close as it gets to an official admission of that explanation, but only TypeSafe knows for sure.
By contrast, in the same window the gate-hosted openjev and semif completed all twenty calls successfully, all under 1.4s — the self-hosted open-source reproductions were actually more stable on availability, just a tier behind on judgment quality. That settles their position: second level of the fallback chain, not the primary judge.
The engineering implication changed accordingly: the rag_jev_on_error=fallback_llm fallback went from "recommended" to "proven mandatory" — wiring Jev into production without a fallback is betting on the official endpoint's uptime.
Converting to per-question recall cost (3.16 candidate documents per question in this corpus): serial judgment takes ≈2.8s for Jev, ≈2.1s for openjev, ≈2.4s for semif, ≈9.5s for Flash, and ≈15.3s for Pro; while the "vector prefilter + Jev precision judgment" combo — roughly 0.4s of retrieval plus second-scale judgment on just the top-k — remains the best latency/quality trade-off.
The awkward position of vector search
This round also measured its ranking ability: for each question's gold document, the similarity rank hit top-1 in 16/25 and top-3 in 25/25. So vectors work well as a top-k prefilter — cut the candidates from N down to 3, then hand them to Jev or an LLM for relevance judgment; they don't work as a standalone filter. This is fully consistent with how we already position hybrid-index (embedding + vector store). But personally, I think vectors have always occupied a very awkward position: intrinsically weak effectiveness, hard to improve, not part of the Scaling-law paradigm — and they're not naturally compatible with large language models or natural language at all, since everything must be vectorized first. That means preprocessing, updating and the like all become hard problems, and the context is usually small — typically only 4K or 8K — with very low information density. So my personal view: avoid it when you can.
How to read this data
First, the evidence chain for "crushing victory" is complete: same corpus, same samples, same contract, same threshold — official Jev 92.2% versus openjev 75.9% / semif 79.7%. The reproductions are contract-compatible with beautiful latency (0.71s, the fastest tier), but their probability calibration is a tier below and they let through too much — on the System One path, the quality of the model weights' training is still the moat; aligning the interface alone doesn't close the gap.
Second, Jev remains the primary judge, but today gets a footnote: accuracy tied with production Flash, gold retained 25/25, 11× faster than Flash, about $0.00018 per call, and calibration so good it's threshold-insensitive — nothing to fault on judgment quality. But the official endpoint's availability was genuinely shaky this afternoon.
That said, the current open-source reproductions haven't been through secondary post-training — they've only been made capability-compatible. We genuinely hope the open-source and closed-source versions can develop in sync, in both directions. Personally, I think Jev actually represents a real paradigm shift. And we hope to see more new paradigms ahead — not just today's pure-LLM paradigm.