Jev and typeddecision models

On 683 multi-way questions, renaming members to neutral letters without touching descriptions changed 52.42% of answers and dropped accuracy from .5637 to .2782 (open marker head).

Limits Pool accuracy is low (.5637); authors treat the arm as directional.

Type-Safe Is Not Error-Free · §4.5; Table 4

Controlled interventions showed that reliability depends on action-set size and on near-valid alternatives near authorisation boundaries.

Limits Effects concentrate in authority and risk families.

REFLEX with Jev · Abstract; controlled interventions

Across three comparison models, seven wrong selections on one culture-history question changed downstream counts while preserving the correct final claim label.

Limits Errors were made by comparison models; the point is about evaluation design, not about Jev.

Jev for Scientific Decisions · Abstract; §3 (correct labels can conceal wrong quantities)

On 30 fixed anchors answered twelve times each, Jev kept 23 correct in every response (Claude Sonnet 5: 24; Claude Haiku 4.5: 19). Five of Jev’s anchors received the same wrong label every time, so agreement across repeats overstated correctness.

Limits The Sonnet–Jev difference is one target, with an interval spanning zero. Jev’s output-order condition changes question-key order only.

Jev on Contract Inference · §4.2; Table 2

Seven API-only experiments (about 7,000 calls) report that removing the abstain option drove accuracy on unanswerable items from 0.950 to 0.000 at 0.79 confidence, that a Noul and a two-option Choice over the same question differed by 0.125 on average, that 50 identical requests gave 15 distinct answers, and that option order produced no argmax flips in 400 trials.

Limits Community report; not peer-reviewed.

jujumilk3/jev-calibration-audit ↗ · README “Results” table

A pre-registered audit reports Jev’s chosen-option probabilities as calibrated on CLINC150 (ECE 0.0204) but systematically overconfident on Banking77 (ECE 0.0936), with zero type errors across 8,576 responses.

Limits Community report; protocol frozen before queries, per the README.

jourdanlabs/assay-001 ↗ · README verdict

Playing 300 survey respondents, jev-1.13.0 asked through Noul led GPT-4.1 on all six distribution measures, but asked through a two-option Choice it trailed on the distribution gap (0.1985 vs 0.1789): the primitive changes the conclusion.

Limits “Survey” here means a questionnaire study, not a literature review. Community report.

jjd-lab/jev-synthetic-survey ↗ · README summary

The vendor’s jev-1.13 “jaggedness” page lists known failure modes: literal reading, arithmetic and counting, date comparison, indirection, large irrelevant state, adversarial content, contradictory instructions and criteria, structural invariants, and generation.

Limits A boundary statement by the vendor, not a neutral evaluation.

TypeSafe: Jev 1.13 jaggedness ↗ · Failure-mode table (last reviewed 2026-09-17)

A three-seed JevLite ensemble reached AUROC .974 with calibration error .052 on 577 turn-level decisions, non-inferior to an LLM judge (MiniMax-M3) at a pre-registered .02 margin, with no false alarms on legitimate calls and decisions 1.14 turns earlier.

Limits Recipe selection had test-set exposure. All callers synthetic. Fine-tuned ModernBERT not significantly worse.

Open-Jev on CallScreenBench · Abstract; results

Raw sentence-level probabilities had ECE 0.1192 on the ReXErr test set (open NLI: 0.1045); isotonic calibration fitted on half of the studies lowered Jev’s ECE on the other half from 0.1235 to 0.0077.

Limits Calibration fitted and tested on halves of the same benchmark. Measured on the multi-question configuration’s support outputs.

Jev as a Radiology Report Judge · §5.6; Appendix G

Propagating a fitted human–model discrepancy gave retrospective coverage of 0.93 and 0.96 at nominal 0.80 and 0.90 on 37 unseen studies, against 0.29 and 0.36 from human sampling uncertainty alone.

Limits Retrospective; fitted on studies selected for reliable contrasts.

KITE · §8; Figure 3

Seven API-only experiments (about 7,000 calls) report that removing the abstain option drove accuracy on unanswerable items from 0.950 to 0.000 at 0.79 confidence, that a Noul and a two-option Choice over the same question differed by 0.125 on average, that 50 identical requests gave 15 distinct answers, and that option order produced no argmax flips in 400 trials.

Limits Community report; not peer-reviewed.

jujumilk3/jev-calibration-audit ↗ · README “Results” table

A pre-registered audit reports Jev’s chosen-option probabilities as calibrated on CLINC150 (ECE 0.0204) but systematically overconfident on Banking77 (ECE 0.0936), with zero type errors across 8,576 responses.

Limits Community report; protocol frozen before queries, per the README.

jourdanlabs/assay-001 ↗ · README verdict

With English instructions and a Spanish state, jev-1.13.0 lost 3.0–6.4 accuracy points on all four datasets (XNLI, PAWS-X, MASSIVE, Belebele) across 3,200 paired items and 19,200 calls; ECE roughly doubled on the two hardest tasks.

Limits Community report; pre-registered, per the README.

marcosmartinez/jev-acento ↗ · README “Findings”

On a rule-based task whose label is not recoverable from the text, the chosen level averaged 0.74 probability and a temperature of about 3.4 was needed to make probabilities honest; the returned probabilities were quantised to 0.01 and the gateway exposed no model version.

Limits Unpinned model alias; post-hoc temperature diagnostic.

scienthoon/jev-ood-calibration ↗ · README “Results” and synthetic-task section

For a frozen Qwen3-8B on BANKING77 (20-way, 300 items), label-free permutation calibration reduced answer flips under option reversal from 0.230 to 0.073 and ECE from 0.240 to 0.184; with 100–500 labels ECE fell to 0.095 and the share decidable at ≤5% error rose from 7.7% to 52.0%.

Limits Point estimate at n=300; wide interval per README.

nokia-applied-research/AnyJev ↗ · README “What it does” table

The vendor states that calibration is measured across groups of predictions and does not guarantee that an individual answer is correct.

TypeSafe: System One ↗ · “How it differs from an LLM”

Jev trailed the per-task best LLM on 14 of 15 evaluation tasks, by a median 11.6 macro-F1 points, at a median 44 times lower measured cost.

Limits Zero-shot protocol for every model; tasks from one established suite; older corpora may overlap with training data.

Decision Models for CSS Annotation · Abstract; §4

Routing low-confidence items to an LLM matched or exceeded that LLM alone at roughly a quarter to half of its measured cost; items at or above 0.9 confidence reached a median accuracy of 0.815 at 0.376 coverage.

Limits Measured on the tested tasks and LLMs only.

Decision Models for CSS Annotation · Abstract; §4 (cascades)

Against 2,416 blinded human judgments the typed model reached F1 0.908; one frontier model was 0.059 higher and the other indistinguishable.

Limits Sampling design gives an effective sample of about 1,265.

Calibrated Decisions at Scale · Abstract; §4.3–4.4

Without caching, median end-to-end latency on requests both systems completed correctly was 11.1–25.3% lower and API fees per correct completion 69.0–70.6% lower; correct on-time completion matched DeepSeek in 7 of 8 conditions and exceeded it in 1.

Limits Caching-disabled conditions only for the latency figure.

Replacing LLMs with Jev at the Edge · Abstract; end-to-end latency section

Baseline accuracy was 77.38% [75.99, 78.81] for Jev against 78.96–83.21% for the seven hosted language models; Gemini 3.5 Flash-Lite gained 5.16 points (paired 95% interval 3.49–6.98) at about six times the cost, while the gain for GPT-5.6 Luna was not established.

Limits Invalid responses count as incorrect for every model. The held-out split does not establish absence from training data.

Jev on Contract Inference · §4.1; Table 1; Appendix A

A one-question Jev configuration reached Kendall τ 0.573 on RadEvalX and 0.398 on RadEvalExpert with expert total-error counts, exceeding an open NLI judge with the same statements and aggregation; RadMatch on a local 27B model was stronger on clinically significant errors (0.491 vs 0.346 on RadEvalX).

Limits Agreement with reference text, not with the image. Local baselines use a substituted 27B backend.

Jev as a Radiology Report Judge · Abstract; §5.1; Table 1

Flagship anchors covering 1.7% of Epstein states lowered effect MAE from 0.0305 to 0.0180 (paired improvement 0.0125, 95% interval 0.0064–0.0167); on 37 held-out SocSci210 experiments, 0.5–1.5% anchor coverage raised captured decision gain from 0.268 to 0.386.

Limits Preregistered policy-value and heterogeneity primaries were not shown. Corrections can import the flagship’s errors.

KITE · Abstract; §7; Table 2

The Jev kernel passed the content-fidelity gates in all 15 new countries of a 16-country study but failed both intervention-sensitivity gates, which GPT-6 Astra passed.

Limits Models read English in all countries; this does not test multilingual inference.

KITE · §6; Table 1

Median Jev response time was 0.422 s (P95 0.531 s) in full games against 27.60 s for GPT-6 planning calls; of USD 3.71 estimated model cost per game, Jev accounted for USD 0.15 and planning for 96%.

Limits Costs are standard-API-equivalent estimates from recorded usage. Real-time games overlap inference with play.

JEV-Star · §4.6; Tables 5–6

For a frozen Qwen3-8B on BANKING77 (20-way, 300 items), label-free permutation calibration reduced answer flips under option reversal from 0.230 to 0.073 and ECE from 0.240 to 0.184; with 100–500 labels ECE fell to 0.095 and the share decidable at ≤5% error rose from 7.7% to 52.0%.

Limits Point estimate at n=300; wide interval per README.

nokia-applied-research/AnyJev ↗ · README “What it does” table

The vendor’s jev-1.13 “jaggedness” page lists known failure modes: literal reading, arithmetic and counting, date comparison, indirection, large irrelevant state, adversarial content, contradictory instructions and criteria, structural invariants, and generation.

Limits A boundary statement by the vendor, not a neutral evaluation.

TypeSafe: Jev 1.13 jaggedness ↗ · Failure-mode table (last reviewed 2026-09-17)

A first-stage screen returned answers for 499,500 Texas narratives at a cost of $25.23; the second stage coded 195,857 narratives with a 27-question schema. Cost is governed by schema size rather than narrative length.

Limits Single region and schema.

Calibrated Decisions at Scale · Abstract; §3; §4.1

JevLite took 64.5 ms per decision on one consumer GPU, 4.9× lower than the same backbone fine-tuned to generate its answer.

Limits Single-model latency; the three-model ensemble quality is a different scope.

Open-Jev on CallScreenBench · Abstract; Table 1

this-that-model-1.0 decided in 30.9 ms end to end on one laptop GPU and sustained 32 decisions per second, generating no output tokens.

Limits Local timing; not comparable to hosted end-to-end latency that includes the network.

this-that-model-1.0 · Abstract; §4

One pass of the 42-family internal suite cost $0.000217 of electricity (80 W at $0.30/kWh) against $10.636 and 155.2 minutes for the most accurate hosted model measured.

Limits Electricity-only marginal cost excludes hardware, operations and margin; not an API price. Internal suite is not released.

this-that-model-1.0 · Abstract; Table 1 notes

With correct interpretation on all evaluated contracts, Jev reduced median decision latency by 22.4% against DeepSeek and 61.9% against Gemini.

Limits Live API timing from one location and period.

Intent-Driven 6G Edge Orchestration · Abstract; interpretation results

In the real image-reading service reached over simulated NR access, Jev completed 459 of 1,080 requests correctly and on time, against 463 for DeepSeek and 435 for Qwen.

Limits Faster decisions did not raise completions above DeepSeek. Radio access simulated.

Intent-Driven 6G Edge Orchestration · Abstract; real image service results

Across three consecutive measurement blocks, Jev reduced median client decision latency by 15.9–26.5% against a short structured-output DeepSeek deployment.

Limits Provider, region and time-window specific.

Replacing LLMs with Jev at the Edge · Abstract; §§ on decision latency

Without caching, median end-to-end latency on requests both systems completed correctly was 11.1–25.3% lower and API fees per correct completion 69.0–70.6% lower; correct on-time completion matched DeepSeek in 7 of 8 conditions and exceeded it in 1.

Limits Caching-disabled conditions only for the latency figure.

Replacing LLMs with Jev at the Edge · Abstract; end-to-end latency section

Applying the same repeated-request cache to all backends largely removed the latency difference: the savings are concentrated in fresh interpretation.

Limits Cache policy specific to the tested service.

Replacing LLMs with Jev at the Edge · Abstract; cache boundary section

On the 123-contract baseline Jev had the lowest cost and median response time of the ten configurations: $0.000228 per contract and 1.24 s (P95 1.59 s), against $0.001353 and 1.60 s for Gemini 3.5 Flash-Lite and $0.000931 and 1.74 s for GPT-5.6 Luna.

Limits Costs are token usage times public prices; latency includes network and provider infrastructure. One client; providers and collection times differ.

Jev on Contract Inference · §4.1; Table 1; Appendix M

One support question per statement kept expert agreement close to the seven-question design with 43–45% fewer input tokens; the judgment calls cost about $0.000226–0.000258 per report pair, under three cents per hundred pairs, excluding local statement extraction.

Limits Priced from token counts at the list price; local decomposition cost excluded.

Jev as a Radiology Report Judge · Abstract; §5.3; Appendix E

Fresh Jev predictions cost $0.0227 per thousand with median latency 213 ms (p95 304 ms); GPT-6 Astra cost about 312–455 times as much per prediction, and a million agents executed 20 tabulated steps in 0.90 s on a laptop.

Limits Flagship costs are subscription-billed usage converted at list prices. Tabulated execution time excludes fresh inference.

KITE · §9; Table 3

Median Jev response time was 0.422 s (P95 0.531 s) in full games against 27.60 s for GPT-6 planning calls; of USD 3.71 estimated model cost per game, Jev accounted for USD 0.15 and planning for 96%.

Limits Costs are standard-API-equivalent estimates from recorded usage. Real-time games overlap inference with play.

JEV-Star · §4.6; Tables 5–6

On a third party’s recorded cohort of 68 questions over 17 states, this-that-model scored 0.941 accuracy (Brier 0.042) against Jev’s 0.765 (0.133); the authors note the 18-point gap rests on 12 questions.

Limits Small cohort; wording chosen by the third party.

this-that-model-1.0 · Abstract; Table 1; comparison discussion

On the 2,250-question subset of the released benchmark every hosted system answered, Jev scored 0.803 against this-that-model’s 0.844 — a comparison the authors call not clean because their model was trained on those question shapes (training moved it from 0.409 to 0.844).

Limits Training-distribution advantage for the open model.

this-that-model-1.0 · Comparison discussion; Table 3

For a frozen Qwen3-8B on BANKING77 (20-way, 300 items), label-free permutation calibration reduced answer flips under option reversal from 0.230 to 0.073 and ECE from 0.240 to 0.184; with 100–500 labels ECE fell to 0.095 and the share decidable at ≤5% error rose from 7.7% to 52.0%.

Limits Point estimate at n=300; wide interval per README.

nokia-applied-research/AnyJev ↗ · README “What it does” table

Jev is not fine-tuned or LoRA-adapted with customer data; it is trained with RLCD and the same weights serve every account. Architecture and training data are not documented.

TypeSafe: Models (Jev 1.13) ↗ · “Customizing Jev”

LocalJev returns Jev-shaped responses by asking a model for JSON probabilities, validating, retrying and normalising them; its README states this is wire-compatible but not mathematically equivalent to a logit read.

githubnext/localjev ↗ · README “Why a bridge is needed”

SemIf (formerly OpenJev) reproduces the interface pattern with a frozen open model’s option-token probabilities and states that it does not reproduce Jev’s undisclosed model or training.

TheoLeeCJ/SemIf ↗ · README introduction

Kev states that it follows an architecture described in a third-party blog post about Jev; that description is community speculation, not a disclosure by TypeSafe.

jaredpalmer/kev ↗ · README introduction

On LoCoMo, Jev-Mem reached an overall LLM-as-a-Judge score of 0.777 (+11.0% relative to the strongest baseline), built memory in 158 s (6.6× faster than the fastest competing memory system) and averaged 0.93 s per query (36.7% lower than the fastest memory baseline at 1.47 s).

Limits Gains belong to the whole system; each number has a different baseline; LLM-judge metric.

Jev-Mem · Abstract; Experiments; Table 1

In the real image-reading service reached over simulated NR access, Jev completed 459 of 1,080 requests correctly and on time, against 463 for DeepSeek and 435 for Qwen.

Limits Faster decisions did not raise completions above DeepSeek. Radio access simulated.

Intent-Driven 6G Edge Orchestration · Abstract; real image service results

Flagship anchors covering 1.7% of Epstein states lowered effect MAE from 0.0305 to 0.0180 (paired improvement 0.0125, 95% interval 0.0064–0.0167); on 37 held-out SocSci210 experiments, 0.5–1.5% anchor coverage raised captured decision gain from 0.268 to 0.386.

Limits Preregistered policy-value and heterogeneity primaries were not shown. Corrections can import the flagship’s errors.

KITE · Abstract; §7; Table 2

With GPT-6 plans, the controller won all four completed non-cheating full games, including two against Lv7; the earlier Jev-only controller reached its 20-minute limit against Lv2 without ever expanding.

Limits The two controllers also differ in candidate descriptions and execution handling. Opponents are not matched between controllers.

JEV-Star · Abstract; §4.2; Table 1

Across 35 SMAC-Hard maps with three episodes each, the combined system raised mean enemy elimination from 16.50% to 37.69% and wins from 3 to 7 out of 105.

Limits One shared plan per map; the three episodes are not independent planner samples.

JEV-Star · §4.4; Table 3

The reported 0.0% empirical prompt-injection bypass carries a 95% Wilson interval of 0.0–30.8%, which implies a very small number of evaluated vectors.

Limits Not a security guarantee.

Universal Fractal Decision Map · Abstract

On a rule-based task whose label is not recoverable from the text, the chosen level averaged 0.74 probability and a temperature of about 3.4 was needed to make probabilities honest; the returned probabilities were quantised to 0.01 and the gateway exposed no model version.

Limits Unpinned model alias; post-hoc temperature diagnostic.

scienthoon/jev-ood-calibration ↗ · README “Results” and synthetic-task section

The “193.6× faster, 444.6× cheaper” headline comes from the vendor’s workflow evaluations, whose reference answers average two external frontier models; the vendor expects these to be on the higher end of real-world gains and notes the workflows were written by its capabilities team.

Limits Reference answers are model outputs, not ground truth. Possible selection bias acknowledged by the vendor.

TypeSafe: Introducing System One Models & Jev ↗ · “Workflow evals” and “Nuance”