InternLM Open-Sources Multimodal Decision Models That Outperform Jev at 90% Accuracy

Shanghai AI Lab's InternLM team published the technical report for Intern-Decision v1 on October 15 as an unreviewed preprint on arXiv, releasing the first open-weight decision model family with a natively trained multimodal architecture capable of returning structured choices and per-candidate probabilities from both text and images. The family — available at 0.8B, 2B, and 4B parameters under an Apache-2.0 license — arrives as the decision-model category crystallizes into a recognized segment of AI infrastructure, with TypeSafe AI's Jev and Microsoft's Decision-1 having shipped within the same two-week window. The 4B model scores 90.02% unweighted average accuracy across seven evaluation suites, beating Jev v1.13.0 at 88.74% on the same benchmarks, while delivering a mean inference latency of 44 milliseconds on a single RTX 4090 — compared with 109 milliseconds for Jev's API endpoint under equivalent testing conditions.

The practical consequence for AI engineers is a self-hostable decision model with no per-call fee, natively trained multimodal input support, and latency low enough to embed inside real-time agent loops without a dedicated inference farm.

What a Decision Model Is and Why It Is Not a Generative LLM

Standard large language models generate free-form text token by token. A decision model returns a typed, structured output — a named option, a numerical score, or a binary yes/no — along with probability estimates over each candidate. The difference is not cosmetic. In an agent pipeline that routes calls, classifies inputs, or approves actions at high frequency, a general-purpose LLM incurs unnecessary latency and token overhead generating prose that is then parsed downstream. Decision models skip the generation step and produce probabilities directly.

Intern-Decision achieves this through an approach called autoregressive masked language modeling (AMLM). During training, the model receives a complete assistant response skeleton — a JSON structure that already contains the decision fields — with <decision> placeholder tokens substituted for each answer. Ground-truth answer symbols appear only in the supervision labels, not in the input context. Every field is evaluated in a single causal forward pass using the backbone's native language model head, with no separate decode loop. The result is that the model assesses all decision options in one inference step rather than generating them sequentially, as described in the Intern-Decision v1 technical report.

The three supported question types — choice for named options, score for numerical ranges, and noul for yes/no binary — cover the decision classes that appear most frequently in structured agent workflows.

How Intern-Decision Compares with Jev and Microsoft-Decision-1

The decision-model category went from a TypeSafe AI niche product to a three-vendor contest inside two weeks. TypeSafe's Jev v1.13.0 is the established commercial leader, built around a training method the company calls Reinforcement Learning for Calibrated Decisions (RLCD), priced at $0.042 per million input tokens with output free, and available to enterprise customers via waitlist. Microsoft-Decision-1 went generally available on Microsoft Foundry on October 9, post-trained on a Qwen3.5-9B backbone, priced identically to Jev at $0.042 per million input tokens. It processes text only, carries a 32K context window, and Microsoft has not yet published accuracy benchmark scores for it.

Intern-Decision-4B, in InternLM's own testing, scores 90.02% average accuracy versus Jev's 88.74% across the same seven benchmarks — a lead driven primarily by the Typed Decision suite (80.55% vs. 73.35%) and ToolACE (96.45% vs. 91.29%). On Jevbench-Hard, the most discriminating of the three Jevbench splits, Intern-Decision-4B reaches 73.87% versus Jev's 72.07%. Jev retains a notable lead on WildJailBreak, the adversarial robustness suite: 96.29% versus Intern-Decision-4B's 89.86%, a gap worth noting for deployments where prompt injection resistance matters. The full benchmark results are available in the Intern-Decision GitHub repository.

The latency comparison is starker. At 44 milliseconds mean on a single consumer-grade RTX 4090, the 4B model runs at roughly 40% of Jev's 109-millisecond API mean under equivalent 289-token, three-field requests. The 0.8B and 2B models run at approximately 33–34 milliseconds each. InternLM measured Jev's latency against its own API endpoint; Jev has not independently confirmed these figures.

Intern-Decision was the first decision model in the category with a natively trained, production-stable multimodal architecture. Fine-tuning freezes the vision tower and projector inherited from the Qwen3.5 vision-language architecture and updates only the language backbone — a design choice that preserves multimodal capability while limiting the training footprint. Microsoft-Decision-1 remains text-only. Jev v1.14.0, released October 14 — one day before the Intern-Decision technical report — added experimental image inputs under a TypeSafe-Danger-Zone-Images: true opt-in header, which TypeSafe describes as not yet meeting its production quality standards; the benchmarks in this article compare against Jev v1.13.0, the version available during evaluation.

Read more: TypeSafe Jev 1.14 ships experimental image inputs and Box type

Calibration and Confidence-Based Routing

Decision models that report calibrated probabilities — where a stated 80% confidence actually corresponds to 80% empirical accuracy — are more useful in agent pipelines than accurate-but-overconfident ones, because calibrated confidence scores can be used to gate escalation to a larger model.

InternLM applies temperature scaling after training: a scalar T is fitted by minimizing negative log-likelihood on a held-out calibration set, then applied to previously saved candidate logits without any retraining. The process does not change any argmax decisions — the model's actual choices remain identical — but it compresses overconfident probability distributions closer to the empirical accuracy curve. On a separate 96-case known-distribution pilot, temperature scaling reduces Expected Calibration Error from 0.213 to 0.089 and Brier score from 0.628 to 0.550 for the 4B model. On the full Jevbench-Hard benchmark, the calibrated Intern-Decision-4B achieves an ECE of 0.065, below Jev v1.13.0's 0.095 on the same suite.

The tech report validates this calibration in a routing study on 111 Jevbench-Hard items. Routing the 42 items where Intern-Decision-4B's confidence fell below a threshold to a larger model (Atria-Dawn-Preview) raised overall accuracy from 73.87% to 87.39% on the subset. The implication is architectural: compact decision models function not as standalone replacements for capable general LLMs but as efficient first-pass filters that escalate uncertain cases. At 44 milliseconds mean latency, the cost of that first pass is low enough to insert it into nearly any agent loop without measurable user-facing delay.

What the Benchmarks Cannot Confirm

Several caveats limit how strongly these numbers should be interpreted. All benchmark results come from InternLM's own evaluation pipeline. Jevbench was designed by TypeSafe AI — the commercial competitor whose product Intern-Decision beats on it — creating a structural conflict of interest that independent reproduction would resolve. No third-party evaluation of Intern-Decision has appeared in the three weeks since the models were uploaded quietly to HuggingFace on September 26. The tech report was submitted to arXiv three weeks after the weights became available, and the models accumulated only a small number of GitHub stars in that window, suggesting limited early testing outside the lab.

The latency figures measured against Jev's API endpoint reflect InternLM's testing conditions, not a jointly controlled benchmark. API latency varies with server load, geographic routing, and batching behavior in ways that a single measurement run does not capture.

Open Weights, Multimodal Input, and the Cost Arithmetic

The economics of Intern-Decision differ categorically from its commercial competitors. Under Apache-2.0, the models are free to run. On a single RTX 4090 — a GPU available for under $2,000 on the consumer market or at modest cloud spot pricing — the 4B model processes decision requests at approximately 44 milliseconds each, implying throughput of roughly 20–25 requests per second per card before batching optimizations. At Jev's $0.042 per million input tokens, a 289-token request costs approximately $0.000012 per call; at 20 requests per second sustained, that totals roughly $21 per day in API fees. A self-hosted setup amortizes the hardware cost in days for sustained workloads.

The multimodal input capability opens use cases that production-stable text-only alternatives cannot yet match: visual quality assessment, image-based routing, document classification from scanned inputs, and any agent context that includes screenshots or diagrams. Microsoft-Decision-1 remains text-only. Jev's experimental image support, introduced the same week under a danger-zone opt-in, has not been cleared for production use — TypeSafe rates its multimodal inference as more variable than its text-only path and withholds the usual stability guarantees. Intern-Decision's multimodal capability ships as a first-class output, trained natively into the model architecture, with no experimental qualifier.

Read more: vLLM-SR releases Decision 3.0, an open multimodal suite topping Jev

What Comes Next for the Decision-Model Category

The rapid convergence of Jev, Microsoft-Decision-1, and Intern-Decision into the same two-week window is the clearest signal yet that structured decision inference is becoming standard AI infrastructure rather than an experimental niche. The open question is whether independent benchmarks will confirm InternLM's accuracy and latency claims, and whether the WildJailBreak gap — where Jev's 96.29% versus Intern-Decision-4B's 89.86% suggests meaningfully better adversarial robustness — reflects a durable difference in safety-relevant fine-tuning or a gap that additional data could close.

For developers evaluating options today, the combination of Apache-2.0 weights, multimodal input, sub-50-millisecond latency on commodity hardware, and benchmark-comparable accuracy positions Intern-Decision as the first viable self-hosted alternative to the commercial decision-model APIs — provided those benchmarks hold under independent evaluation.