ScaleDown's New AI Models Skip Token Generation to Cut Classification Costs

ScaleDown released three production AI models on October 13, 2026 — Classify, Extract, and Summarize — built on what the company calls a Constrained Decoding Model (CDM) architecture that deliberately avoids the token-by-token generation loop that defines large language models. The launch, confirmed through integrations with LangChain, OpenRouter, Vercel, and LiteLLM, marks one of the clearest attempts yet to offer a structural alternative to general-purpose LLM APIs for bounded enterprise NLP tasks — and it arrives as a second company, TypeSafe AI, releases a parallel update to its own non-autoregressive decision model, Jev, adding experimental vision input and object localization to the same category of constrained-output AI.

Together, these developments suggest a maturing divergence within applied AI: one path continues scaling general-purpose models toward broader reasoning and multimodal capability, while another builds narrower, faster, cheaper systems optimized for the structured prediction tasks that make up the majority of enterprise AI volume — classification, extraction, and summarization at scale.

Why Autoregressive Generation Is the Wrong Tool for Bounded Tasks

To understand what ScaleDown is selling, it helps to understand what it is replacing. Standard large language models generate output one token at a time. At each step, the model scores every entry in a vocabulary of roughly 100,000 possible tokens, samples from that distribution, and feeds the result back into the next step. For open-ended writing, code generation, or complex multi-step reasoning, this iterative process is necessary — the model cannot know what word should follow until it has generated what came before.

For a task like classifying a customer service message into one of six categories, this mechanism is a significant mismatch. The answer is already constrained to a short list. The model still generates through its full vocabulary at every step, still processes the same number of forward passes through billions of parameters, and still returns a text string that an application must then parse back into a structured label. The overhead is real: latency scales with output length, and pricing for most LLM APIs is metered on both input and output tokens.

CDM architecture, as ScaleDown describes it, reads the input context once and uses a task-specific output head — rather than an autoregressive decoder — to select from a bounded result space. Technical analysis notes that for classification and extraction, the head reads the model's internal representation of the input and selects a label or span, producing calibrated probabilities for each candidate without iterating through the full vocabulary. For extraction, it identifies spans in the source text rather than regenerating their content as new tokens. This architectural choice has a direct pricing consequence: ScaleDown charges $0.05 per million input tokens (the official list price; some aggregators including OpenRouter currently display a discounted rate of $0.04 per million) and nothing for output — a pricing model that is structurally coherent only if output is not metered as generated text.

This positions CDM architecturally closer to BERT-era encoder models with classification heads than to modern instruction-following decoders. The key difference from "GPT-4o mini with JSON mode" is not model size but generation mechanism: both a fine-tuned small LLM and a large frontier model still generate output autoregressively, token by token. CDM, as described, does not.

The Three Models and What They Return

ScaleDown launched three models under the CDM architecture, each targeting a different structured prediction task. LangChain's integration blog post published on launch day confirms all three alongside a fourth pre-existing model, Compress.

Classify accepts developer-defined label sets and returns a calibrated probability score for each label alongside a human-readable reason for the classification. Calibrated probabilities — where a 0.8 confidence score corresponds empirically to 80% accuracy across similar examples — are a specific engineering property that matters in production: uncalibrated confidence scores from standard LLMs are unreliable signals for downstream decision-making. ScaleDown's claim to offer calibrated output is architecturally plausible given that task-specific heads can be post-hoc calibrated through temperature scaling or Platt scaling, though the company does not specify its calibration approach. Classify is intended for customer service routing, intent detection, and agent-dispatch decisions.

Extract takes a natural-language description of the information to find and returns structured JSON with confidence scores and source position — the character or token offset of the extracted span in the original document. Source position is a reliability feature: it allows applications to verify extraction results against the original text rather than trusting that the model reproduced the content accurately. Intended applications include named entity recognition across documents, contract clause extraction, and PII identification for data masking pipelines.

Summarize accepts text, images, or PDFs up to one million tokens and returns a summary bounded at 1,000 tokens. ScaleDown estimates that generating 10,000 summaries daily costs approximately $7.20 using its model versus roughly $58 per day using GPT-4.1 Mini — an 8× cost difference attributed to both pricing and the model's efficiency. The company reports in human evaluation that summary quality was comparable to GPT-4.1 Mini's output, though the evaluation methodology is not publicly detailed.

The Summarize model deserves a note of architectural caution. Generating a coherent 1,000-token summary of a complex document is not the same operation as selecting one of N labels or extracting a span from existing text. Independent technical analysis notes that Summarize generates text abstractively, which means it does not share the strictly non-autoregressive architecture of Classify and Extract. ScaleDown's pricing (output free) holds for all three models, but how it is achieved technically for Summarize specifically is not clarified in available documentation.

A fourth ScaleDown model, Compress — designed to trim agent context windows rather than generate summaries — was not part of the October 13 launch and is described separately.

The Pricing Structure Behind Free Output Tokens

ScaleDown's pricing of $0.05 per million input tokens with free output is worth examining as both a business model and a technical statement. Most LLM APIs price on a combined input/output basis because output tokens represent real compute: each generated token requires a full forward pass through the model. At $0.40 per million output tokens for GPT-4.1 Mini or roughly $3.00 per million for mid-tier Claude models, output costs dominate for any task that produces substantive text.

A non-autoregressive architecture that selects from a constrained output space — returning a label, a JSON object with a fixed schema, or a span pointer to an existing text position — does not generate new tokens in the traditional sense. The output computation is fundamentally different: the model reads the input, produces a representation, and the output head applies a learned mapping to a fixed-size result. There are no iterative forward passes over output positions, no sampling from vocabulary distributions, and therefore no economically meaningful per-output-token cost. Free output is a natural consequence of the architecture rather than a promotional pricing strategy.

This changes the economics of high-volume applications significantly. Consider a document processing pipeline that classifies 5 million customer support tickets per month. At $0.40 per million output tokens (assuming an average five-token output per classification), output cost alone on GPT-4.1 Mini would be approximately $2 per million classifications — or $10,000 monthly for output alone, before accounting for the larger input cost. On ScaleDown CDM, the effective cost at $0.05 per million input tokens (assuming each ticket averages 200 tokens) would be approximately $50 per million tickets, or $250 monthly for the entire workload. At that ratio, the economics make evaluation mandatory even for teams that are otherwise satisfied with their LLM provider. Pricing details are available through OpenRouter's ScaleDown listing.

The caveat is that these calculations assume the task fits cleanly within what CDM models can do. The moment the application requires explanation, reasoning about edge cases, or output outside a defined schema, CDM no longer applies and the LLM cost structure returns.

Integration Into the LLM Middleware Stack

The October 13 launch included simultaneous integration into LangChain, with the CDM models available through LangSmith's LLM Gateway alongside three middleware types: Summarization, ToolCompression, and Extraction. The ToolCompression middleware addresses a specific problem in long-running agent pipelines: tool results accumulate in the context window over multi-step tasks, increasing both token count and cost. By running CDM Summarize or Extract against tool outputs before they are appended to context, agents can maintain effective working memory without the cost of passing full tool results to a frontier model at every step.

This integration positions ScaleDown not primarily as a replacement for general LLMs in agent systems but as a pre-processing and post-processing layer — handling structured tasks at lower cost while frontier models handle the reasoning and decision-making steps that require broader capability. The practical deployment pattern is a hybrid: CDM handles classification, extraction, and context compression; a general model handles complex judgment calls.

ScaleDown also added support for OpenRouter, Vercel, and LiteLLM on launch day, giving developers access through standard OpenAI-compatible API interfaces rather than requiring a new SDK. The input context supports text, images, and PDFs, with the one-million-token window available across all three modalities under the Summarize model.

The Open-Weight Alternative: Fine-Tuned Small Models

Before evaluating CDM and Jev, enterprise teams typically face a three-way choice rather than a binary one. The third option is fine-tuning a small open-weight model on task-specific data and deploying it in their own infrastructure. Models like Qwen 0.5B, Llama 3.2 3B, or Gemma 2B are available under permissive licenses, require minimal GPU resources for inference, and can achieve strong performance on specific tasks when trained on labeled examples from the target domain.

This approach has real advantages over both frontier LLM APIs and specialized services like CDM. Fine-tuned open-weight models incur no per-call API cost after the initial training investment. They can be run on commodity hardware or even CPUs at small scales. They operate behind the organization's own firewall, eliminating data residency concerns that arise from routing sensitive documents to third-party APIs. And for tasks with stable, well-defined output schemas and sufficient labeled training data, fine-tuned small models routinely match or exceed the performance of much larger general models.

The disadvantages are operational rather than technical. Fine-tuning requires labeled training data, which is expensive to produce. It requires MLOps infrastructure for model versioning, deployment, and monitoring. When the task changes — new product categories, new entity types, new summarization length requirements — the model must be retrained. And at very small scale (under ~1 million calls per month), the operational cost of maintaining fine-tuning pipelines often exceeds the API cost it avoids.

ScaleDown's CDM positions itself at the intersection: the cost efficiency of a task-specific model without the operational overhead of self-hosting. The implicit claim is that CDM provides most of the benefit of a fine-tuned task-specific model while requiring only a schema definition rather than labeled training data and a deployment pipeline. Whether that trade-off holds in practice depends heavily on whether a developer's task matches ScaleDown's training distribution — a question that, like most benchmark-adjacent claims in this space, awaits independent evaluation.

ScaleDown's own benchmark data shows 91.7% accuracy for its CDM versus 88.6% for Qwen 0.5B on equivalent classification tasks, and claims performance comparable to Qwen 72B on the same tasks — a model roughly 144 times larger. These are vendor-reported figures and have not been independently reproduced. If accurate, they suggest CDM is competitive with fine-tuned small open-weight models without requiring the developer to run the fine-tuning step; the comparison against Qwen 72B, if confirmed, would argue that 144× more parameters add nothing for bounded classification tasks, which is architecturally plausible but requires independent verification.

TypeSafe AI's Jev Takes a Different Architectural Path

The CDM launch arrived one day before TypeSafe AI released Jev 1.14.0, extending its own non-autoregressive decision model into vision, object localization, and longer context. Jev and ScaleDown's CDM occupy similar market territory — both replace LLM generation with constrained structured output — but their architectures and design philosophies differ in important ways.

Read more: Jev 1.14 ships Box type for object localization and experimental image inputs

TypeSafe AI, which exited stealth in September 2026 with $40 million in seed funding led by DCVC at a reported $200 million valuation, frames Jev as a "System One Model" — a reference to the dual-process theory distinction between fast, automatic cognition and slow, deliberate reasoning. In this framing, large reasoning LLMs are System Two: they deliberate slowly and at high cost. Jev is System One: it is trained to make fast, reliable, bounded decisions with calibrated confidence.

The training mechanism differs from CDM. TypeSafe AI uses Reinforcement Learning for Calibrated Decisions (RLCD), which trains the model not just to select correctly but to produce confidence scores that accurately reflect empirical accuracy distributions. A Jev Choice output at 0.75 confidence is designed to correspond to a 75% empirical accuracy rate across similar examples, enabling downstream systems to act on or escalate based on confidence thresholds. The company was co-founded by Diogo Almeida, who co-authored InstructGPT and was a co-inventor of ChatGPT, alongside Erik Gafni and Sasha Sheng.

Jev 1.14.0 extends the model's output types and input modalities in three directions, all currently experimental. Composable image inputs allow developers to pass images as part of the context or as candidate answer examples — a gesture-recognition system, for instance, can provide sample images for "thumbs up" and "open palm" rather than describing those categories in text. The Box output type adds object localization: the model returns a single bounding box expressed as four edge coordinates plus a confidence score. This makes Jev usable for product identification, visual moderation, and shelf-space analysis tasks without routing to a full vision model. Long-context support, also experimental, extends to 262,000 tokens when accessed with the appropriate TypeSafe experimental header; the model's default context window is 64,000 tokens. TypeSafe AI has flagged all three capabilities with a "Danger Zone" designation, indicating they are not yet appropriate for production deployments.

Notably, TypeSafe AI disclosed that Jev 1.13 already contained the underlying vision capability — it was simply not exposed to users in that release. The deliberate decision to ship a capability internally before enabling it externally suggests that TypeSafe AI applied a quality gate to image input before public exposure, a practice less common among AI companies racing to announce multimodal features.

Where the Competitive Numbers Stand — and Where to Be Careful

ScaleDown released comparative performance data alongside the CDM launch. On its own benchmark comparing against Qwen 0.5B and Qwen 72B on CDM-relevant tasks, the company reports CDM achieved 91.7% accuracy against 88.6% for Qwen 0.5B — while Qwen 72B, a model roughly 144 times larger, performed at a similar or lower rate on those specific tasks. These figures illustrate a structural point: for tasks with constrained output spaces, a specialized model can outperform a much larger general model by not spending capacity on capabilities it does not need. They are, however, vendor-reported numbers with no independent reproduction.

Industry coverage has referenced DecisionBench, described as an independent third-party benchmark with 43 tasks, 23,900 test rows, and coverage across 28 domains, on which ScaleDown CDM reportedly scored 77.2% and Jev 1.13 scored 72%. These figures have not been independently confirmed through primary English-language sources or the DecisionBench publisher's own site, and should be treated as secondarily sourced claims pending direct verification.

Read more: Open-weight decision models challenge proprietary alternatives

ScaleDown's benchmark data also shows significant cost and accuracy advantages compared against Claude, GPT, and Gemini series on specific NLP tasks. On text classification, ScaleDown's vendor-reported figures show approximately 8% higher accuracy than Claude-series models at roughly 1/5,250th the per-call cost, an 8.72% accuracy advantage against OpenAI models at 1/89th the cost, and a 9% accuracy advantage against Gemini at 1/29th the cost. These comparisons were conducted by ScaleDown itself on tasks the company selected for its model's strengths, and compare its purpose-built classifier against frontier models evaluated in zero-shot or few-shot configurations. They do not measure general reasoning, code generation, or open-ended text quality. A fine-tuned frontier model on the same task would likely perform differently.

Calibration as a First-Class Property

One technical property shared by both CDM and Jev — and conspicuously absent from most general LLM APIs — is calibration: the alignment between a model's stated confidence and its actual empirical accuracy. ScaleDown's Classify model claims to return calibrated probabilities for each label; Jev's RLCD training is specifically designed to produce calibrated decision outputs.

Calibration matters in production because uncalibrated confidence scores are operationally useless. If a model returns a 0.9 confidence score for classifications that are actually correct only 60% of the time, the confidence score cannot be used to route uncertain cases to human review, to set quality thresholds for downstream processing, or to compare reliability across tasks. Most frontier LLMs, when prompted to estimate their own confidence, produce scores that are poorly calibrated: they tend toward overconfidence on familiar patterns and underconfidence on novel phrasings, and the relationship between stated confidence and actual accuracy varies across tasks and input distributions.

BERT-era encoder models addressed this by applying Platt scaling or isotonic regression to the output probabilities of classification heads after training, mapping raw logits to calibrated probability estimates. It is likely that ScaleDown uses a similar post-hoc calibration approach — temperature scaling applied to the output head's probability distribution — though the company does not specify its calibration methodology in available sources. TypeSafe AI's RLCD training approach trains calibration directly into the reward signal during reinforcement learning, which is a distinct approach that aims for calibration without requiring a separate post-hoc calibration step.

The commercial value of well-calibrated output is significant for enterprise applications. A customer service routing system that trusts confidence scores can automatically handle high-confidence classifications without human review while escalating low-confidence cases — reducing both cost and error rate simultaneously. A document extraction pipeline that receives calibrated confidence on extracted spans can flag low-confidence extractions for human verification rather than passing them silently to downstream systems. Calibration converts a model output from a binary decision into a continuous signal that integrates naturally with business-rule logic.

The Fastino Precedent: Commodity Hardware, CPU Inference

Before ScaleDown and TypeSafe AI converged on non-autoregressive architectures, Fastino established the commercial viability of a different approach to task-specific AI: training small task-specific language models on consumer gaming GPUs and running them in production on CPUs.

Fastino, which raised $7 million in pre-seed funding from M12 (Microsoft Ventures) and Insight Partners in November 2024 and a $17.5 million seed round from Khosla Ventures in May 2025 — Khosla being an early OpenAI investor — argues that the hardware cost of AI model development, typically dominated by GPU cluster spending, is not inherent to useful AI. Its TLMs are trained for under $100,000 in hardware, by the company's account, and can be deployed on CPUs without GPU infrastructure. Fastino's approach is autoregressive but at minimal scale: its models are small enough that the latency and cost profile approaches that of specialized non-autoregressive systems for enterprise applications.

The convergence of Fastino's TLMs, TypeSafe AI's RLCD, and ScaleDown's CDM across 2025 and 2026 reflects a shared market thesis: that a substantial fraction of enterprise AI workloads does not require — and cannot economically sustain — general-purpose frontier model inference.

Infrastructure, Latency, and the Agent Pipeline Case

ScaleDown's integration into the LangChain middleware stack reveals the primary deployment pattern its team is targeting: not standalone API calls for isolated tasks, but embedded processing layers inside multi-step AI agent pipelines.

A long-running agent that executes dozens of steps against external tools — searching, reading documents, calling APIs, comparing results — generates substantial context over time. By step 20 or 30, the accumulated tool outputs may be larger than the LLM's effective working-memory capacity, both degrading reasoning quality and increasing per-call cost as the full context is re-sent on each inference. The LangSmith LLM Gateway integration for ScaleDown's ToolCompression middleware offers a structural answer: CDM's Summarize or Extract models process tool outputs into compact representations before they are appended to the agent's context, keeping the frontier model's input window focused on decision-relevant information rather than the full text of every prior tool result.

This hybrid architecture — CDM for preprocessing and compression, frontier model for reasoning and orchestration — is not a replacement for the frontier LLM but a structural efficiency layer around it. The economic case depends on the ratio of preprocessing calls to frontier LLM calls: if an agent makes 30 tool calls per task and compresses each result with CDM before passing it to the reasoning model, the 30 CDM calls at $0.05 per million tokens represent a rounding error in the overall cost while potentially reducing frontier model input size by 50–70%, with a proportional reduction in frontier LLM cost.

Latency also matters in this configuration. If CDM processes input once without a token-by-token generation loop, it should be significantly faster than asking a frontier LLM to summarize or classify the same content — making it viable as a synchronous preprocessing step without introducing meaningful user-visible delay. ScaleDown's claim of up to 200× speed advantage over frontier models is vendor-reported and applies to its self-selected benchmark tasks, but the directional claim — non-autoregressive structured prediction is faster than autoregressive generation for bounded tasks — is architecturally sound.

For organizations with strict data residency requirements, the hybrid architecture introduces a data-flow question: CDM's API receives the full uncompressed tool output, which may contain sensitive enterprise data. Until ScaleDown publishes clear data processing agreements and regional data handling documentation, organizations subject to GDPR, HIPAA, or sector-specific data regulations need to evaluate whether routing document content through CDM's API is consistent with their compliance obligations. This is not a concern specific to ScaleDown — all third-party API providers require the same evaluation — but it is a prerequisite for enterprise adoption that does not yet appear prominently in ScaleDown's public documentation.

What Task-Specific Models Do Not Replace

The market framing of "task-specific models vs. LLMs" obscures the more useful engineering distinction. CDM architecture, as described by ScaleDown, is appropriate for tasks where the output space is already known: classification into predefined categories, extraction of predefined entity types, summarization of documents into predefined lengths. It requires that developers define the task schema in advance. It does not handle novel reasoning tasks, open-ended question answering, code generation, multi-step agent orchestration, or any situation where the appropriate output cannot be specified in advance.

Jev has the same structural constraint. TypeSafe AI's Choice, Score, and Boolean output types are powerful calibration tools within their defined scope, but they require that the developer know what question the model is answering before the answer is needed.

This means neither CDM nor Jev is an alternative to frontier LLMs in the general case. In agent pipelines, they function more naturally as preprocessing steps, routing layers, confidence gates, and output validators around a general model — or as direct replacements only for the subset of agent tasks that are structurally equivalent to classification or structured extraction.

The brittleness risk is real: a task-specific model trained on customer service intent categories will fail more abruptly than a general LLM when presented with inputs from outside its training distribution. A general model may hallucinate; a specialized model may confidently misclassify. The failure mode is different, and production teams need to account for input distribution shift as a first-class concern when deploying task-specific systems.

For the classification and extraction tasks where constrained decoding models are well-suited, the cost structure argument is strong. The cost gap between frontier LLM APIs and $0.05-per-million-token specialized models is large enough to be determinative in high-volume applications. Organizations running millions of classification calls per day are already building abstraction layers to route simpler queries to cheaper models; CDM gives that routing a new, cheaper destination with a structural rather than just a size-based cost advantage.

The Architecture Question Behind the Market Story

The broader significance of these launches is not pricing — it is architecture. The AI industry's dominant assumption since 2020 has been that scaling general-purpose decoder-only transformers is the right path toward better performance on essentially all tasks. That assumption has been productive: GPT-4, Claude 3 Opus, Gemini 1.5, and their successors do perform better on classification, extraction, and summarization than their predecessors. But performing better is not the same as performing optimally for those tasks, and the marginal capability gains from scaling a general model now compete with the structural advantages of task-specific design for bounded prediction problems.

ScaleDown's CDM represents a bet that the autoregressive decoder, for constrained-output tasks, is the wrong tool regardless of its scale — not because small models are smarter than large ones, but because the architectural choices that make a model useful for open-ended generation are unnecessary costs for tasks with known output schemas. TypeSafe AI's RLCD training approach makes a similar bet from a different direction: if the task is to produce a calibrated confidence score for a bounded decision, training with a calibration reward signal produces better-calibrated outputs than prompting a general model and asking it to estimate its own confidence.

Whether these architectures prove durable in production — or whether frontier models become cheap enough to displace them through pricing competition — remains an open empirical question. Google, Anthropic, and OpenAI have each moved to reduce the cost of their smaller model tiers significantly over the past year. At the current trajectory, the gap between frontier model pricing and CDM pricing could narrow faster than ScaleDown's enterprise customer relationships can deepen.

The independent benchmarks that would clarify where CDM genuinely outperforms alternatives — particularly against fine-tuned small open-weight models on the same tasks under equivalent evaluation conditions — have not yet appeared in publicly available sources. DecisionBench may provide that clarity, but its figures for these systems remain secondarily sourced. Until independent evaluators run controlled comparisons across the CDM, Jev, fine-tuned Qwen, and frontier baseline configurations, the performance claims remain commercially motivated and technically plausible rather than empirically settled.

What is already clear is that at $0.05 per million input tokens with free output, the cost argument for evaluating CDM on classification and extraction workloads is strong enough that most engineering teams should not need to wait for independent benchmarks to run a pilot. The architectural bet — that output-constrained tasks do not need autoregressive generation — has a clear enough technical rationale and a concrete enough pricing advantage to earn a place in the evaluation stack.

The next milestone to watch is not another model release but an independent benchmark — specifically one that compares CDM, Jev, fine-tuned open-weight models, and frontier LLM APIs under equivalent evaluation conditions and with disclosed task distributions. DecisionBench's 43-task, 23,900-row evaluation is designed for exactly this comparison, but its figures for CDM and Jev remain secondarily sourced in current reporting. When an independent evaluation under controlled conditions confirms or revises the performance gap between constrained decoding architectures and autoregressive baselines, it will either validate the design choices that ScaleDown and TypeSafe AI have committed to or identify the boundaries within which their advantages hold. In a market segment moving this quickly, that evidence matters more than any single company's benchmark release.