Local LLM Inference Costs More Than You Think: Hardware, Not Electricity, Drives the Bill

Apple's new Mac Studio M5 Ultra, announced August 25 and set to begin shipping September 22, restores the 256GB and 512GB memory configurations that the company had quietly removed earlier this year during a global LPDDR5 shortage — a shortage that simultaneously pushed an RTX 5090 graphics card from a $1,999 MSRP to street prices above $3,000, and drove DDR5 system RAM to prices five times above 2024 lows. The M5 Ultra's arrival makes this the right moment to examine the math that most local AI hardware discussions avoid: not whether frontier open-weight models can run on desktop hardware, but whether doing so makes economic sense when cloud APIs are offering the same models for a fraction of a dollar per million tokens.

The short answer: electricity is almost irrelevant to that decision. Hardware amortization accounts for 85 to 98 percent of every monthly local inference bill. And the cloud's cheapest options now cost nearly the same per token as local electricity alone — before counting a single dollar of hardware depreciation.

Why MoE Architecture Changed the Local Inference Math

Understanding the current economics requires understanding what Mixture-of-Experts architecture does to inference cost. A dense transformer activates all parameters for every token it generates. A MoE model maintains a large pool of specialized subnetworks and routes each token to only a small fraction of them. DeepSeek V4 Flash 0731 — which topped OpenRouter's weekly model-usage rankings through late August to claim the top global position — has 284 billion total parameters but activates roughly 13 billion per token.

This sparsity changes what hardware actually does during inference. The full 284 billion parameters must reside in memory (about 145GB at 4-bit quantization), but memory bandwidth consumed per token is proportional only to the activated subset. Memory bandwidth — not compute FLOPS or parameter count — is what determines generation speed. At 273 GB/s on the NVIDIA DGX Spark, theoretical decode throughput for a MoE model activating 3.5GB per token reaches around 78 tokens per second. Run a dense 70B model on the same hardware — where 40GB of weights must stream for every token — and that ceiling collapses to around 7 tokens per second. Real-world measurements from August 2026 confirm this: a single DGX Spark running DeepSeek V4 Flash measured approximately 26 tokens per second; a two-unit configuration reached 82.

Apple's M5 Ultra reaches 1.2 TB/s — 4.4 times the DGX Spark's 273 GB/s, and 4.7 times the 256 GB/s of AMD Strix Halo mini-PCs. For MoE models with small activated fractions, that bandwidth advantage narrows. For dense models, it translates almost linearly to generation speed. Apple's launch claim of 4.3x peak AI performance over M3 Ultra is a compute metric, not a generation speed figure — the company's own MLX benchmarks showed sustained generation improving only 19 to 27 percent, with prefill (the compute-bound input processing phase) improving up to 4x.

What Running a Model Actually Costs per Token, and per Month

At PG&E's E-TOU-C residential rate schedule — a blended off-peak cost of approximately $0.40 per kilowatt-hour — the marginal electricity cost of running a specific hardware and model pairing is straightforward to calculate. A 96GB Mac Studio running an MoE model with roughly 5 billion activated parameters, at 40 tokens per second and 120 watts, produces electricity costs near $0.09 per million mixed tokens (at a 3:1 input-to-output ratio). An RTX 5090 desktop running a 27B model at 500 watts and 45 tokens per second costs about $0.33 per million mixed tokens in electricity.

These figures are accurate for hardware already fully depreciated. They describe marginal cost. They tell you nothing about whether you should buy the hardware.

At four-year depreciation, a $3,300 Strix Halo mini-PC (128GB, Minisforum MS-S1 Max or equivalent) costs approximately $69 per month before any tokens are generated. A $5,000 RTX 5090 desktop runs about $104 per month. The $5,499 Mac Studio M5 Ultra at 96GB costs $115 per month. The 256GB configuration adds roughly $4,000 and reaches approximately $198 per month in depreciation alone. At 30 million mixed tokens per month — realistic for an active individual developer — the total monthly bills (amortization plus electricity plus idle power) run from $75 for the Strix Halo to $206 for the 256GB Mac Studio. Across all configurations, 85 to 98 percent of that bill is hardware depreciation. Electricity is a rounding error.

That cost structure also explains a counterintuitive result: adding memory to run better models increases, not decreases, the effective cost per token at moderate usage. The jump from a $5,499 Mac Studio (96GB) to the ~$9,500 256GB configuration roughly doubles monthly depreciation while generating the same number of tokens per session. The 256GB machine's advantage is capability — access to models that simply cannot load on 96GB hardware — not cost efficiency. Hardware upgrades in this market buy capability headroom, not lower unit economics.

Read more: NVIDIA AI Server Prices Rise as Memory Costs Hit the Stack

What Developers Are Actually Paying on OpenRouter

OpenRouter's weekly rankings through late August 2026 function as a revealed-preference experiment. The most-used model — DeepSeek V4 Flash 0731 — carries a list input price of $0.14 per million tokens on standard providers, but the cheapest providers on the platform currently undercut that to under $0.07 per million input, with output pricing as low as $0.10 per million. At a 3:1 mixed-token ratio and after OpenRouter's 5.5 percent credit-load fee, cost-optimized routing can bring the effective mixed price below $0.10 per million tokens. The #4 model by volume, GLM-5.3-Flash — publicly confirmed as the same model behind the previously anonymous Ox Alpha endpoint — carried a comparable price. Xiaomi's MiMo-V2.5 placed fifth in the same period.

What this means for the local hardware comparison is precise: cloud MoE pricing has converged with or fallen below the electricity cost alone of running the equivalent model on local hardware. The full-cost Mac Studio option at moderate volume costs substantially more than cloud routing — and that margin disappears entirely once you account for what cloud provides that local hardware cannot: a 1.31 million token context window and unlimited concurrent requests.

OpenRouter's prepaid model also addresses a concern about API costs being uncontrollable. Per-key credit limits with daily or monthly reset cycles, and model whitelists, mean a misconfigured agent loop that runs a five-dollar daily limit stops at exactly five dollars. The "local inference avoids runaway bills" argument applies only to the electricity line item, which — as the depreciation analysis shows — is not where the money goes.

Context Window: The Variable Most Hardware Comparisons Ignore

Local hardware's most significant structural weakness relative to cloud is not generation speed. It is context window. Models advertised with 128K or 256K context windows cannot achieve those lengths on most local hardware because KV cache — the memory structure storing computed attention states for the full context — grows linearly with context length and competes directly with model weights for the same physical memory.

Academic measurements put LLaMA-3.1-405B's KV cache requirement at approximately 123GB for a 128K context — more than the model weights themselves. A 96GB Mac Studio running a 120B MoE model has approximately 14GB of memory remaining after loading the model weights. That 14GB of cache headroom falls far short of the 128K context ceiling the model is advertised to support. Ollama enforces this reality through automatic context selection: under 24GB of GPU-accessible memory defaults to 4K context; 24 to 48GB defaults to 32K; 48GB and above defaults to 256K. The thresholds apply to GPU-accessible memory, not total unified memory — a 36GB Mac allocates roughly 24GB to the GPU under macOS.

The 256GB M5 Ultra is the first desktop configuration that changes this substantially. After loading a 145GB MoE model, approximately 60GB remains for KV cache — enough for practical long-context operation, though still well below the 1.31 million token windows available on cloud endpoints for the same models. At extended contexts, cache retrieval also increasingly dominates inference time: one 2026 measurement found attention operations consuming 84.7 percent of per-token latency at 128K context, rising to 97.9 percent at 1 million tokens.

When Local Inference Makes Structural Sense

None of the above eliminates the case for local inference. It clarifies what that case actually is.

The strongest scenario: hardware purchased for other primary purposes — video production, scientific computing, 3D rendering — where inference runs at marginal electricity cost only. At roughly $0.09 per million tokens in electricity, local inference beats every cloud option. The depreciation has already been allocated to a different budget line.

The second scenario: genuine data sovereignty constraints. Several of the most-used models on OpenRouter originate from Chinese AI labs. For organizations in regulated sectors where data cannot route through non-domestic endpoints — government, defense, healthcare, certain financial services — the 256GB Mac Studio is the first desktop configuration that can load DeepSeek V4 Flash's full 145GB weights locally. Whether it runs them efficiently will not be measurable by independent testers until the M5 Ultra ships September 22.

One technical development that changes the dense-model picture at the local level is worth noting separately. Multi-Token Prediction — a form of speculative decoding — landed in llama.cpp's mainline on May 16, 2026. On a dual RTX 5090 setup, MTP pushed a dense 27-billion-parameter Qwen model from 51 to 117 tokens per second, a 2.3x speedup achieved by predicting and verifying multiple tokens per forward pass rather than generating them sequentially. For dense models, where memory bandwidth constrains generation far more severely than for MoE models, MTP partially sidesteps the bottleneck. Local inference on unified-memory hardware for dense 70B models has consistently trailed cloud endpoints on speed; MTP narrows that gap without requiring a hardware upgrade.

Offline operation with deterministic latency, short-context high-frequency tasks like classification and extraction, and a fixed monthly cost regardless of token volume round out the genuine use cases. What does not survive full-cost analysis: the premise that purchasing dedicated hardware for inference is primarily a cost-saving move. At personal and small-team usage levels in September 2026, cloud MoE pricing has converged with or fallen below local electricity costs. Purpose-purchasing hardware to reduce token expenses requires volume above 1 billion mixed tokens per month before depreciation begins to close the gap — and at that volume, single-request serial inference itself becomes the bottleneck.

Read more: FreeToken Rewrites Local MoE Serving, 2x Faster Than Ollama on the Right Hardware

The memory crisis that reshaped hardware pricing throughout 2026 also clarified something about what local AI inference is fundamentally for. The value was never primarily in the electricity bill. It has always been in what you cannot, or choose not to, send to someone else's server — and increasingly, in which specific models you need to run while keeping data entirely under your own control. The benchmark that actually matters in 2026 is not kilowatt-hours per million tokens. It is whether the model you need, at the context length you need, can legally and practically run anywhere else at all.