One GPU, Zero API Bills: ComfyUI Now Runs Local LLMs Without OOM Errors

Running a local large language model and a full image generation pipeline on the same GPU has long been an exercise in frustration. Load Flux or SDXL into VRAM, start Ollama for prompt generation, and the result is a predictable out-of-memory crash. The standard advice was blunt: buy a second GPU, or pay for API tokens. That calculus has changed.

ComfyUI's March 2026 release quietly enabled Dynamic VRAM by default — a custom memory allocator that makes the GPU share its physical memory cooperatively across workloads. Combined with a mature set of community-built VRAM handoff nodes, practitioners can now run a complete local AI pipeline — LLM for prompt generation, diffusion model for image creation — on a single consumer card without spending a dollar on inference APIs.

Why One GPU Was Never Enough Before

The collision happens at the hardware level. A 24GB GPU looks spacious until both competitors show up: a 7B-parameter LLM quantized to 4-bit occupies roughly 4–5GB of VRAM; a Flux Dev model in fp8 takes another 10–16GB. The two models cannot coexist on the same card when both are resident, and standard GPU memory management provides no graceful way to arbitrate between them. The result was chronic OOM crashes, forcing users to restart ComfyUI between LLM and diffusion steps, or run two machines.

How Dynamic VRAM Solves the Allocation Problem

The architectural fix arrived in early March 2026 when Comfy.org enabled Dynamic VRAM as the default memory mode. The system introduces a component called aimdo — the AI Model Dynamic Offloader — which replaces PyTorch's standard VRAM allocator with a demand-paging model tailored to generative workloads.

Each loaded model receives a VBAR (Virtual Base Address Register): a reservation of GPU virtual address space that costs no physical VRAM at load time. Model weights inside a VBAR begin unallocated; they are promoted to physical VRAM only when a layer actually needs them, through a fault() call at execution time. If VRAM is available, the layer stays resident and benefits subsequent runs. If the allocator is under pressure, it evicts lower-priority weights to make room, using a watermark system where the most recently active model holds the highest memory-retention priority and is the last to have its physical pages freed. The GPU never crashes with an OOM error; instead it temporarily copies a needed layer's weights, runs the operation, and frees or reuses the copy.

The practical effect for LLM+diffusion workflows: ComfyUI's diffusion model can remain nominally "loaded" while the LLM is using the GPU, because the diffusion weights are simply uncommitted in virtual space. The OS memory-maps the model file without reading it into RAM, using disk-backed memory that is reclaimed when genuinely needed and reloaded transparently from storage.

Benchmarks published by Comfy.org show improved generation performance on an RTX 5060 and an RTX Blackwell 6000 Pro under the new system, though these figures are company-reported and have not been independently reproduced. Dynamic VRAM currently supports NVIDIA GPUs on Windows and Linux only; AMD GPU support is listed as a roadmap item, and WSL is not supported.

The Handoff Node Ecosystem That Makes It Practical

Dynamic VRAM handles passive coexistence, but active handoff — explicitly releasing one application's VRAM before the other starts — produces cleaner, more reliable results. A community ecosystem of dedicated nodes has grown around this exact use case.

Setmaster's comfyui-llamacpp represents the most architecturally complete approach: it runs the LLM in a separate llama-server process rather than inside ComfyUI's Python runtime, giving it deterministic lifecycle control. The bidirectional handoff works in both directions — an optional unload_comfy_models_before_start flag evicts ComfyUI's managed models before the LLM allocates GPU memory, and a matching Release llama.cpp VRAM node stops the server process when image generation begins. The pack supports GGUF models across CUDA, Vulkan, ROCm, and Metal backends, with vision models handled via matching mmproj projectors. The release defers if generation is active and waits for all leases to expire before actually freeing memory, avoiding the half-released state that causes cascading OOM errors.

For users running external Ollama or vLLM servers, zyd232's LLM Unload node sends an explicit unload signal to the running server and blocks until the model actually disappears from memory — not just until the HTTP request returns. ComfyUI-llamacpp-Unloader takes a similar approach for standalone llama.cpp servers, calling the /models endpoint to identify loaded models and issuing targeted unload calls for each.

For workflows where simplicity matters more than surgical control, SeanScripts/ComfyUI-Unload-Model provides passthrough "Unload Model" and "Unload All Models" nodes that can be dropped anywhere in the graph to free VRAM at a natural handoff point, such as after a CLIP encoder step and before the main diffusion sampler.

For those preferring a containerized approach, llmc wraps llama.cpp, ComfyUI, and Open WebUI behind a reverse proxy that routes OpenAI-compatible requests to the LLM and image requests to ComfyUI, enforcing GPU exclusivity automatically — only one service runs at a time, with mode switching via a CLI command.

What a Single-GPU Workflow Looks Like Now

The practical pipeline that was previously impossible on consumer hardware is now well-documented: a ComfyUI workflow loads a GGUF-quantized LLM through a llama.cpp node, generates a detailed text-to-image prompt from a simple description or input image, then issues a handoff — unloading the LLM and optionally triggering ComfyUI's internal cache clear — before the diffusion sampler loads Flux or SDXL and produces the final image.

A 7-billion-parameter model such as Qwen2.5-7B quantized to 4-bit (approximately 4–5GB VRAM) can generate a prompt, unload, and hand off to a Flux Dev fp8 model (approximately 10–16GB) on a single RTX 3090 or RTX 4090 without crashing. The latency cost is real — the model reload on each run takes several seconds — but it eliminates the recurring per-token cost entirely and keeps all data local.

Limitations and What Remains Unresolved

Several constraints remain. Dynamic VRAM's VRAM consumption profile has changed: because weights are promoted to physical memory on demand and may remain resident across runs, apparent VRAM usage can be higher than the previous behavior, which sometimes alarms users monitoring GPU memory without understanding the new allocator semantics.

Node placement errors are also a persistent failure mode. Placing an LLM unload node upstream of the LLM step in a workflow causes the model to be evicted before it has executed — a wiring mistake that the nodes cannot prevent by design. Users learning the LLM+ComfyUI pattern need to understand that handoff nodes are outputs, not inputs: they fire after the LLM step completes, not before.

AMD GPU users remain excluded from Dynamic VRAM's benefits for now. The community node ecosystem works independently of Dynamic VRAM for explicit handoff workflows, but the passive coexistence layer that aimdo provides is NVIDIA-only until further hardware support ships.

The broader question is whether sequential GPU timesharing — however well-implemented — represents the right long-term architecture, or whether purpose-built unified pipelines (models jointly designed for text and image generation on shared memory) will eventually make the handoff approach obsolete. For now, the handoff ecosystem delivers practical results with available hardware, and that matters to the solo creator or small studio running a single consumer card as their entire production infrastructure.