Most self-hosted apps are a rounding error on a modern box. A local LLM is not. The moment you add Ollama or an inference server to a homelab, you've changed the machine's capacity math — and if you treat it like installing any other container, the model quietly takes the resources your other apps needed. Self-hosted AI is less an install and more a capacity decision you should make on purpose. (Once the host is sized, the per-service knobs — memory limits, timeouts — are covered in sizing self-hosted AI and search apps; this is the layer above that: the box itself.)
VRAM is the real currency for GPU inference
If you want responsive inference, you want a GPU, and the constraint is VRAM, not system RAM. The model's quantized weights must fit in VRAM (with room for context):
- ~8 GB VRAM comfortably runs 7B-class models quantized.
- ~12–16 GB opens up 13B-class.
- 24 GB+ is where larger models and longer context become practical.
Spilling to system RAM works but collapses throughput. Size the GPU for the largest model you actually intend to run, not the smallest one you'll start with.
CPU inference is viable — if you set expectations
No GPU? CPU inference runs, but it's seconds-to-tokens, not tokens-per-second. It's fine for occasional, non-interactive use (a nightly summarization job) and frustrating for a chat UI. Budget plenty of system RAM (the whole model sits in it) and accept the latency, or move the workload to a box that has a GPU.
Disk: models are quiet space hogs
Each model is multiple gigabytes, and it's easy to accumulate a dozen "just to try." Put the model cache on its own volume with room to grow, and prune unused models — the same disk-headroom discipline every stateful service needs. Reserve 20–30% free or the next pull fails mid-download.
Thermals and power are part of capacity
Sustained inference pins a GPU or CPU at load for minutes. In a mini PC or a closet rack that means heat and power draw your other workloads now compete with. Check that the box can run the model continuously without thermal throttling — a quick burst benchmark hides the steady-state problem. A real performance benchmark rather than the spec sheet is the honest way to know what the hardware sustains.
The real decision: shared box or dedicated?
This is the planning call self-hosted AI forces:
- Shared box — fine if the model is small, usage is occasional, and you've capped its resources so it can't starve your databases and media apps.
- Dedicated box — the right answer once inference is interactive, frequent, or GPU-bound. Isolating it protects everything else and lets you size the AI machine for AI without compromising the rest.
If you're consolidating other services to make room, watch that you don't just recreate database sprawl on the remaining box.
Capacity checklist before you deploy
- Target model size chosen; VRAM (or RAM for CPU) sized to fit it + context.
- Inference path decided: GPU for interactive, CPU only for batch.
- Model cache on its own volume with 20–30% headroom.
- Sustained-load thermals/power verified, not just a burst test.
- Shared-vs-dedicated decision made deliberately, with resource caps if shared.
Plan the capacity first and self-hosted AI slots cleanly into a homelab. Skip it and the model makes the capacity decision for you — usually at the expense of everything else on the box.