VRAM Math: Can I Actually Run This?

Module FT01 · Course 3 — LLM Fine-Tuning Masterclass

45 minutes · Cross-cutting foundations: memory, not compute, is the binding constraint

The gatekeeper question before every other technique in the course. The best steering method is useless if you can't afford to load the model.

Pillar 0 — Foundations

Memory is the binding constraint

You will run out of VRAM long before you run out of FLOPs.

Compute is a budget — slower is still progress.

Memory is a cliff — step over it and the job doesn't slow down, it OOMs and dies.

The cost framing: Full FT of 7B needs ~$50K of H100s. QLoRA of the same 7B runs on a $1,500 RTX 4090 — or free on Colab T4. That's not 20%; it's "procurement + budget cycle" vs "I start tonight."

The three VRAM consumers

Every training OOM is one of these three.

1. Model weights — params at stored precision. FP16 7B ≈ 14GB · 4-bit 7B ≈ 3.5GB. The precision lever (why QLoRA exists).

2. Optimizer states + gradients — AdamW: FP32 master + first moment m + second moment v + FP16 grad ≈ 16 bytes / trainable-param. 7B full FT ≈ 112GB before activations.

3. Activations — intermediate forward-pass values. Scale with context, batch, layers. The part you control with knobs.

The rules of thumb

Anchor on the GB numbers — they are what you plan against.

Method	Rule	7B	70B	GPU class
QLoRA (4-bit base)	~1.5–2× 4-bit size	~10–16 GB	~48–60 GB	7B → 4090 24GB 70B → 1× A100 80GB
LoRA (16-bit base)	~2–3× FP16 size	~18–30 GB	~250–400 GB	7B → A100 40/80
Full FT	~16 B/param + act	~100–160 GB	~1.0–1.4 TB	7B → multi-A100 (~$50K) 70B → 8–16× H100 node

QLoRA vs Full FT for the same 7B: a ~10× spread. The price of optimizer states + full gradients. Default: start at QLoRA, escalate only with evidence.

Why PEFT is cheap (the optimizer argument)

Full FT's cost is dominated by optimizer states that PEFT never materializes.

Full FT

All params trainable.

AdamW states over 7B params ≈ 112 GB.

+ FP16 weights + activations → ~118 GB total.

QLoRA

~1% params trainable (adapter).

AdamW states over adapter ≈ <0.5 GB.

+ 4-bit weights + activations → ~15 GB total.

Same model. The optimizer isn't. ~10× cheaper for the same steering result.

The three-question framework

Before you open a model loader — three numbers in, one GPU class out.

#	Question	Why it matters
1	How big is the model?	1B / 3B / 7B / 70B / 405B
2	Full FT, or PEFT (LoRA/QLoRA)?	The 10× cost lever. Default QLoRA.
3	What context length?	The quadratic lever — most underestimated.

1B QLoRA @ 4K → Laptop / Mac / Colab T4

7B QLoRA @ 2–4K → RTX 4090 24GB

7B Full FT @ 4K → multi-A100 (~$50K)

70B QLoRA @ 4K → 1× A100 80GB

The five knobs that move the numbers

Knob	Effect	Cost
Base precision (FP16→4-bit)	~4× less weight memory	minor quality (the QLoRA lever)
Full FT → QLoRA	optimizer states ~98GB → <0.5GB	~10× cheaper (the big one)
Gradient checkpointing	~60–70% activation cut	~30% slower
Gradient accumulation	physical batch → 1 (act scales down)	wall-clock only
FlashAttention 2/3	quadratic → linear attention	free (also ~20–30% faster)
AdamW 8-bit (bitsandbytes)	~halves optimizer states	negligible quality

Rank by leverage: method choice (10×) > precision (4×) > checkpointing (~3× activations) > batch/accum > optimizer.

Context length: the quadratic cliff

Without FlashAttention

Naive attention stores an N×N matrix per layer per head.

Double context → 4× memory.

At 8K on a 7B, attention activations can EXCEED the weight footprint.

With FlashAttention 2/3

Fused — the N×N never materializes.

Memory goes linear. Plus ~20–30% faster.

Effectively mandatory. If attn_implementation ≠ flash_attention_2, it's a bug.

Always size for the 99th-percentile sequence length, not the mean. The classic OOM: budget for 2K mean, hit 8K tail, die at step 3.

Inference VRAM (the simpler half)

No optimizer states, no full gradients. Weights dominate; KV cache + activations grow with context.

Format	Bytes/param	7B footprint	Runs in
FP16	2.0	~14 GB	—
INT8	1.0	~7 GB	—
4-bit (NF4)	0.5	~3.5–4 GB	—
Q4_K_M GGUF	~0.5 + overhead	~6 GB total	laptop / Ollama
Q4 70B	0.5 + overhead	~40 GB total	1× 48/80GB card

Add ~1–2 GB for KV cache at short context (climbs fast — a 70B at 32K can spend 10+ GB on KV cache). Inference is the floor; training always costs more for the same model.

Apple Silicon: a first-class small-model path

M-series Macs have unified memory — GPU & CPU share one pool.

A 64GB Mac presents ~48–58 GB of usable GPU memory with no duplication.

7B QLoRA (~10–14 GB) → fits a 32GB Mac
1B QLoRA → fits a 16GB Mac
Via MLX or PyTorch MPS backend

The catch: lower throughput than a dedicated NVIDIA card, and not every kernel is optimized. For iteration, debugging, and small steering experiments — a laptop you own beats a cloud GPU you rent. First-class target for PEFT, not a fallback.

Anti-patterns

Underestimating context length. Budget for 2K mean, hit 8K tail, OOM at step 3. Naive attention is N×N. Size for the 99th pctile; turn on FlashAttention.

Forgetting optimizer states in full FT. "14GB weights, 80GB card, full FT fits." No — full FT is ~16 bytes/param = ~112GB before activations. The inference footprint is irrelevant to the training budget.

Ignoring activation memory. VRAM = weights + optimizer + activations. A 7B QLoRA is ~3.5GB weights and ~6–10GB overhead (mostly activations). The weights are the floor; activations are what OOMs you.

Reaching for full FT by default. Paying 10× for capability you won't observe. Start at QLoRA.

What you can now do

Explain why memory, not compute is the binding constraint — and name the three consumers.
Apply the rules of thumb to place a job in the right GPU class (7B QLoRA → 4090; 7B full FT → multi-A100).
Use the three-question framework — model size × method × context — to pick a card before loading.
Name the five knobs and rank them by leverage (method > precision > checkpointing > batch > optimizer).

The lab: build a vram_estimate() function and validate it against three real jobs (7B QLoRA, 70B QLoRA, 7B Full FT). Make the numbers yours.

Next: FT02 — The Open Spectrum · Can I audit the base?