Course: Course 3 — LLM Fine-Tuning Masterclass Module: FT01 — VRAM Math: Can I Actually Run This? Duration: 45 minutes Level: Senior Engineer and above Prerequisites: FT00 (The Steering Stack)
After completing this module, you will be able to:
Here is the single most important fact about fine-tuning large language models in 2026: you will run out of VRAM long before you run out of FLOPs. A modern GPU can do the math for a 7B fine-tune in reasonable time. Whether you can hold the working set in memory is what decides whether the job runs at all. Compute is a budget; memory is a cliff. Step over the cliff and the job doesn't go slower — it OOMs and dies.
This is why VRAM math is a foundations module. FT00 gave you the mental model (the stack, the steering thesis). FT02 and FT03 will help you pick and select a base. But before any of that matters, there is a gatekeeper question: can you actually afford to load it? Get this wrong and you will spend a week curating a dataset for a model you cannot train, or rent an 8×H100 node for a job that would have run on a single 4090.
The cost framing makes the stakes concrete. Full fine-tuning a 7B model needs on the order of $50K of H100s (multiple 80GB cards, often a node). QLoRA fine-tuning the same 7B model runs on a $1,500 RTX 4090 — or free on a Colab T4 for the smaller bases. That is not a 20% difference. It is the difference between "I need procurement and a budget cycle" and "I start tonight." Most steering problems (FT00: format, preference, reasoning activation) do not need full fine-tuning. Most engineers who reach for full FT are paying 30× for capability they will never observe in the output. The VRAM math is what tells you which camp you are in.
A note on the numbers that follow. Every figure here is a field rule of thumb, sourced from the Introl infrastructure guide (Dec 2025), the Spheron PEFT guide, ApXML, the QLoRA paper, and HuggingFace PEFT docs. They will get you within ~20% of reality, which is exactly the resolution you need for capacity planning. They will not match your nvidia-smi to the megabyte — that depends on your context length, batch size, optimizer, and whether you turned on FlashAttention. The lab (artifact 07) builds a bottom-up estimator so you can see exactly where each gigabyte goes.
When you train, VRAM is eaten by three things. Memorize them. Every OOM is one of these three.
The parameters themselves, stored at whatever precision the base is held in:
| Precision | Bytes/param | 7B footprint | 70B footprint |
|---|---|---|---|
| FP16 / BF16 | 2.0 | ~14 GB | ~140 GB |
| INT8 | 1.0 | ~7 GB | ~70 GB |
| 4-bit (NF4 / GPTQ / AWQ / GGUF Q4) | 0.5 | ~3.5 GB | ~35 GB |
QLoRA freezes the base at 4-bit. LoRA (16-bit base) holds it at FP16/BF16. Full fine-tuning holds it at FP16/BF16 and also keeps an FP32 master copy for the optimizer. The precision choice for the base is the single biggest VRAM lever, which is exactly why QLoRA exists.
This is where full fine-tuning explodes. AdamW — the default optimizer — keeps, for every trainable parameter:
m (4 bytes)v (4 bytes)That is 16 bytes per trainable parameter before a single activation is stored. For a fully-trainable 7B model: 16 × 7B ≈ 112 GB, just for weights + gradients + optimizer states. Add activations and you land in the 100–160 GB range — multi-A100 territory. This is the line that separates "fine-tuning" (cheap, PEFT) from "full fine-tuning" (a datacenter event).
QLoRA and LoRA escape this because only the adapter parameters are trainable — typically 0.1–2% of the model. The optimizer states for an adapter are rounding error. The base weights stay frozen and contribute no gradient and no AdamW states. This is the entire economic argument for PEFT, and it is worth stating bluntly: full fine-tuning's cost is dominated by optimizer states that PEFT never materializes.
Activations are the intermediate values saved during the forward pass so the backward pass can compute gradients. They scale with sequence length, batch size, and number of layers — not just parameter count. This is the consumer you control with knobs:
Here are the field rules. Anchor on the concrete GB numbers — they are what you plan against.
| Method | Rule of thumb | 7B | 70B | Typical GPU class |
|---|---|---|---|---|
| QLoRA (4-bit base) | ~1.5–2× the 4-bit model size (shorthand); in practice 4-bit weights + ~6–10 GB overhead | ~10–16 GB | ~48–60 GB | 7B → RTX 4090 (24GB); 70B → 1× A100 80GB |
| LoRA (16-bit base) | ~2–3× the FP16 model size | ~18–30 GB | ~250–400 GB | 7B → 1× A100 40/80GB or 4090 with checkpointing |
| Full FT | ~16 bytes/param (weights + grads + AdamW) + activations; call it ~8–12× the FP16 footprint | ~100–160 GB | ~1.0–1.4 TB | 7B → multi-A100; 70B → 8–16× H100 multi-node |
Read that table once more. The spread between QLoRA (10–16 GB) and full FT (100–160 GB) for the same 7B model is roughly 10×. That is the price of optimizer states and full gradients. Almost no steering task (FT00) justifies paying it. Module FT10 is the dedicated full-FT-vs-PEFT decision; for now, internalize the default: start at QLoRA, escalate only with evidence.
A note on the QLoRA shorthand. "1.5–2× the 4-bit model size" reproduces the 7B number tolerably (4-bit 7B ≈ 3.5 GB → ~5–7 GB, plus overhead lands you near 10–16 GB) but gets looser at 70B because the activation overhead grows sub-linearly with parameter count. This is exactly why the lab builds a bottom-up estimator rather than trusting a single multiplier: the multiplier is a sanity check, the bottom-up sum is the plan.
Inference is easier because there are no optimizer states and no full gradients. VRAM is dominated by the weight footprint, with a smaller chunk for the KV cache and activations that grows with context.
Weights ≈ params × bytes-per-param.
Add roughly 1–2 GB for KV cache and activations at short context, and watch it climb as context grows (the KV cache scales with layers × context × hidden, linearly per token, but it adds up — a 70B at 32K context can spend 10+ GB on KV cache alone). This is why paged attention / KV-cache quantization exist in production servers (vLLM, Module FT20). For capacity planning during fine-tuning, the inference footprint is your floor — training always costs more than inference for the same model.
Five variables. Learn what each one buys you.
1. Context length (quadratic without FlashAttention, linear with it). This is the most underestimated cost in the field. Naive attention stores an N×N attention matrix per layer per head. Double the context and that matrix quadruples. At 8K context on a 7B model you may spend more VRAM on attention activations than on the weights. FlashAttention 2/3 fuses the attention computation so the N×N materialization never happens — memory goes from quadratic to linear, with a speed bonus as well. It is, in 2026, effectively mandatory for anything beyond toy context. If your training config does not have attn_implementation="flash_attention_2", treat that as a bug.
2. Batch size (linear — use gradient accumulation). Physical batch size multiplies activation memory linearly. The fix is gradient accumulation: run micro-batches of size 1 or 2 and sum gradients over N steps before the optimizer step. You get the effective batch size of N (the gradient noise profile you wanted) at the physical memory cost of 1. This is the single most common fix for a training OOM and it costs you nothing but wall-clock time.
3. Gradient checkpointing (~60–70% activation memory saved for ~30% slower). Selectively discards activations during the forward pass and recomputes them in the backward pass. gradient_checkpointing=True in your training args. It is the cheapest GB you can buy. Default to on for anything that does not fit. The ~30% slowdown is almost always worth it versus the alternative of not running at all.
4. Optimizer choice (AdamW 8-bit via bitsandbytes vs full AdamW). Full AdamW carries the 16 bytes/param bill. paged_adamw_8bit (bitsandbytes) quantizes the optimizer states to 8-bit, roughly halving the optimizer-state footprint at negligible quality cost. For QLoRA the optimizer states are already tiny (adapter only), so this matters most for full FT and for LoRA with large adapter ranks. It is essentially free quality-wise and should be your default unless you have a specific reason.
5. Precision of the base (the QLoRA lever). Going from FP16 (2 bytes) to 4-bit (0.5 bytes) quarters the weight footprint. For QLoRA this is the whole point. For LoRA you keep the base at FP16 (that is what defines LoRA-vs-QLoRA). For inference, 4-bit is usually the right default for deployment (Module FT19).
Before you open a model loader, answer three questions. They determine the GPU class.
That is it. Three numbers in, one GPU class out:
You will rarely need more precision than this for planning. The lab turns it into a function.
Do not overlook Apple Silicon. An M-series Mac has unified memory — the GPU and CPU share the same pool, so a 64GB Mac presents ~48–58 GB of usable GPU memory with no duplication. For small-model QLoRA (1B–7B) this is a genuinely viable training path, especially via MLX (Apple's framework) or PyTorch's MPS backend. A 7B QLoRA at ~10–14 GB fits comfortably on a 32GB Mac; a 1B QLoRA fits on a 16GB Mac.
The catch: MPS throughput is lower than a dedicated NVIDIA card, and not every kernel is optimized. But for iteration, debugging, and small steering experiments, a laptop you already own beats a cloud GPU you have to rent. For the course's orientation work and many PEFT experiments (Pillar 2), Apple Silicon is a first-class target, not a fallback.
The classic OOM. You plan for a 7B QLoRA at "2K context," compute ~12 GB, provision a 16 GB card, then load your real data and discover the longest sequences are 8K. Attention activations blow past the budget and the job dies at step 3. Always size for your 99th-percentile sequence length, not the mean, and confirm FlashAttention is on.
"We have a 7B model at 14 GB and an 80 GB card, so full fine-tuning will fit comfortably." It will not. Full FT carries ~16 bytes/param — ~112 GB before activations — because of the FP32 master copy and AdamW's two moments. The 14 GB inference footprint is irrelevant to the training budget. This mistake is how teams accidentally rent multi-GPU nodes they didn't budget for.
Treating VRAM as "just the weights." It is weights + optimizer + activations, and activations are the part that scales with your data (context, batch). A 7B QLoRA is ~3.5 GB of weights and ~6–10 GB of overhead, most of it activations. The weights are the floor; the activations are what actually OOMs you.
Paying 10× (in VRAM, in dollars, in time) for full fine-tuning when QLoRA would achieve the same steering result. Full FT is the exception, justified by specific evidence (Module FT10), not the starting point. Start at QLoRA.
Running without attn_implementation="flash_attention_2" on any job with context >2K. You are paying a quadratic memory bill and a speed penalty for no benefit. It is free, it is stable, turn it on.
| Term | Definition |
|---|---|
| The three VRAM consumers | Model weights, optimizer states + gradients, activations. Every training OOM is one of these. |
| AdamW states | The optimizer's FP32 master weight + first moment m + second moment v — 12 bytes/param on top of the FP16 weight + gradient (4 bytes), for ~16 bytes/trainable-param total. |
| QLoRA | LoRA on a 4-bit-quantized frozen base. 7B trains in ~10–16 GB. The default starting method. |
| LoRA (16-bit base) | LoRA with the base held at FP16/BF16. ~2–3× the FP16 model size to train. |
| Full fine-tuning | All parameters trainable. ~16 bytes/param + activations; ~100–160 GB for 7B. Multi-GPU only. |
| Gradient checkpointing | Discard-and-recompute activations; ~60–70% activation-memory cut for ~30% slower. |
| Gradient accumulation | Sum gradients over N micro-batches before the optimizer step; decouples effective from physical batch size. |
| FlashAttention 2/3 | Fused attention that drops the N×N materialization; memory goes quadratic → linear. Effectively mandatory. |
| AdamW 8-bit (bitsandbytes) | 8-bit optimizer states; roughly halves optimizer footprint at negligible quality cost. |
| The three-question framework | (1) model size, (2) full-FT vs PEFT, (3) context length → GPU class. |
| KV cache | Inference-time cache of key/value tensors; grows with context. The main non-weight cost at serve time. |
See 07-lab-spec.md — "The VRAM Calculator." You will build a Python function vram_estimate(model_params_b, method, context_len, batch_size) -> gb from first principles and validate it against three real jobs (7B QLoRA, 70B QLoRA, 7B full FT). The point is to make the rules of thumb your numbers, not something you look up.
paged_adamw_8bit, gradient checkpointing, and FlashAttention integration flags.# Module FT01 — VRAM Math: Can I Actually Run This? **Course**: Course 3 — LLM Fine-Tuning Masterclass **Module**: FT01 — VRAM Math: Can I Actually Run This? **Duration**: 45 minutes **Level**: Senior Engineer and above **Prerequisites**: FT00 (The Steering Stack) --- ## Learning Objectives After completing this module, you will be able to: 1. Explain why **memory, not compute, is the binding constraint** in fine-tuning — and state the three VRAM consumers (weights, optimizer states, activations) and which dominates for each method. 2. Apply the field's rules of thumb to **estimate training VRAM** for QLoRA, LoRA, and full fine-tuning, and place a job in the right GPU class (consumer card / single datacenter GPU / multi-GPU / multi-node). 3. Apply the **three-question framework** (model size × method × context length) to pick a GPU class before you touch a model loader. 4. Explain how **context length, batch size, gradient checkpointing, optimizer choice, and FlashAttention** move the numbers — and which knobs give you memory back at the cost of speed or quality. --- # 1.1 — The Problem: Memory Is the Binding Constraint Here is the single most important fact about fine-tuning large language models in 2026: **you will run out of VRAM long before you run out of FLOPs.** A modern GPU can do the math for a 7B fine-tune in reasonable time. Whether you can *hold the working set in memory* is what decides whether the job runs at all. Compute is a budget; memory is a cliff. Step over the cliff and the job doesn't go slower — it OOMs and dies. This is why VRAM math is a foundations module. FT00 gave you the mental model (the stack, the steering thesis). FT02 and FT03 will help you pick and select a base. But before any of that matters, there is a gatekeeper question: **can you actually afford to load it?** Get this wrong and you will spend a week curating a dataset for a model you cannot train, or rent an 8×H100 node for a job that would have run on a single 4090. The cost framing makes the stakes concrete. Full fine-tuning a 7B model needs on the order of **$50K of H100s** (multiple 80GB cards, often a node). QLoRA fine-tuning the same 7B model runs on a **$1,500 RTX 4090** — or free on a Colab T4 for the smaller bases. That is not a 20% difference. It is the difference between "I need procurement and a budget cycle" and "I start tonight." Most steering problems (FT00: format, preference, reasoning activation) do not need full fine-tuning. Most engineers who reach for full FT are paying 30× for capability they will never observe in the output. The VRAM math is what tells you which camp you are in. A note on the numbers that follow. Every figure here is a **field rule of thumb**, sourced from the Introl infrastructure guide (Dec 2025), the Spheron PEFT guide, ApXML, the QLoRA paper, and HuggingFace PEFT docs. They will get you within ~20% of reality, which is exactly the resolution you need for capacity planning. They will not match your `nvidia-smi` to the megabyte — that depends on your context length, batch size, optimizer, and whether you turned on FlashAttention. The lab (artifact 07) builds a bottom-up estimator so you can see exactly where each gigabyte goes. --- # 1.2 — Training VRAM: The Three Consumers When you train, VRAM is eaten by three things. Memorize them. Every OOM is one of these three. ### Consumer 1 — Model weights The parameters themselves, stored at whatever precision the base is held in: | Precision | Bytes/param | 7B footprint | 70B footprint | | --- | --- | --- | --- | | FP16 / BF16 | 2.0 | ~14 GB | ~140 GB | | INT8 | 1.0 | ~7 GB | ~70 GB | | 4-bit (NF4 / GPTQ / AWQ / GGUF Q4) | 0.5 | ~3.5 GB | ~35 GB | QLoRA freezes the base at 4-bit. LoRA (16-bit base) holds it at FP16/BF16. Full fine-tuning holds it at FP16/BF16 and *also* keeps an FP32 master copy for the optimizer. The precision choice for the base is the single biggest VRAM lever, which is exactly why QLoRA exists. ### Consumer 2 — Optimizer states and gradients (the one people forget) This is where full fine-tuning explodes. AdamW — the default optimizer — keeps, for every trainable parameter: - the FP16 weight (2 bytes) - the FP16 gradient (2 bytes) - an FP32 **master copy** of the weight (4 bytes) - the first moment `m` (4 bytes) - the second moment `v` (4 bytes) That is **16 bytes per trainable parameter** before a single activation is stored. For a fully-trainable 7B model: 16 × 7B ≈ **112 GB**, just for weights + gradients + optimizer states. Add activations and you land in the 100–160 GB range — multi-A100 territory. This is the line that separates "fine-tuning" (cheap, PEFT) from "full fine-tuning" (a datacenter event). QLoRA and LoRA escape this because **only the adapter parameters are trainable** — typically 0.1–2% of the model. The optimizer states for an adapter are rounding error. The base weights stay frozen and contribute no gradient and no AdamW states. This is the entire economic argument for PEFT, and it is worth stating bluntly: *full fine-tuning's cost is dominated by optimizer states that PEFT never materializes.* ### Consumer 3 — Activations (the one that scales with your data) Activations are the intermediate values saved during the forward pass so the backward pass can compute gradients. They scale with **sequence length, batch size, and number of layers** — not just parameter count. This is the consumer you control with knobs: - **Context length** is quadratic in attention's naive form. Double the context and attention activation memory roughly quadruples — *unless* you use FlashAttention (Section 1.4), which makes it linear. This is why FlashAttention is effectively mandatory, not optional. - **Batch size** is linear in activation memory. Use **gradient accumulation** to decouple the *effective* batch size (what your gradients look like) from the *physical* batch size (what fits in VRAM). - **Gradient checkpointing** trades compute for memory: it discards most activations during the forward pass and recomputes them during the backward pass. The rule of thumb is a **~60–70% cut in activation memory for ~30% slower** training. Turn it on by default for anything that OOMs; it is the cheapest GB you will ever buy. ### Putting it together: the rules of thumb Here are the field rules. Anchor on the concrete GB numbers — they are what you plan against. | Method | Rule of thumb | 7B | 70B | Typical GPU class | | --- | --- | --- | --- | --- | | **QLoRA** (4-bit base) | ~1.5–2× the 4-bit model size (shorthand); in practice 4-bit weights + ~6–10 GB overhead | **~10–16 GB** | **~48–60 GB** | 7B → RTX 4090 (24GB); 70B → 1× A100 80GB | | **LoRA** (16-bit base) | ~2–3× the FP16 model size | ~18–30 GB | ~250–400 GB | 7B → 1× A100 40/80GB or 4090 with checkpointing | | **Full FT** | ~16 bytes/param (weights + grads + AdamW) + activations; call it ~8–12× the FP16 footprint | **~100–160 GB** | **~1.0–1.4 TB** | 7B → multi-A100; 70B → 8–16× H100 multi-node | Read that table once more. The spread between QLoRA (10–16 GB) and full FT (100–160 GB) for the *same 7B model* is roughly **10×**. That is the price of optimizer states and full gradients. Almost no steering task (FT00) justifies paying it. Module FT10 is the dedicated full-FT-vs-PEFT decision; for now, internalize the default: **start at QLoRA, escalate only with evidence.** A note on the QLoRA shorthand. "1.5–2× the 4-bit model size" reproduces the 7B number tolerably (4-bit 7B ≈ 3.5 GB → ~5–7 GB, plus overhead lands you near 10–16 GB) but gets looser at 70B because the activation overhead grows sub-linearly with parameter count. This is exactly why the lab builds a bottom-up estimator rather than trusting a single multiplier: the multiplier is a sanity check, the bottom-up sum is the plan. --- # 1.3 — Inference VRAM (the simpler half) Inference is easier because there are no optimizer states and no full gradients. VRAM is dominated by the weight footprint, with a smaller chunk for the KV cache and activations that grows with context. **Weights ≈ params × bytes-per-param.** - FP16 7B ≈ **14 GB** - INT8 7B ≈ **7 GB** - 4-bit 7B ≈ **3.5–4 GB** (NF4) ; a **Q4_K_M GGUF** of 7B runs in **~6 GB** once you include runtime overhead. - 4-bit 70B ≈ **~35 GB** weights; runs in **~40 GB** with overhead → fits a single 48GB/80GB card. Add roughly **1–2 GB for KV cache and activations** at short context, and watch it climb as context grows (the KV cache scales with `layers × context × hidden`, linearly per token, but it adds up — a 70B at 32K context can spend 10+ GB on KV cache alone). This is why paged attention / KV-cache quantization exist in production servers (vLLM, Module FT20). For capacity planning during fine-tuning, the inference footprint is your *floor* — training always costs more than inference for the same model. --- # 1.4 — The Knobs: What Moves the Numbers Five variables. Learn what each one buys you. **1. Context length (quadratic without FlashAttention, linear with it).** This is the most underestimated cost in the field. Naive attention stores an `N×N` attention matrix per layer per head. Double the context and that matrix quadruples. At 8K context on a 7B model you may spend more VRAM on attention activations than on the weights. **FlashAttention 2/3** fuses the attention computation so the `N×N` materialization never happens — memory goes from quadratic to linear, with a speed bonus as well. It is, in 2026, effectively mandatory for anything beyond toy context. If your training config does not have `attn_implementation="flash_attention_2"`, treat that as a bug. **2. Batch size (linear — use gradient accumulation).** Physical batch size multiplies activation memory linearly. The fix is **gradient accumulation**: run micro-batches of size 1 or 2 and sum gradients over N steps before the optimizer step. You get the *effective* batch size of N (the gradient noise profile you wanted) at the *physical* memory cost of 1. This is the single most common fix for a training OOM and it costs you nothing but wall-clock time. **3. Gradient checkpointing (~60–70% activation memory saved for ~30% slower).** Selectively discards activations during the forward pass and recomputes them in the backward pass. `gradient_checkpointing=True` in your training args. It is the cheapest GB you can buy. Default to on for anything that does not fit. The ~30% slowdown is almost always worth it versus the alternative of not running at all. **4. Optimizer choice (AdamW 8-bit via bitsandbytes vs full AdamW).** Full AdamW carries the 16 bytes/param bill. **`paged_adamw_8bit`** (bitsandbytes) quantizes the optimizer states to 8-bit, roughly halving the optimizer-state footprint at negligible quality cost. For QLoRA the optimizer states are already tiny (adapter only), so this matters most for full FT and for LoRA with large adapter ranks. It is essentially free quality-wise and should be your default unless you have a specific reason. **5. Precision of the base (the QLoRA lever).** Going from FP16 (2 bytes) to 4-bit (0.5 bytes) quarters the weight footprint. For QLoRA this is the whole point. For LoRA you keep the base at FP16 (that is what defines LoRA-vs-QLoRA). For inference, 4-bit is usually the right default for deployment (Module FT19). --- # 1.5 — The Three-Question Framework Before you open a model loader, answer three questions. They determine the GPU class. 1. **How big is the model?** (Parameters: 1B / 3B / 7B / 8B / 13B / 70B / 405B.) 2. **Full FT, or PEFT (LoRA / QLoRA)?** This is the 10× cost lever. Default to QLoRA unless you have a specific reason (Module FT10). 3. **What context length do you need?** This is the quadratic lever and the most underestimated. Be honest — your training data's longest sequence sets the floor. That is it. Three numbers in, one GPU class out: - **1B, QLoRA, 4K context** → laptop / Apple Silicon / Colab T4 free tier. - **7B, QLoRA, 2–4K context** → RTX 4090 (24GB) or A100 40GB. The sweet spot for most real steering work. - **7B, LoRA (16-bit), 4K** → A100 80GB, or 4090 with aggressive checkpointing. - **7B, full FT** → multi-A100 (100–160 GB). The expensive path. - **70B, QLoRA, 4K** → 1× A100 80GB (~48–60 GB). - **70B, full FT** → 8–16× H100 multi-node (~1.0–1.4 TB). Nobody does this casually. You will rarely need more precision than this for planning. The lab turns it into a function. --- # 1.6 — Apple Silicon / MPS: The Viable Small-Model Path Do not overlook Apple Silicon. An M-series Mac has **unified memory** — the GPU and CPU share the same pool, so a 64GB Mac presents ~48–58 GB of usable GPU memory with no duplication. For small-model QLoRA (1B–7B) this is a genuinely viable training path, especially via **MLX** (Apple's framework) or PyTorch's MPS backend. A 7B QLoRA at ~10–14 GB fits comfortably on a 32GB Mac; a 1B QLoRA fits on a 16GB Mac. The catch: MPS throughput is lower than a dedicated NVIDIA card, and not every kernel is optimized. But for iteration, debugging, and small steering experiments, a laptop you already own beats a cloud GPU you have to rent. For the course's orientation work and many PEFT experiments (Pillar 2), Apple Silicon is a first-class target, not a fallback. --- ## Anti-Patterns ### Underestimating context length The classic OOM. You plan for a 7B QLoRA at "2K context," compute ~12 GB, provision a 16 GB card, then load your real data and discover the longest sequences are 8K. Attention activations blow past the budget and the job dies at step 3. **Always size for your 99th-percentile sequence length**, not the mean, and confirm FlashAttention is on. ### Forgetting optimizer states in full FT "We have a 7B model at 14 GB and an 80 GB card, so full fine-tuning will fit comfortably." It will not. Full FT carries ~16 bytes/param — ~112 GB before activations — because of the FP32 master copy and AdamW's two moments. The 14 GB inference footprint is irrelevant to the training budget. This mistake is how teams accidentally rent multi-GPU nodes they didn't budget for. ### Ignoring activation memory Treating VRAM as "just the weights." It is weights + optimizer + activations, and activations are the part that scales with *your data* (context, batch). A 7B QLoRA is ~3.5 GB of weights and ~6–10 GB of overhead, most of it activations. The weights are the floor; the activations are what actually OOMs you. ### Reaching for full FT by default Paying 10× (in VRAM, in dollars, in time) for full fine-tuning when QLoRA would achieve the same steering result. Full FT is the exception, justified by specific evidence (Module FT10), not the starting point. Start at QLoRA. ### Leaving FlashAttention off Running without `attn_implementation="flash_attention_2"` on any job with context >2K. You are paying a quadratic memory bill and a speed penalty for no benefit. It is free, it is stable, turn it on. --- ## Key Terms | Term | Definition | | --- | --- | | **The three VRAM consumers** | Model weights, optimizer states + gradients, activations. Every training OOM is one of these. | | **AdamW states** | The optimizer's FP32 master weight + first moment `m` + second moment `v` — 12 bytes/param on top of the FP16 weight + gradient (4 bytes), for ~16 bytes/trainable-param total. | | **QLoRA** | LoRA on a 4-bit-quantized frozen base. 7B trains in ~10–16 GB. The default starting method. | | **LoRA (16-bit base)** | LoRA with the base held at FP16/BF16. ~2–3× the FP16 model size to train. | | **Full fine-tuning** | All parameters trainable. ~16 bytes/param + activations; ~100–160 GB for 7B. Multi-GPU only. | | **Gradient checkpointing** | Discard-and-recompute activations; ~60–70% activation-memory cut for ~30% slower. | | **Gradient accumulation** | Sum gradients over N micro-batches before the optimizer step; decouples effective from physical batch size. | | **FlashAttention 2/3** | Fused attention that drops the `N×N` materialization; memory goes quadratic → linear. Effectively mandatory. | | **AdamW 8-bit (bitsandbytes)** | 8-bit optimizer states; roughly halves optimizer footprint at negligible quality cost. | | **The three-question framework** | (1) model size, (2) full-FT vs PEFT, (3) context length → GPU class. | | **KV cache** | Inference-time cache of key/value tensors; grows with context. The main non-weight cost at serve time. | --- ## Lab Exercise See `07-lab-spec.md` — "The VRAM Calculator." You will build a Python function `vram_estimate(model_params_b, method, context_len, batch_size) -> gb` from first principles and validate it against three real jobs (7B QLoRA, 70B QLoRA, 7B full FT). The point is to make the rules of thumb *your* numbers, not something you look up. --- ## References 1. **Dettmers et al. (2023)** — *QLoRA: Efficient Finetuning of Quantized LLMs*. arXiv:2305.14314. The 4-bit base + LoRA method that put 7B fine-tuning on a consumer GPU; the original VRAM measurements. 2. **Aghajanyan et al. (2020)** — *Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning*. arXiv:2007.07784. Why so few trainable params suffice — the theoretical license for QLoRA/LoRA's tiny optimizer footprint. 3. **Dao et al. (2022)** — *FlashAttention: Fast and Memory-Efficient Exact Attention*. arXiv:2205.14135. The fused-attention kernel that turns quadratic attention memory linear. 4. **Introl (2025)** — *LLM Training VRAM Requirements — Infrastructure Guide* (Dec 2025). The field rules of thumb for full-FT vs LoRA vs QLoRA footprints and GPU-class mapping. 5. **Spheron Network** — *PEFT / LoRA / QLoRA Resource Guide*. The multiplier rules of thumb (QLoRA ~1.5–2×, LoRA ~2–3×, full FT ~16–20×) and the activation-memory guidance. 6. **ApXML** — *Applied ML / LLM memory estimation reference*. The bytes-per-param tables and the optimizer-state accounting. 7. **HuggingFace PEFT & bitsandbytes docs** — `paged_adamw_8bit`, gradient checkpointing, and FlashAttention integration flags. 8. **Hu et al. (2021)** — *LoRA: Low-Rank Adaptation*. arXiv:2106.09685. The adapter method; the basis for the optimizer-footprint argument.