Training a model from scratch is time consuming and can be expensive, which is why many teams rely on fine-tuning — take an open-weight base model that already knows English and how to reason, and teach it your domain, your tone, and your data. Parameter-efficient fine-tuning (PEFT) methods like LoRA (Low-Rank Adaptation) go one step further: freeze the base model entirely and train only a small set of extra weights — the adapters. Each adapter attaches to one of the linear layers inside a transformer block — its projections, in the language of the architecture — and works by taking the high-dimensional representation flowing through that layer, projecting it down into a much smaller space, adjusting it there, and projecting it back up. Some information is lost in the low-rank detour, but enough survives to meaningfully adapt the model, and because only the adapter parameters train, the trainable count is a small fraction of the base: a billion-parameter base ends up with roughly six million trainable parameters. Less than one percent. Going from full model training to fine-tuning changes how we think about each line item in the memory budget. The static costs that dominated the training article — gradients, optimizer state — become a rounding error. A new line item appears: the frozen base itself, whose weights still have to reside in the card's memory even though they never receive an update. This article covers fine-tuning on a single GPU with LoRA. Full fine-tuning (updating every weight, no adapters) uses the training article's math directly — this article's LoRA-specific accounting doesn't apply. Multi-GPU strategies will be covered in future series. This article builds directly on the training article's four-line-item framework. The definitions for weights, gradients, optimizer states, activations, activation checkpointing, and the BF16 weight cache all still apply here. Readers who haven't read the training article should start there — the fine-tuning story is much shorter when you don't have to reconstruct the baseline. When sizing GPU memory for a LoRA fine-tune, two line items dominate: activations and the frozen base. Together they hold 11.2 GB in the worked configuration below. Everything else combined is less than 1% of the total memory budget. The full memory picture All figures below use one worked configuration, carried through the article: TinyLlama-1.1B base (1.10 billion frozen parameters, 22 layers, hidden dimension 2,048, 32 attention heads, 4 key-value heads for grouped-query attention, feed-forward width 5,632, vocabulary 32,000), LoRA rank 8 on all seven projection targets — Q, K, V, O (the four attention projections) and gate, up, down (the three MLP projections) — across all 22 layers (6.3 million trainable parameters, 0.57% of the base), sequence length 512 tokens, batch size 8, BF16 base storage, BF16 mixed precision, standard Adam. Rank sets the adapter's capacity (8 is the LoRA paper's default). Figures labeled measured come from a 23-configuration benchmark of this model on an H100 80GB — see the note on measurement at the end. Unlabeled figures in tables are calculated.    Line item No checkpointing With checkpointing Levers to reduce it Frozen base weights 2.20 GB 2.20 GB INT8 base (halves it); INT4 base — QLoRA — quarters it; smaller base Adapter weights 0.025 GB 0.025 GB — Gradients 0.025 GB 0.025 GB — Optimizer states 0.050 GB 0.050 GB — Autocast weight cache 0.013 GB 0.013 GB — Activations ~9.0 GB ~2.4 GB (at their own peak) Activation checkpointing; smaller batch; shorter sequence; adapters only on upper N layers Matrix-multiply workspace ~1.5 GB released under checkpointing — Peak allocated (measured) 12.80 GB 4.58 GB   Allocator overhead: cached blocks and fragmentation (measured) ~1.7 GB ~1.0 GB   Total to size against (measured reserved) * 14.49 GB 5.57 GB   * Add 0.6 GB on top for the CUDA context — driver overhead that lives outside the allocator's pool, same fixed cost the training article accounted for. The frozen base is the largest static number on the page, and it's the one line item that has a real precision lever without the training-instability cost that showed up during full training. Weights that never receive an update cannot lose small updates to rounding, because there are no updates to lose. That's what makes INT4 (QLoRA — INT4 base + FP32 adapters) work as a routine choice here where it would be exotic in full training. During full training the four small items above summed to 18 GB — a sizing constraint. Here they're a footnote. The drivers behind each row: Frozen base weights. The base sits in memory because the forward pass runs through it, even though it never trains. The base can be held at BF16, INT8, or INT4 without measurable loss to fine-tune quality, since they are not the weights we are training, only the adapter weights are. Activations. Same rules as during training — the forward pass is unchanged by which weights are frozen. The fine-tuning-specific lever is adapter placement: attach adapters to only the top N layers, and the framework skips saving activations below the shallowest adapter. Everything else. Adapter weights, gradients, optimizer states, and the autocast cache all scale with the trainable count — 113 MB combined at these settings. The matrix-multiply workspace adds ~1.5 GB when checkpointing is off and vanishes when it's on. Running the numbers Frozen base weights frozen weight memory = base parameter count × bytes per parameter The benefit of freezing the base is that the base weights, which are not being modified, can be represented in lower precision without harming the fine-tune. The adapter weights, which we are modifying, stay at FP32 to preserve the accuracy of their small updates. The base's precision therefore becomes a free variable, ranging from 4 bytes (FP32) down through 2 bytes (BF16), 1 byte (INT8, 8-bit integer), and 0.5 bytes (INT4, 4-bit integer — two parameters packed per byte) — 8× cheaper storage on the largest static line item on the page, without the training-instability cost that would follow from doing the same in full training. Worked configuration at BF16: 1.10e9 × 2 bytes = 2.20 GB (calculated; measured 2.20 GB). The same base at INT4 would be 0.55 GB — a 1.65 GB reduction on the largest static line item, from a single config flag. QLoRA — the paper that popularized INT4 base + FP32 adapters — is the recipe that makes this practical. On a 70B-parameter base, the difference between BF16 weight storage (140 GB) and INT4 (35 GB) is the difference between "needs a multi-GPU setup" and "fits on one 80 GB card." The article's companion spreadsheet shows the crossover explicitly. Activations Activations follow the training-article rules — same forward pass, same saved tensors, same dependence on batch size, sequence length, and checkpointing. Two things adjust the footprint in this article: the base architecture (specifics vary by model family — see the appendix note on LLaMA's GQA and SwiGLU if you want the mechanism) and adapter placement, the one lever that is unique to fine-tuning. Worked configuration, non-checkpointed: 8.96 GB (calculated; measured 8.96 GB). The formula in predictions_finetune.py breaks this into six named terms per layer plus four once-per-model terms; the article's numbers match those to within 0.05%. The fine-tuning-specific lever is adapter placement. Full training assumes every layer trains, so every layer saves activations. In fine-tuning you can attach adapters to only the top N layers. Layers below the shallowest adapter have no trainable parameters downstream, and the framework saves no activations for their forward pass. The lever is exact: adapters on the upper N layers save activations across N layers, not 22. At upper-11, activations drop from 8.96 GB to 5.40 GB (calculated); at upper-3, they drop to 2.82 GB (calculated). This is the only single-lever choice that reaches the largest line item on the page without touching batch, sequence, or checkpointing. Activation checkpointing Activation checkpointing works exactly as described in the training article — same mechanism, same 4-bytes-per-element budget. The fine-tuning outcome is different: the optimizer step's memory demand doesn't take over the peak the way it does in full training, so cutting activations cuts the peak directly. Worked configuration, checkpointed: activations drop from 8.96 GB to 2.44 GB (calculated) — a 3.7× reduction on the dominant line item. Total allocated drops from 12.80 GB to 4.58 GB (both measured), a 2.8× reduction on the whole run. The training-time cost is the same as before: roughly a third more wall clock per step, because part of the forward pass runs twice. Everything else All four items below scale with the trainable-parameter count and collapse to rounding error when that count is <1% of the base. Formulas are here for anyone chasing a specific number, but no single item has a lever worth pulling on its own. Adapter weights: Trainable parameter count × 4 bytes. LoRA replaces each target projection W with a small trainable pair whose product is added on top; the count depends on the rank and on how many targets receive adapters per layer. Adapters stay at FP32 so their small updates aren't rounded away. Worked configuration: 6.3M parameters × 4 bytes = 25 MB (calculated; measured 25.23 MB). Rank sets adapter capacity (8 is the LoRA paper's default). Gradients: Trainable parameter count × 4 bytes. The framework allocates gradient storage only for trainable weights; the frozen base has none. Worked configuration: 25 MB (calculated; measured 25.23 MB). Optimizer states: Trainable parameter count × 8 bytes for Adam (momentum + variance). Same arithmetic as during full training, applied to <1% of the parameter count. Worked configuration: 50 MB (calculated; measured 50.46 MB). 8-bit Adam would cut this to ~13 MB — a lever from the training article that stops earning its keep here. Autocast weight cache: 2 bytes per trainable weight when the base is already loaded at BF16, because there is nothing left to cast for the base. Only the FP32 adapters produce a cast-and-cache residual. Worked configuration: 6.3M × 2 bytes = 13 MB (calculated; measured within the peak's noise floor). If the base is loaded at FP32 (unusual for fine-tuning), the full training-era cache reappears — the spreadsheet models this. Matrix-multiply workspace Matrix-multiply workspace is a cache the framework holds for the many matrix multiplications a fine-tune step performs. It's empirical rather than derived from architecture, and we don't go further into its mechanics here because there is no lever specific to it. Worked configuration, non-checkpointed: ~1.5 GB (matches measurements to within a few tenths of a GB across the sweep). Under activation checkpointing the allocator releases these caches between recompute segments and the term drops to essentially zero — it comes with the run when checkpointing is off, and goes away when it's on. Two take-aways First, the entire trainable-side static footprint — adapters plus gradients plus optimizer plus autocast cache — is 113 MB. Line them up next to the 18 GB that dominated full training and you can see why fine-tuning is on a different order of hardware. Second, in full training, activation checkpointing shrinks activations dramatically — but the optimizer step has its own memory demand (about 20 bytes per trainable parameter), and once activations shrink, that demand becomes the new peak. In fine-tuning, that constraint disappears: the optimizer step only touches the adapters, so checkpointing cuts the peak by nearly 3× instead of running into a new ceiling. The frozen base is a spectator to the update. What this means in practice: TinyLlama-1.1B at these settings fine-tunes on a 24 GB consumer card with headroom to spare. With checkpointing on, it fits with room to run a batch four times larger. This is the size gap the opening paragraph named — the same math as full training, now demanding roughly one-fifth the hardware for the same model. Checked against hardware Every formula in this article was validated against a 23-configuration LoRA fine-tune sweep of TinyLlama-1.1B on an H100 80GB — varying batch size, sequence length, LoRA rank, adapter placement, base storage precision, and checkpointing one lever at a time for the core sweeps, plus four deliberately stacked combinations at the extremes. For every configuration that completed, predicted memory landed within ~8.3% of measured; every configuration that ran out of memory was predicted to run out. The two configurations that OOM'd — batch 32 with sequence 2,048, at rank 8 across all layers, with and without 8-bit Adam — needed checkpointing to fit, and checkpointing rescued both. Predicted vs. measured peak allocated memory for the 23-configuration LoRA fine-tune sweep on TinyLlama-1.1B, colored by lever. Points sit on or inside the shaded ±10% band around the y=x diagonal; the one point below the line at ~41 GB predicted / ~38 GB measured is the stacked-lever combined configuration where the formula is 8.3% high. Every configuration in the sweep, formula against reality: 23 points, one 8.3% miss, no fudge multipliers. Colors mark which lever was moved from the baseline. The fit boundaries for TinyLlama-1.1B at the worked settings — measured where the sweep covered them, calculated where it did not:       No levers With checkpointing Batch size (at seq 512) 32 fits (measured 44 GB); formula predicts OOM near batch 48 32 fits (calculated 14 GB); formula predicts much larger sizes fit Sequence length (at batch 32) 2,048 fits (measured 44 GB); formula predicts OOM near seq 3,000 2,048 fits (calculated 11 GB); formula predicts much larger sizes fit Combined (batch 32, seq 2,048) measured OOM on 80 GB measured 38 GB, fits with headroom That last row is the pattern this article was written to make visible. The stacked-lever configuration — every dial pushed hard — does not fit on the 80 GB card without checkpointing. Turn checkpointing on and it fits with 40 GB of headroom. Same model, same card, one config flag. The worst prediction miss was on the two stacked-lever checkpointed configurations, where the formula predicts +8.3% over what was measured. The formula estimates the checkpointed peak by assuming all saved layer boundaries and one segment's transient tensors are alive at the same time; the allocator in practice releases some of the intermediate transients slightly earlier under memory pressure. That's an honest formula-side gap, not a fudge factor's worth of slack. It is reported rather than tuned away, matching the training article's disclosure of its own 11.6% miss from that sweep. What these numbers do not cover Full fine-tuning. These formulas cover LoRA and its variants — recipes where a frozen base hosts a small trainable adapter. Full fine-tuning updates every weight; it uses the training article's math directly, sized to the base model's parameter count. The line-item collapse from this article does not apply. Adapters beyond LoRA. The general shape — frozen base plus small trainable adjustment — carries over to related PEFT families (IA³, DoRA, and prefix-tuning, for instance), but the specific per-target parameter arithmetic is LoRA's. Each other family needs its own trainable-parameter count; the rest of the memory story is unchanged. Ragged batches and data pipelines. Same caveats as during full training: peak memory is driven by the longest sequence in the batch, not the average, and GPU-resident data pipelines can hold significant VRAM outside the model. Budget for both separately. A note on measurement. Figures labeled measured come from a 23-configuration LoRA benchmark on a single H100 80GB. The static line items are measured directly — by summing the actual bytes of every base parameter, adapter parameter, gradient, and optimizer tensor on the device — not inferred. Activations are the difference between two measured allocator readings. Peak-composition claims come from replaying the CUDA allocator's event history (benchmark/debug_finetune.py). A companion spreadsheet — assets/gpu-finetune-memory-sizing.xlsx — turns this math into a sizing tool. Plug in your base model (LLaMA-2-7B, LLaMA-3-8B, and larger presets are included), your LoRA rank and target set, your batch and sequence, and it tells you whether the job fits on the GPU you have. And whether QLoRA on a smaller card would fit it more comfortably. Appendix: LLaMA architecture notes Two LLaMA-family details show up in the activation math above. Neither is a lever — the reader doesn't tune them to size the job — but they explain why TinyLlama's per-layer activation footprint differs from a plain-vanilla transformer's. Skip this section unless you want the mechanism. Grouped-query attention (GQA). LLaMA-family models split attention heads into query heads and key-value heads at different totals — TinyLlama-1.1B uses 32 query heads and 4 KV heads, a ratio of 8:1. The saved K and V tensors are therefore num_kv_heads / num_heads as wide as the query tensor, which shrinks the attention-side activation footprint. It's a real save — around 1 byte per element per layer in this configuration — but small compared with the feed-forward hidden layer covered next. Gated MLP (SwiGLU). Instead of the classic two-linear MLP, LLaMA's variant uses three linear projections, saving roughly twice as many intermediate tensors per layer — about four instead of two — which is why the per-layer activation footprint is larger than in a plain transformer. Next: Inference — where a different memory consumer, the KV cache, takes over. (coming soon) ... View more

This is Article 1 in a series on sizing GPU infrastructure for foundation models. For most predictive models, GPU memory is an afterthought. You train the model, it fits, you move on. Transformer models change that. Memory becomes a primary planning concern, and the decisions that drive it are made before training starts — not discovered hours into a run. What this article offers is a straightforward way to think about how pre-training choices map to hardware requirements. Model size, sequence length, batch size, and weight representation each push memory in a specific direction. Once you can see which choice drives which cost, you can also see your levers to bring memory down, and what each of those levers costs you. This article covers training on a single GPU. Multi-GPU strategies are a separate article. Memory during a full training run comes down to four line items — plus one small fifth that exists only under mixed precision, flagged where it appears: Weights. The most straightforward. The number of parameters is the number of weights; storage precision sets how many bytes each one takes — 4 in practice, not the 2 that a "BF16 training" label suggests, for reasons covered below. Gradients. During the backward pass, each weight gets a gradient — the adjustment to apply to that weight. One gradient per weight, same formula as above. Optimizer states. Adaptive optimizers keep running statistics of past gradients to scale each weight's update. For Adam that history is two full-precision values per weight — momentum and variance — or 8 bytes per weight. (No separate "master copy" of the weights is needed when the weights themselves are already stored in full precision.) Activations. The intermediate tensors the forward pass must keep so the backward pass can compute gradients. Memory here is a function of three choices: sequence length, how long your training examples are in tokens; batch size, how many examples you process at once; and activation checkpointing, which lets you keep a partial record of activations rather than all of them at once. That last one has outsized benefits, with a cost in training time. The first three line items are static. They do not change whether your batch size is 1 or 4,096. The fourth is dynamic, and it is usually the largest number on the page — often by a wide margin. The line items, the levers, and what the levers cost All figures below use one worked configuration, carried through the whole article: 1.01 billion parameters, 20 layers, hidden dimension 2,048, 16 attention heads, feed-forward width 4× the hidden dimension, output vocabulary 1,000, sequence length 100 tokens, batch size 256 examples, BF16 mixed precision (BF16 compute, FP32 storage), standard Adam optimizer. Figures labeled measured come from a 24-configuration benchmark of this model on an H100 80GB — see the note on measurement at the end. Unlabeled figures in tables are calculated. Line item Memory (calculated) Levers to reduce it What the lever costs you Weights 4.0 GB Fewer parameters; true low-precision storage (rare in training — see below) A smaller model is a smaller model; low-precision storage risks training instability Gradients 4.0 GB Gradient accumulation across micro-batches Trades wall-clock time for memory; the total memory saved comes from the smaller micro-batch, not from the gradients themselves Optimizer states 8.1 GB 8-bit Adam; a stateless optimizer such as SGD 8-bit Adam carries a small accuracy risk; SGD often converges slower or to a worse result BF16 weight cache 2.0 GB None worth pulling — it buys BF16 compute speed; disappears in pure FP32 Pure FP32 nearly doubles activation memory (calculated 1.85×) and forfeits the speedup Activations ~43 GB Activation checkpointing; smaller batch size; shorter sequence length Activation checkpointing adds roughly a third to training time; smaller batches can slow or destabilize convergence; shorter sequences limit what the model can learn Two observations. First, optimizer states are twice the weights — the largest static cost, and the one most often left out of a budget; weights, gradients, and optimizer state come to ~16 GB before activations enter the picture (the 2.0 GB BF16 cache exists mid-forward but is released before the peak). Second, activations dominate everything else combined, and they are also where your cheapest lever lives. Running the numbers Weights weight memory = parameter count × bytes per parameter Bytes per parameter is set by storage precision: 4 bytes for FP32, 2 bytes for BF16 or FP16, 1 byte for INT8. And in a standard mixed-precision recipe, storage precision is FP32: the framework casts weights to BF16 on the fly for each matrix multiply, but the stored weights — the ones this line item counts — never change. True BF16 storage exists but is rare in training, because applying small optimizer updates to low-precision weights loses them. Worked configuration: 1.01e9 × 4 bytes = 4.0 GB (calculated; measured 4.05 GB). Gradients gradient memory = parameter count × bytes per parameter Same shape as weights, because there is exactly one gradient per weight, and gradients match the storage precision of their weights: FP32. Worked configuration: 1.01e9 × 4 bytes = 4.0 GB (calculated; measured 4.05 GB). Optimizer states optimizer memory = parameter count × bytes per parameter of optimizer state For Adam, two values are held per parameter, both in FP32: Momentum (first moment): 4 bytes Variance (second moment): 4 bytes That is 8 bytes per parameter. Worked configuration: 1.01e9 × 8 bytes = 8.1 GB (calculated; measured 8.09 GB). You may have seen 12 bytes per parameter quoted for Adam. That figure includes a full-precision "master copy" of the weights, which only exists in recipes that store weights in BF16 — the optimizer keeps an FP32 copy so small updates aren't lost, applies updates there, and casts back down. When weights are stored in FP32, as here, the weights are their own master copy and the third term disappears. Worth noting: mixed precision does not reduce your static memory either way. BF16 weights plus BF16 gradients plus 12 bytes of Adam state is 16 bytes per parameter. FP32 weights plus FP32 gradients plus 8 bytes of plain Adam state is also 16 bytes per parameter. Mixed precision buys you speed and lower activation memory, not a smaller static footprint. If this line is your constraint, 8-bit Adam stores momentum and variance in 1 byte each instead of 4, taking the total from 8 bytes to roughly 2 bytes per parameter — about 2 GB for this model. Measured: switching the benchmark to 8-bit Adam cut total training memory by 6.04 GB, within 0.5% of what this arithmetic predicts. The BF16 weight cache One small line item exists only under mixed precision: when autocast casts an FP32 weight to BF16 for a matrix multiply, it caches that BF16 copy for reuse across the rest of the forward pass. The cache is parameter count × 2 bytes — 2.0 GB here (measured in the checkpointed run's forward phase: disabling the cache lowers that peak by 1.98 GB). It disappears in pure FP32 training, and there is no lever worth pulling on it: it is the cost of the BF16 compute speedup, and it releases before the run's peak — the same toggle makes no measurable difference to the non-checkpointed peak, which lands in the backward pass after the cache is freed. Activations Activations are the only line item that scales with your data, and they are the reason a model that looks like it needs 16 GB will not fit on a 40 GB card. activation memory ≈ layers × batch size × sequence length × hidden dimension × bytes per element per layer Where: layers is the number of transformer blocks (20 here) batch size is examples processed simultaneously (256 here) sequence length is tokens per example (100 here) hidden dimension is the model's internal width (2,048 here) bytes per element per layer is the term that surprises people: roughly 40 bytes without activation checkpointing (the per-tensor accounting gives exactly 40; the measured value implies ~41 — once-per-model terms plus a residual under 2%). Worked configuration: 20 × 256 × 100 × 2,048 × 40 bytes ≈ 42 GB (calculated: 42.2 GB; measured: 42.9 GB on an H100 80GB; see the note on measurement below). That last constant is where most back-of-the-envelope estimates go wrong. The intuitive guess is 2 bytes per element per layer — one BF16 tensor of shape (batch × sequence × hidden dimension) saved per layer. The real number is roughly twenty times that, because a transformer block does not save one intermediate tensor. It saves fourteen of them: seven attention-side tensors (the pre-norm input, the projection input, query, key, value, the attention output, and the output-projection input), two feed-forward inputs, the feed-forward hidden layer twice — at four times the hidden dimension each — and three dropout masks. All of them are needed again in the backward pass. If you take one number from this article, take that one. Estimating activations at 2 bytes per element per layer will tell you a job fits when it needs twenty times the memory you budgeted. What "batch" actually means here Batch size and sequence length both appear as straight multipliers, which means what really drives activation memory is their product: tokens per batch, not rows per batch. For anyone coming from tabular machine learning, this is the trap. A batch of 256 rows sounds small. If each row tokenizes into 100 features, that is 25,600 tokens per batch flowing through every layer. Doubling your feature count has exactly the same memory effect as doubling your batch size. Batch size and sequence length Both scale activation memory linearly. Halving either one halves this line item. Going from a batch size of 256 examples to 128 examples takes activations from roughly 43 GB to roughly 21.5 GB (calculated). The other three line items — weights, gradients, optimizer states — do not move at all. They stay at 16.2 GB combined (calculated; measured 16.3 GB). Sequence length behaves the same way in the formula above, with one caveat. If your implementation stores the full attention probability matrix, memory grows with the square of sequence length rather than linearly, and long sequences get expensive fast. Memory-efficient attention implementations avoid storing that matrix. If long sequences are a requirement rather than a preference, confirming you have memory-efficient attention is the first thing to check. Activation checkpointing Activation checkpointing is the highest-leverage decision on this page. The default behavior is to store every intermediate tensor from the forward pass, because the backward pass needs them to compute gradients. Activation checkpointing stores only the input to each layer and discards the interior. When the backward pass reaches a layer, the framework recomputes that layer's interior from the stored input, uses it, and discards it again. So instead of holding every layer's full working set at once, you hold: (layers × batch size × sequence length × hidden dimension × 4 bytes) ← the saved layer inputs one layer's full working set, plus the gradient flowing through it ← the recomputation peak One trap for estimators: the saved layer inputs are 4 bytes per element even under BF16 mixed precision. Layer norms and residual additions run in full precision, and the tensor passed between layers — which is exactly what checkpointing saves — comes out of those FP32 ops. Budget 4 bytes, not 2, or your dominant term is off by half. Worked configuration: 4.2 GB of saved layer inputs plus roughly 2.7 GB for the layer being recomputed and its gradients ≈ 6.9 GB (calculated). That is roughly 43 GB down to 7 GB — a sixfold reduction on the largest line item in the budget. And at this configuration it shrinks activations so far that they stop setting the peak at all: the run's measured high-water mark, 20.3 GB, comes from the optimizer step — weights, gradients, optimizer state, plus Adam's own update scratch, about 20 bytes per parameter — not from the activations. Once checkpointing is on, that 20-bytes-per-parameter floor is the number a bigger batch has to beat before it costs you anything. The cost is training time. You are running part of the forward pass twice. In the benchmark run behind this article, per-step time went from 413 ms to 544 ms, a 31% increase (measured). That is a real cost, but it is predictable, and it is almost always a better trade than not being able to train at all. Putting it together Same worked configuration: 1.01 billion parameters, 20 layers, hidden dimension 2,048, sequence length 100 tokens, batch size 256 examples, BF16 mixed precision, standard Adam. Line item Without activation checkpointing With activation checkpointing Weights 4.0 GB 4.0 GB Gradients 4.0 GB 4.0 GB Optimizer states 8.1 GB 8.1 GB BF16 weight cache 2.0 GB (released before the peak) 2.0 GB (released before the peak) Activations ~43 GB ~7 GB (at their own peak) Peak allocated (measured) 59.2 GB 20.3 GB Allocator overhead: cached blocks and fragmentation (measured) ~1.0 GB ~6.1 GB Total to size against (measured reserved) 60.2 GB 26.5 GB Add roughly another 0.6 GB on top of the reserved figure for the CUDA context itself — driver and kernel-library state that lives outside the framework's memory counters entirely. The columns do not sum exactly to the peak, and that is not rounding: memory peaks at a moment, not as a ledger total. Without checkpointing, the peak lands early in the backward pass, while all activations are still alive but after the BF16 cache has been released, so the sum less the cache is close. With checkpointing, the peak lands on the optimizer step — when almost no activations are alive either — so the peak is the 20-bytes-per-parameter optimizer floor, not the column sum. The practical read: without activation checkpointing, this job needs an 80 GB card and has no room to grow. With it, the same job fits on a 40 GB card with headroom to raise the batch size. One configuration flag moved this training run across a hardware tier. Checked against hardware Every formula in this article was validated against a 24-configuration training sweep of the worked model on an H100 80GB — varying batch size, sequence length, optimizer, precision, and checkpointing one lever at a time for the core sweeps, plus deliberate combinations (checkpointing crossed with batch and sequence, and an all-levers-at-once set). For every configuration that completed, predicted memory landed within ~2% of measured; every configuration that ran out of memory was predicted to run out of memory, and none that fit were predicted to OOM. The measured fit boundaries for the worked 1B-parameter model on the 80 GB card (figures are peak allocated; reserved runs 1–6 GB higher, as above):   No levers With checkpointing Batch size (at seq 100) 256 fits (59.2 GB); 1,024 OOMs 1,024 fits (40.0 GB); 4,096 OOMs Sequence length (at batch 256) 100 fits (59.2 GB); 512 OOMs 512 fits (47.8 GB); 1,024 OOMs That table is the whole argument of this article in four cells: the same model, on the same card, either trains or cannot train depending on choices that cost nothing but a config flag — and each lever buys roughly one 4× step in batch or sequence before the wall moves back in.   Sequence length is the fastest way to hit the wall (worked model, batch 256, BF16 mixed precision): predicted total demand grows linearly through 236, 452, and 884 GB at seq 512, 1,024, and 2,048 — the last of them eleven cards' worth — for what looks like a modest config change. The hatched bars are configurations that ran out of memory; the black diamond is the one that fit. That is the pattern worth internalizing. The static line items are set by decisions you have probably already made — model size, optimizer, precision. The dynamic line item is set by decisions you can still change, and it is usually the one that determines what hardware you need. What these numbers do not cover A few caveats, and where the numbers come from. Allocated versus reserved. The formulas above predict allocated memory — bytes held by live tensors. What the driver actually takes from the card is reserved memory, which includes cached free blocks and fragmentation. Size your GPU against reserved. The measured gap was about 1 GB in the run without activation checkpointing and about 6 GB with it, since checkpointing frees and reallocates constantly and fragments the pool. Ragged batches. These numbers assume fixed-shape examples. Real batches have variable lengths, and peak memory is driven by the longest sequence in the batch, not the average. If you size using your average sequence length, you will run hotter than these predictions. Data pipeline memory. The four line items account for the model. They do not account for your data sitting in GPU memory. GPU-resident pipelines — RAPIDS and cuDF in particular — can hold significant VRAM outside the model, and for smaller models that pipeline can be the dominant consumer. Budget for it separately. A note on measurement. Figures labeled measured come from a 24-configuration training benchmark on a single H100 80GB. The static line items are measured directly — by summing the actual bytes of every parameter, gradient, and optimizer tensor on the device — not inferred. Activations are the difference between two measured allocator readings (peak allocated minus allocated after optimizer setup). Peak composition claims (which tensors are alive at the peak moment) come from replaying the CUDA allocator's own event history. A companion spreadsheet will turn this math into a sizing tool — plug in your model size, batch size, and precision, and it tells you whether the job fits on the GPU you have. ... View more

Frontier models have changed what's possible. Work that previously took days now takes hours. But hosted model services will not address every business need — cost, data governance, throughput ceilings, and control over model behavior all limit the ability of these services to meet all enterprise AI needs. Open-weight models — now trailing frontier capability by months rather than years — are a game changer. Fine-tuning adds real customization at a favorable total cost of ownership. And, of course, there will still be business cases that demand full model training. The optimal play is no longer a single model; it's an assortment: the right model for the right job. Model selection — and how that model is trained, customized, and deployed — has direct consequences for your hardware requirements. Considering that demand for GPU capacity routinely outpaces supply, design choices are critical. Selections such as model size, sequence length, batch size, and many more determine how much GPU memory you need and how many GPUs it takes to get there. Your starting point may be the opposite: given the hardware and software available, which models can you actually run, and what choices get you there while still meeting your usage requirements? Regardless of how you are thinking about model development and deployment choices, the work falls into the same three areas: training a model from scratch, fine-tuning an existing model, and serving a model in production. We will be issuing an article for each area, along with a working sizing tool. But before sizing anything, let's establish a shared framing: the model types available and the work each one implies. The Model Selection Framework Most IT and AI teams are aware of the four model types: frontier model APIs, self-hosted open-weight models, fine-tuned open-weight models, and fully custom-trained models. What most teams lack is a clear picture of what each option demands in hardware. So they default to whichever one they already have experience with and build the justification afterward. Figure 1 lays out the four types and maps each to the activities you own — the activities this series sizes. Figure 1 shows the four model types, running left to right from most turnkey to most self-managed. Each model type maps to a set of activities. The exception is the frontier model API, which maps to none — it sits outside this series. The other three all require inference: any model you host, you serve. Fine-tuning applies to the fine-tuned open-weight model. Full training applies to the fully custom-trained model. These activities — training, fine-tuning, and inference — are the work this series sizes. Each gets its own article. Find your model type in Figure 1, follow its arrows, and those are the articles that apply to you. Where this series goes next The series runs the bottom row of Figure 1 in reverse. We start with training because it establishes the fundamentals — the memory line items — that fine-tuning and inference reuse. Build the full picture once, use it three times. Training — the four line items of GPU memory: weights, gradients, optimizer states, activations. The math for a 1-billion-parameter model, and a spreadsheet to size your own runs before your first out-of-memory error. Fine-tuning — where the math gets friendlier. (coming soon) Inference — where a different memory consumer, the KV cache, takes over. (coming soon) Each article will ship with a working sizing tool.  https://github.com/odog96/gpu-sizing-playbook.git   ... View more

While generative AI dominates today's headlines, traditional predictive machine learning models continue to drive critical business decisions across industries. To ensure predictive models achieve a solid ROI, well after models are initially deployed,  establishing a Machine Learning Operations (MLOps) plan is essential. MLOps is the practice of streamlining the entire lifecycle of machine learning models—from development and training to deployment, monitoring, and maintenance—in a repeatable, scalable, and governable way. Think of it as bringing software engineering discipline to machine learning, ensuring that your AI investments don't remain theoretical exercises but become dependable business assets that continue to deliver value over time. Without robust MLOps practices, models often degrade in production as data shifts over time. What begins as an impressive prototype can quickly become unreliable, leading to poor decision quality with real financial consequences. Poor model accuracy directly impacts business outcomes, diminishing your ML investment's ROI and potentially creating compliance risks.  Implementing MLOps can seem daunting, but with the right platform and processes, organizations can establish systems that maximize their ROI. The first step is to understand the critical steps and phases of the machine learning life cycle. Then identifying the framework and tools required to handle these phases. Cloudera AI offers an integrated environment designed to address each critical stage of the machine learning lifecycle. The Machine Learning Lifecycle with Cloudera AI: Machine Learning Operations with Cloudera Business Inputs & Data Engineering Leverage Cloudera's data connections to seamlessly access data from diverse sources Integrate business requirements directly into the ML pipeline through Cloudera's Feature Store Data Science Work in customizable Sessions with pre-configured runtimes for Python, R, and Spark and use integrated JupyterLab and Workbench environments for collaborative development Apply secure data access controls through Cloudera SDX Model Security framework Model Training Track experiments through native MLflow integration within Cloudera's Model Catalog Scale training with distributed computing resources via Kubernetes Machine Learning Operations Packaging: Containerize models with dependencies automatically managed through Cloudera SDX Deployment & Serving: Deploy models as REST APIs with a few clicks through Cloudera's Model Governance system Monitoring: Track model performance and detect drift through dedicated monitoring dashboards Closed Loop ML Implement automated retraining pipelines when monitoring triggers performance thresholds Ensure continuous model improvement with feedback loops from production to training Enterprise Governance Implement comprehensive model governance through Cloudera SDX (Shared Data Experience) providing unified security and governance Leverage the Cloudera Data Catalog to track model assets, metadata, and maintain governance across the ML lifecycle This end-to-end MLOps framework ensures organizations can efficiently operationalize machine learning while maintaining security, governance, and scalability throughout the entire lifecycle. Hands-On MLOps: The Banking Marketing Campaign Example To see these capabilities in action, let's explore the banking marketing campaign example available in the cml-banking-mlop-marketing-campaign repository. This project implements a complete MLOps workflow for a common banking use case: predicting which customers are likely to subscribe to a term deposit during a marketing campaign. The repository provides a step-by-step guide through the entire process: Data acquisition and storage using Cloudera's data connections to ingest the UCI Bank Marketing dataset and store it in a data lake with Apache Iceberg format, ensuring version control and proper governance. Exploratory data analysis with JupyterLab to understand customer characteristics and their correlation with campaign outcomes, demonstrating Cloudera AI's interactive analysis capabilities. Model training with MLflow to systematically experiment with different XGBoost configurations, tracking all parameters, metrics, and artifacts. This showcases how Cloudera AI's integrated experiment tracking simplifies model development. Model deployment as a REST API using Cloudera AI's Models functionality, making predictions available to other applications through a standardized interface with proper authentication and monitoring. Automated retraining and updating through a sequence of Jobs that simulate new data arrival, retrain models, and update deployments—demonstrating Cloudera AI's automation capabilities. Performance monitoring with a dashboard that tracks model accuracy over time, alerting when performance degrades and triggering the retraining workflow. This example showcases Cloudera AI's ability to orchestrate the entire MLOps lifecycle without requiring complex integration of disparate tools. Each component—from data connections to experiment tracking to model deployment—works together seamlessly, allowing data scientists and ML engineers to focus on creating value rather than managing infrastructure. The Banking Marketing MLOps lab demonstrates a practical example of managing a machine learning model throughout its lifecycle. The use case focuses on a common challenge in banking: predicting which customers are likely to subscribe to a term deposit during a marketing campaign. The lab begins with real customer data from the UCI Bank Marketing dataset, which contains information about customer demographics, previous interactions, and whether they subscribed to term deposits. This historical data serves as the foundation for training our initial classification model using XGBoost and tracking experiments with MLflow. This lab simulates the passage of time – a critical element often overlooked in ML examples. After deploying the initial model as a REST API endpoint, the lab uses Cloudera's data generation capabilities to create synthetic customer data that represents new interactions over time. This mimics the real-world scenario where models must process fresh data that may differ from their training distribution. ... View more

While generative AI dominates today's headlines, traditional predictive machine learning models continue to drive critical business decisions across industries. To ensure predictive models achieve a solid ROI, well after models are initially deployed,  establishing a Machine Learning Operations (MLOps) plan is essential. MLOps is the practice of streamlining the entire lifecycle of machine learning models—from development and training to deployment, monitoring, and maintenance—in a repeatable, scalable, and governable way. Think of it as bringing software engineering discipline to machine learning, ensuring that your AI investments don't remain theoretical exercises but become dependable business assets that continue to deliver value over time. Without robust MLOps practices, models often degrade in production as data shifts over time. What begins as an impressive prototype can quickly become unreliable, leading to poor decision quality with real financial consequences. Poor model accuracy directly impacts business outcomes, diminishing your ML investment's ROI and potentially creating compliance risks.  Implementing MLOps can seem daunting, but with the right platform and processes, organizations can establish systems that maximize their ROI. The first step is to understand the critical steps and phases of the machine learning life cycle. Then identifying the framework and tools required to handle these phases. Cloudera AI offers an integrated environment designed to address each critical stage of the machine learning lifecycle. The Machine Learning Lifecycle with Cloudera AI: Machine Learning Operations with Cloudera Business Inputs & Data Engineering Leverage Cloudera's data connections to seamlessly access data from diverse sources Integrate business requirements directly into the ML pipeline through Cloudera's Feature Store Data Science Work in customizable Sessions with pre-configured runtimes for Python, R, and Spark and use integrated JupyterLab and Workbench environments for collaborative development Apply secure data access controls through Cloudera SDX Model Security framework Model Training Track experiments through native MLflow integration within Cloudera's Model Catalog Scale training with distributed computing resources via Kubernetes Machine Learning Operations Packaging: Containerize models with dependencies automatically managed through Cloudera SDX Deployment & Serving: Deploy models as REST APIs with a few clicks through Cloudera's Model Governance system Monitoring: Track model performance and detect drift through dedicated monitoring dashboards Closed Loop ML Implement automated retraining pipelines when monitoring triggers performance thresholds Ensure continuous model improvement with feedback loops from production to training Enterprise Governance Implement comprehensive model governance through Cloudera SDX (Shared Data Experience) providing unified security and governance Leverage the Cloudera Data Catalog to track model assets, metadata, and maintain governance across the ML lifecycle This end-to-end MLOps framework ensures organizations can efficiently operationalize machine learning while maintaining security, governance, and scalability throughout the entire lifecycle. Hands-On MLOps: The Banking Marketing Campaign Example To see these capabilities in action, let's explore the banking marketing campaign example available in the cml-banking-mlop-marketing-campaign repository. This project implements a complete MLOps workflow for a common banking use case: predicting which customers are likely to subscribe to a term deposit during a marketing campaign. The repository provides a step-by-step guide through the entire process: Data acquisition and storage using Cloudera's data connections to ingest the UCI Bank Marketing dataset and store it in a data lake with Apache Iceberg format, ensuring version control and proper governance. Exploratory data analysis with JupyterLab to understand customer characteristics and their correlation with campaign outcomes, demonstrating Cloudera AI's interactive analysis capabilities. Model training with MLflow to systematically experiment with different XGBoost configurations, tracking all parameters, metrics, and artifacts. This showcases how Cloudera AI's integrated experiment tracking simplifies model development. Model deployment as a REST API using Cloudera AI's Models functionality, making predictions available to other applications through a standardized interface with proper authentication and monitoring. Automated retraining and updating through a sequence of Jobs that simulate new data arrival, retrain models, and update deployments—demonstrating Cloudera AI's automation capabilities. Performance monitoring with a dashboard that tracks model accuracy over time, alerting when performance degrades and triggering the retraining workflow. This example showcases Cloudera AI's ability to orchestrate the entire MLOps lifecycle without requiring complex integration of disparate tools. Each component—from data connections to experiment tracking to model deployment—works together seamlessly, allowing data scientists and ML engineers to focus on creating value rather than managing infrastructure. The Banking Marketing MLOps lab demonstrates a practical example of managing a machine learning model throughout its lifecycle. The use case focuses on a common challenge in banking: predicting which customers are likely to subscribe to a term deposit during a marketing campaign. The lab begins with real customer data from the UCI Bank Marketing dataset, which contains information about customer demographics, previous interactions, and whether they subscribed to term deposits. This historical data serves as the foundation for training our initial classification model using XGBoost and tracking experiments with MLflow. This lab simulates the passage of time – a critical element often overlooked in ML examples. After deploying the initial model as a REST API endpoint, the lab uses Cloudera's data generation capabilities to create synthetic customer data that represents new interactions over time. This mimics the real-world scenario where models must process fresh data that may differ from their training distribution. ... View more

According to a survey conducted by Kaggle in 2021, Python is still the most commonly used programming language for data science with over 80% of respondents choosing it as their preferred language. However, R continues to be a popular language among data scientists, with over 15% of respondents choosing it as their primary language.   One of the reasons for R's continued popularity is its strong statistical analysis capabilities. R was designed specifically for statistical computing and provides a rich ecosystem of packages for data analysis and visualization. This makes R a powerful tool for data scientists who need to analyze large datasets and perform complex statistical modeling.   In this article, we'll delve into how to deploy R models in CML, highlighting the steps and key considerations to keep in mind when building and deploying models in this environment.   CMLs Model Framework   As a refresher, let's revisit the key concepts of a model in CML. CML's framework allows for maximum flexibility when it comes to deploying models. Here is the reference diagram showing the fundamental concepts of a model.  Models - Concepts and Terminology   The model artifacts are actually called from within a Python or R script file. Regardless of the runtime used, you will need to embed your prediction logic within a function. The input arguments sent to the CML model are in JSON format. By the time these parameters are ingested by the function within the R script file, it becomes an R list object. This is important to note because this will determine what, if any transformations, need to occur before getting to the prediction step in your code.    Simple Add Model in R Let’s start by looking at a deployed model below for a CML model that adds two numbers. In this case, we take the two elements from the function arguments and add them. R wrapper script The CML model parameters, or in this case the named list elements are defined when the CML model is deployed. Deploying the add 'model' Working with actual prediction models The example above helps us get started with using an R model in CML. Now let’s look at two model examples with a focus on the R script file and how parameters are ultimately passed to the model object. For the two models, we deploy below. We’ll be using the Cars93 dataset.    Simple Linear Regression In the example below we are using the Cylinders and Weight as features (or independent variables) to predict our dependent variable - MPG.City. You can follow the details here to see how the model was built R-CML in github   In this example, you will note that no further transformation is required. The input parameters were passed directly into the prediction step.   Decision Tree Model  In our final model, we’ve gotten slightly more sophisticated, included more features and now using a decision tree model. We trained our model so that it takes R data frame objects as inputs for predictions. Therefore we need the appropriate step to transform our list into a data frame.     Below we can see how we define the json input format for the model.    I hope this has given you enough information to go and build your own R models in CML! Happy model building! ... View more