How much VRAM does a 70B model need? Worked examples from the UBM AI simulator

A 70B model such as Llama 3.3 70B needs 141 GB of GPU memory (VRAM) for its weights at 16-bit, 74 GB at 8-bit and 41 GB at 4-bit, plus a cache for every conversation. At 4-bit with ten 8K-token conversations, it fits one 80 GB GPU. OpenAI's gpt-oss-120b, despite 117B parameters, also fits one 80 GB GPU.

The parameter count alone does not size a server. Two models labelled "70B" can need very different hardware: Jais 2 70B, the Arabic-first model from MBZUAI, Inception and Cerebras, keeps six times as much cache per conversation as Llama 3.3 70B, so at 4-bit the same ten people need a single 141 GB GPU instead of a single 80 GB one.

Every figure below comes from UBM's LLM hardware requirements calculator, the AI infrastructure simulator, run on 1 October 2026, and each worked example links to the same workload in the simulator, where you can change the inputs and see the complete servers it recommends. Memory figures are arithmetic on each model's published architecture plus the simulator's stated allowances (quantised weight sizes, runtime and the 92 percent usable share); speeds are estimates, good to about plus or minus 25 percent. For bandwidth, power, cooling and the serving stack, see our guide to running AI on your own servers in the UAE.

How much VRAM for a 70B model? The formula

Add three things, then compare the total with the part of each card a serving engine can use:

• Weights. Parameters multiplied by bytes per parameter. Meta's published checkpoint of Llama 3.3 70B holds 70.55 billion parameters at 2 bytes each (BF16), or 141.1 GB. The Jais 2 70B checkpoint holds 72.04 billion, or 144.1 GB.
• KV cache. For every token of every conversation, each layer stores a key and a value so it does not recompute them: 2 × layers × key-value heads × values per head × bytes per value.
• Runtime. The simulator reserves 2.5 GB per GPU for the GPU software context and communication buffers, plus working memory for reading prompts in 8,192-token chunks: 4.6 GB in all for Llama 3.3 70B and 4.4 GB for Jais 2 70B on a single GPU, less per GPU when the model is split.
• Usable share. It counts 92 percent of each card as usable. An 80 GB card gives 73.6 GB; a 141 GB card gives 129.7 GB.

GB here means one billion bytes.

Weights at 16-bit, 8-bit and 4-bit

Fewer bytes per parameter means smaller weights. 8-bit formats (FP8 or INT8) store about one byte per parameter, and 4-bit formats such as AWQ and GPTQ about half a byte. The simulator adds the scaling factors these formats store (about 1.015 and 0.53 bytes per parameter in all) and keeps the input and output embedding layers at 16-bit, which is why the 8-bit figures are a little more than half of the 16-bit ones.

Smaller weights can cost some answer quality, and how much depends on the model, the method and the task. Test a quantised model on your own documents, in Arabic and English, before you size hardware around it.

The KV cache: why two 70B models need different GPUs

Because one keeps six times as much memory for every token. Meta's Llama 3 paper gives the 70B architecture as 80 layers, a model dimension of 8,192 and 64 attention heads that share 8 key-value heads, a design called grouped-query attention (GQA). The Llama 3.3 model card confirms that the 70B model uses GQA, and we assume the same layout. The Jais 2 technical report gives Jais 2 70B 68 layers and 56 attention heads with multi-head attention, so every head keeps its own key and value. Llama's 8,192 dimensions over 64 heads make 128 values per head, and the Jais 2 report gives the same head dimension, 128.

At 2 bytes per value, Llama 3.3 70B stores 2 × 80 × 8 × 128 × 2 = 327,680 bytes (320 KiB) per token, and Jais 2 70B stores 2 × 68 × 56 × 128 × 2 = 1,949,696 bytes (1,904 KiB), 5.95 times as much.

Half of gpt-oss-120b's layers look back only 128 tokens, according to its configuration, so their cache stops growing there and the model adds 36 KiB per token beyond that.

An 8-bit cache halves every conversation. vLLM offers it as its quantized KV cache, which its documentation says can significantly reduce the cache's memory footprint; the simulator defaults to it, and every example below uses it. The simulator also reserves each conversation at its full length, the safe assumption for sizing. In practice vLLM allocates the cache in small pages as it fills (the method in the PagedAttention paper), so shorter conversations leave room for more. Jais 2 70B's model card gives its context length as 8,192 tokens, so 8K is the longest conversation it holds.

Worked examples: 10 and 50 users

Here are both models at three precisions, with 10 and then 50 people holding 8K-token conversations at the same moment and the cache at 8-bit. The cache is the same whatever the weight precision: 13.4 GB for 10 users and 67.1 GB for 50 on Llama 3.3 70B, against 79.9 GB and 399.3 GB on Jais 2 70B. The totals below are weights plus cache; add about 4.5 GB of runtime on a single GPU (less per GPU when the model is split), and remember that only 92 percent of each card is usable.

For Llama 3.3 70B, precision decides the card. Follow it down the precisions on a single GPU. At 16-bit, 141.1 GB of weights and 4.6 GB of runtime (145.8 GB) need a 192 GB card such as AMD's Instinct MI300X. At 8-bit, 78.3 GB is more than the 73.6 GB an 80 GB card can use, so it moves up to a 96 GB card such as NVIDIA's RTX PRO 6000 Blackwell Server Edition. At 4-bit, 45.2 GB is 1 GB more than a 48 GB L40S can use, so it lands on one 80 GB H100. Ten users add 13.4 GB (58.6 GB in all) and still fit there; fifty add 67.1 GB (112.3 GB) and push it to one 141 GB H200.

For Jais 2 70B, the number of people decides. At 4-bit, the cache for 50 conversations is nearly ten times the size of the weights.

Which GPU for Llama 3.3 70B and Jais 2 70B?

Count how many identical GPUs hold the weights, the runtime and every conversation. Inside one server, the simulator splits the model evenly across 1, 2, 4 or 8 GPUs (tensor parallelism), which divides both the weights and each conversation's cache between them; beyond eight, it splits the layers between servers.

The smallest count of 1, 2, 4, 8 or 16 identical GPUs whose usable memory holds the weights, runtime and every 8K-token conversation with an 8-bit cache. This is memory only; the simulator checks speed separately, and its ranking also weighs price, power and rack space. 16 means two eight-GPU servers with the model split between them.

The pattern is plain. For Jais 2 70B with 50 users, 4-bit weights save no GPUs at all compared with 8-bit, because the cache, not the weights, fills the cards. If Jais 2 70B is the model you need for Arabic, budget for the cache first.

Five scenarios in detail. The card count is the smallest number of one card type that holds the workload, and the figures are the simulator's calculation for exactly that many cards. Each name links to the same workload in the simulator, which ranks complete servers by cost, each with its own fixed number of cards, so the server it lists first can be a larger or a different one:

Simulator output, 1 October 2026. Speeds are estimates for every conversation generating at once.

Can gpt-oss-120b run on one GPU?

Yes. OpenAI's gpt-oss repository says gpt-oss-120b (117B parameters, 5.1B active) runs on a single 80 GB GPU such as an NVIDIA H100 (OpenAI also names AMD's MI300X, which has 192 GB), and that gpt-oss-20b runs within 16 GB of memory, because the mixture-of-experts weights were post-trained with MXFP4 quantisation. Its other tensors stay in BF16.

It is a useful anchor because it shows that parameter count is not memory. The 120b checkpoint is 65.2 GB for 117 billion parameters, less than half the size of Llama 3.3 70B at 16-bit. Its cache is small as well, so, calculated for a single 80 GB H100, the simulator fits 33 conversations of 8K tokens beside the weights. And because each token reads only the 5.1 billion active parameters, it estimates about 110 tokens per second each for 10 people on that one GPU, against 52 for Llama 3.3 70B at 4-bit on the same card. Try it in the simulator with gpt-oss-120b.

What else changes the answer

• Conversation length. The cache grows with every token. At 32K tokens a Llama 3.3 70B conversation needs 5.4 GB with an 8-bit cache, so one 80 GB H100 with 4-bit weights holds 5 conversations instead of 21, and a 141 GB H200 holds 15 instead of 62.
• Cache precision. A 16-bit cache doubles every conversation: one 141 GB H200 with Jais 2 70B at 4-bit then holds 5 conversations of 8K tokens instead of 10.
• Speed. Memory decides whether the model runs; memory bandwidth decides how fast it answers. A dense 70B model reads all its weights for every token, which is why gpt-oss-120b, with far fewer active parameters, answers faster on the same card.
• Room to grow. A server sized to today's exact head count has nothing left for the next team or a longer document. Leave headroom, or choose a chassis with free GPU slots.

Licences: open weights, different terms

Check the licence with the hardware. Jais 2 70B is released under Apache 2.0, according to its model card, and so is gpt-oss. Llama 3.3 ships under the Llama 3.3 Community License, which Meta's model card calls a custom commercial license. It requires anyone who distributes or makes available the model, or a product or service that contains it, to display "Built with Llama", and anyone whose products had more than 700 million monthly active users in the calendar month before the Llama 3.3 release date to request a licence from Meta. The Open Source Initiative says the Llama 3.x licence terms do not meet its Open Source Definition.

How UBM handles it

We size from the workload, not the parameter count: which model, how many people at once, how long their conversations run and how fast answers must arrive. We run the numbers in the simulator with you, test the model at the precision you plan to use on your own documents during a bounded pilot, and leave headroom for growth before anything is ordered. Servers come from OEMs we already supply, with the accelerator chosen for the model, and we check the room's power and cooling against the maximum draw. Our engineers install the hardware, the network and the serving stack, then support it afterwards. The detail is on our AI solutions page; if the model will span more than one server, our article on Cat 6A or fibre covers the cabling. To size a model for your own team, talk to a UBM engineer.

Questions people ask

How much VRAM do I need to run a 70B model?

For the weights alone: 141 GB at 16-bit, 74 GB at 8-bit and 41 GB at 4-bit for Llama 3.3 70B (144, 75 and 41 GB for Jais 2 70B). Add the cache for every conversation and about 4.5 GB of runtime on a single GPU (less per GPU when the model is split), and count only 92 percent of each card as usable.

Can a 70B model run on a single GPU?

Yes, if the card is big enough for the precision. Among the cards compared here, Llama 3.3 70B on one GPU needs at least a 192 GB card at 16-bit, a 96 GB card at 8-bit and an 80 GB card at 4-bit. A 48 GB card is about 1 GB short even at 4-bit.

Can gpt-oss-120b run on one GPU?

Yes. OpenAI says it runs on a single 80 GB GPU such as an H100, because its expert weights ship in the 4-bit MXFP4 format. Beside its 65.2 GB of weights, the simulator's calculation for a single 80 GB H100 fits 33 conversations of 8K tokens.

Why does Jais 2 70B need more memory than Llama 3.3 70B?

Mostly because of the cache. Jais 2 70B uses multi-head attention, with 56 key-value heads on each of its 68 layers, while Llama 3.3 70B shares 8 across 80 layers. That is 1,904 KiB per token against 320 KiB, about six times as much for every conversation.

How many users can one 80 GB GPU serve on a 70B model?

With 4-bit weights, 8K-token conversations and an 8-bit cache, memory allows 21 on Llama 3.3 70B and 3 on Jais 2 70B. At 32K tokens, Llama drops to 5.

Does 4-bit quantisation reduce answer quality?

It can, and how much depends on the model, the quantisation method and the task. Test the quantised model on your own documents, in every language you work in, before you buy hardware around it.

Sources

• Jais-2-70B-Chat model card (MBZUAI, Inception and Cerebras, 2025)
• Jais 2: A Family of Arabic-Centric Open Large Language Models (Jais Team, 2026)
• Llama-3.3-70B-Instruct checkpoint (Meta, 2024)
• Llama 3.3 model card and Llama 3.3 Community License Agreement (Meta, 2024)
• The Llama 3 Herd of Models (Meta, 2024)
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (Ainslie et al., 2023)
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (Lin et al., 2023) and GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (Frantar et al., 2022)
• Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., 2023)
• Quantized KV Cache (vLLM documentation, 2026)
• gpt-oss repository and README and gpt-oss-120b config.json (OpenAI, 2025)
• Meta's LLaMa license is still not Open Source (Open Source Initiative, 2025)
• NVIDIA L40S, H100, H200 and RTX PRO 6000 Blackwell Server Edition specifications (NVIDIA, 2026)
• AMD Instinct MI300X specifications (AMD, 2026)