Running AI on your own servers in the UAE: LLM hardware requirements, from one GPU to a cluster

LLM hardware requirements come down to three numbers: GPU memory for the model's weights, more memory for a KV cache per conversation, and memory bandwidth for speed. OpenAI's gpt-oss-20b runs on one 24 GB card. A 70B model at 8-bit outgrows one 80 GB GPU but fits one 141 GB GPU or two 80 GB cards.

The server model, the rack and the cooling all follow from those numbers. The worked figures below come from UBM's AI infrastructure simulator, run on 1 October 2026. Memory is exact arithmetic on each model's published architecture; speeds are estimates, good to about plus or minus 25 percent, not a performance promise.

LLM hardware requirements: the three numbers that size a server

A private large language model is a file of numbers (the weights) and a program that serves it. Running it on your own hardware asks three things of the server:

• Weight memory. The whole model must sit in GPU memory: the parameter count multiplied by the bytes stored per parameter.
• Conversation memory (the KV cache). Every conversation in progress keeps its own working memory on the GPU, growing with every token. More people at once, or longer documents, means more memory.
• Memory bandwidth. To write each new token, the GPU reads the model's active weights from memory again. How fast it reads sets how fast the answer appears.

Compute matters mainly when the model reads a long prompt or serves many people at once. For someone waiting on an answer, memory capacity decides whether the model runs at all, and bandwidth decides how quickly it replies.

How much GPU memory do the model weights need?

Multiply the parameters by the bytes per parameter. At BF16, the format the Llama models are released in, each parameter takes 2 bytes, so a 70 billion parameter model needs about 140 GB. FP8 halves that, and 4-bit formats come to about half a byte per parameter once their scaling factors are counted. Smaller formats usually cost a little quality, so test them on your own documents.

Weights from the UBM simulator, in GB of one billion bytes; where a lab publishes the checkpoint, its real size is used.

The gpt-oss models are mixture-of-experts designs: every parameter must be in memory, but each token uses only a few of them. OpenAI's gpt-oss repository says the 120b model (117B parameters, 5.1B active) fits into a single 80 GB GPU and the 20b model runs within 16 GB of memory, because the expert weights ship in the 4-bit MXFP4 format. Both are Apache 2.0 licensed.

The KV cache: why every extra user needs more memory

For each token in a conversation, every layer stores a key and a value so it does not have to recompute them. Per token, that is 2 × layers × key-value heads × values per head × bytes per value. Meta's Llama 3 paper gives the Llama 3 70B architecture as 80 layers with 8 key-value heads, and 8,192 dimensions over 64 heads makes 128 values per head. This article assumes the same architecture for Llama 3.3 70B, whose model card lists 70B parameters and grouped-query attention. At BF16 that is 327,680 bytes (320 KiB) for every token of every conversation.

Architecture changes this a lot. The configuration of gpt-oss-120b has 36 layers, half of which look back only 128 tokens, and 8 key-value heads of 64 values. Its cache grows by 36 KiB per token, about a ninth of Llama's.

One 128K-token conversation on Llama 3.3 70B at BF16 needs almost 43 GB, more than half of an 80 GB GPU, so concurrency and context length belong in the specification as firmly as the model name. The PagedAttention paper behind vLLM describes this cache as huge and growing and shrinking dynamically, which is why vLLM allocates it in small pages instead of reserving the maximum.

At FP8 with 32K-token conversations, the simulator puts the 73.7 GB of weights and 4.7 GB of runtime on one 141 GB GPU and leaves room for 9 conversations of 5.4 GB. Split over two 80 GB GPUs, each card holds 36.8 GB of weights and 3.6 GB of runtime, with room for 12 conversations at 2.7 GB per card. On one 80 GB GPU the weights alone exceed the 73.6 GB the simulator counts as usable.

How fast will it answer? Memory bandwidth and tokens per second

Generating one token means reading the active weights, plus the cache, from GPU memory, so the ceiling is bandwidth divided by bytes read per token. Llama 3.1 8B at BF16 reads 16.1 GB per token, so a 4.8 TB/s H200 tops out near 300 tokens per second; the simulator, allowing for real-world efficiency, estimates 211 for one user with an 8K-token conversation. A dense 70B model at FP8 reads 73.7 GB per token and runs at an estimated 49 tokens per second on the same GPU, while gpt-oss-120b reads only its 5.1B active parameters and reaches an estimated 363.

Memory, bandwidth and power from the vendor pages under Sources; the B200 figures are the DGX B200's 1,440 GB and 64 TB/s divided by eight. Speeds are simulator estimates for one user with an 8K-token conversation; at 32K tokens they are about 5 percent lower. The simulator also applies a software-maturity factor to non-NVIDIA accelerators and shows it with every result.

A worked example: sizing a GPU server for an LLM

Take an internal assistant on Llama 3.3 70B at FP8, used by 30 people at the same moment, with conversations of up to 16K tokens, at no less than 20 tokens per second each. You can open the same scenario in the simulator and change any input.

Its lowest-cost answer on 1 October 2026 was one 2U server with two H200 NVL GPUs. On each GPU the simulator counts 129.7 of the 141 GB as usable (the 92 percent allowance for the serving engine that it lists with every result); each card holds half the model (36.8 GB) and 3.6 GB of runtime, leaving 89.3 GB for conversations at 1.3 GB each. That serves 66 people at the target speed: about 51 tokens per second each with 30 active, and 77 for a single user.

Simulator output, 1 October 2026. Power includes the front-end and management switches the simulator adds; speeds are estimates.

Two inputs move the answer most. Double the conversation length and each conversation's cache doubles. Swap the dense 70B for gpt-oss-120b and the same 30 people fit on four 48 GB L40S cards, because that model is smaller in memory and reads far fewer bytes per token.

One server or a cluster?

Stay inside one server for as long as the model fits. Inside a server, GPUs split every layer between them (tensor parallelism) and exchange results at each layer, so the link matters: NVIDIA lists 900 GB/s of NVLink per H200, against 128 GB/s for PCIe Gen5. Between servers, the model is usually split into stages of layers (pipeline parallelism) over the network. NVIDIA counts NVLink in both directions (its Hopper architecture page gives 900 GB/s bidirectional per GPU, more than seven times PCIe Gen5), so the like-for-like figure for an 800 Gb/s network port is 200 GB/s: 100 GB/s each way.

NVIDIA's eight-GPU DGX H200 holds 1,128 GB of GPU memory and the DGX H100 640 GB, according to the DGX H100/H200 user guide. Llama 3.1 405B at BF16 (811.7 GB) therefore fits the H200 system but not the H100 one, which needs FP8 (416.1 GB). Models with more than about 1 TB of weights need two such servers, or one built on HGX B300, which NVIDIA lists at 2.1 TB. A cluster adds a GPU network switch and fibre between racks; our article on Cat 6A or fibre covers the cabling side.

GPU or CPU: can a server without GPUs run an LLM?

Yes for small models and light use, but bandwidth still sets the pace. vLLM lists x86, ARM and PowerPC CPUs among its supported platforms. AMD's EPYC 9005 datasheet gives up to 12 channels of DDR5-6400 per socket, which works out at 614.4 GB/s at most (12 × 6,400 MT/s × 8 bytes). Reading 73.7 GB per token, a 70B model at FP8 could not pass about 8 tokens per second per socket even in theory, against an estimated 49 on one H200. Unified-memory desktops sit in between: NVIDIA's DGX Spark pairs 128 GB with 273 GB/s and is rated for inference on models up to 200 billion parameters, a sensible size for development and pilots.

Power and cooling for AI servers in UAE server rooms

NVIDIA rates the DGX H200 at 10.2 kW maximum from six 3.3 kW power supplies, in 8U, weighing up to 130.45 kg, and the DGX B200 at about 14.3 kW in 10U. Every watt becomes heat. With the NIST conversion factors, 10.2 kW is about 34,800 BTU/h, or 2.9 tons of refrigeration, and 14.3 kW is about 48,800 BTU/h, or 4.1 tons.

• Cooling. A room sized for a few switches will not absorb 10 kW, and in the UAE it has to hold through the summer peak. The simulator assumes a PUE of 1.4 for an air-cooled UAE room, listed with its other assumptions under every result, and flags racks above 20 kW as needing liquid cooling or rear-door heat exchangers.
• Power. Plan dedicated circuits, redundant PDUs and UPS capacity for the maximum draw, not the typical one.
• Space and weight. Check rack depth, floor loading and the delivery route before the order.

The configurations in the worked example draw 3.55 to 6.25 kW at full load, switches included, which many existing rooms can take after a cooling check. As with the ELV room in our ELV systems guide, these decisions are cheapest on the drawings.

The serving stack: Ollama, vLLM and a chat interface

• Ollama (MIT licence): simple to install, and suited to pilots and single machines.
• vLLM (Apache-2.0): a high-throughput serving engine with PagedAttention and an OpenAI-compatible API server, supporting NVIDIA, AMD and Intel GPUs, with plugins for Intel Gaudi, Huawei Ascend and others. We use it for multi-user serving.
• Open WebUI: a popular chat front end, with a licence caveat. Since v0.6.6 (19 April 2025) it uses BSD-3 plus a branding clause that its licence page says is not OSI-approved. The branding may be removed only with 50 or fewer users in a rolling 30-day period, with written permission as a substantive contributor, or under an enterprise licence. Code up to v0.6.5 remains BSD-3.

The model has its own licence too: gpt-oss is Apache 2.0, while Llama 3.3 ships under the Llama 3.3 Community License, a custom commercial licence. Read it before production.

On-premise AI infrastructure and keeping data in the UAE

A model on your own server keeps prompts, documents and answers on hardware you control, which makes the data residency question simpler to answer. It does not make a system compliant by itself. The UAE Government portal describes Federal Decree-Law No. 45 of 2021 as applying to the processing of personal data through electronic systems inside or outside the country, and as setting requirements for cross-border transfer; the DIFC has its own Data Protection Law (DIFC Law No. 5 of 2020). Access control, logging and retention still have to be designed, for prompts and outputs as for any other record.

How UBM handles it

We start with the workload: which model, how many people at once, how long their conversations run and how fast answers must be. We size it in the simulator with you, and write down the data residency position and the access model before anything is ordered. The servers come from OEMs we already supply (Dell, Lenovo and Huawei, plus workstations from HP, Dell and Lenovo for development), with NVIDIA, AMD, Intel or Huawei accelerators chosen for the model, and we check the room's power, cooling and rack space against the maximum draw. Our engineers install the hardware, the network and the serving stack, run a bounded pilot on real data, and support it as a service afterwards. The detail is on our AI solutions page; to size a model for your own workload, talk to a UBM engineer.

Questions people ask

How much GPU memory does a 70B model need?

About 141 GB for the weights at BF16 and 73.7 GB at FP8, plus a runtime reserve and the KV cache for every conversation. At FP8 that rules out one 80 GB GPU; one 141 GB GPU or two 80 GB GPUs work.

Can I run a private LLM on a server without a GPU?

For small models and light use, yes. A CPU socket with 12 channels of DDR5-6400 peaks at 614.4 GB/s, against 4.8 TB/s on one H200, so a 70B model at FP8 on a CPU stays below about 8 tokens per second per socket.

How many people can one GPU server serve?

It depends on the model, the conversation length and the speed you need. In the simulator, one 2U server with two H200 NVL GPUs serves 66 people at once on Llama 3.3 70B at FP8 with 16K conversations, at 20 tokens per second or more each.

How accurate are the simulator's numbers?

Memory (does it fit, how many conversations) is exact arithmetic on each model's published architecture. Speed is modelled from bandwidth, compute and interconnect, so treat it as plus or minus 25 percent. Prices are indicative; UBM quotes the real figure.

Does running AI on-premises keep our data inside the UAE?

The model and the data stay on hardware you control, wherever you install it. Compliance still depends on the regime that applies to you, such as the federal Personal Data Protection Law (Decree-Law No. 45 of 2021) or the DIFC's own law, and on access, logging and retention being designed in.

Sources

• gpt-oss repository and README (OpenAI, 2025) and gpt-oss-120b config.json (OpenAI, 2025)
• The Llama 3 Herd of Models (Meta, 2024)
• Llama 3.3 model card (Meta, 2024)
• Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., 2023)
• NVIDIA H200 specifications, H100, L40S, L4 and RTX PRO 6000 Blackwell Server Edition (NVIDIA, 2026)
• NVIDIA Hopper architecture (NVIDIA, 2026)
• NVIDIA DGX B200 specifications and NVIDIA HGX platform (NVIDIA, 2026)
• NVIDIA DGX H100/H200 User Guide, hardware overview (NVIDIA, 2026)
• NVIDIA DGX Spark specifications (NVIDIA, 2026)
• AMD Instinct MI300X specifications (AMD, 2026)
• AMD EPYC 9005 Series processor datasheet (AMD, 2025)
• Intel Gaudi 3 AI Accelerator white paper (Intel, 2024)
• NIST Guide to the SI, Appendix B.9 conversion factors (NIST SP 811, 2008)
• Ollama licence (MIT), vLLM repository (Apache-2.0) and Open WebUI licence (2025 and 2026)
• Data protection laws (UAE Government portal, updated December 2025)