{"id":443,"date":"2026-10-16T23:03:44","date_gmt":"2026-10-16T19:03:44","guid":{"rendered":"https:\/\/mdtoman.com\/gpu-server-buying-guide-sizing-llm-ai-workloads\/"},"modified":"2026-10-16T23:03:44","modified_gmt":"2026-10-16T19:03:44","slug":"gpu-server-buying-guide-sizing-llm-ai-workloads","status":"publish","type":"post","link":"https:\/\/mdtoman.com\/ar\/gpu-server-buying-guide-sizing-llm-ai-workloads\/","title":{"rendered":"GPU server buying guide: how to size hardware for LLMs and AI workloads"},"content":{"rendered":"<div class=\"mdt-guide-answer\"><strong>Short answer:<\/strong> Size the GPU memory (VRAM) first, because it decides which models you can run. Running a model at 16-bit precision needs about 2 GB per billion parameters (a 7B model is about 14 GB), and 4-bit quantisation cuts that to roughly a quarter. Full fine-tuning needs about 16 bytes per parameter. Then check power, cooling and networking, because GPU servers draw several kilowatts.<\/div>\n<p>Buying AI hardware is easier when you work from the workload backwards. This guide gives the sizing rules we start from for businesses in Oman and the UAE that want to run language models, computer vision or analytics on their own premises. Every project is still quoted individually after a conversation about the workload.<\/p>\n<h2>Step 1: what is the workload?<\/h2>\n<ul>\n<li><strong>Inference<\/strong> means running a trained model to answer questions or analyse images. It needs enough memory to hold the model and good throughput for your users.<\/li>\n<li><strong>Fine-tuning<\/strong> adapts an existing model to your data. It needs more memory than inference, and much less than training from scratch.<\/li>\n<li><strong>Training from scratch<\/strong> needs many GPUs, fast interconnects and large datasets. Few businesses need it; most need inference and fine-tuning.<\/li>\n<\/ul>\n<h2>Step 2: estimate GPU memory for inference<\/h2>\n<table>\n<thead>\n<tr>\n<th>Model size<\/th>\n<th>Weights at 16-bit<\/th>\n<th>Weights at 4-bit quantised<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>7 billion parameters<\/td>\n<td>about 14 GB<\/td>\n<td>about 4 GB<\/td>\n<\/tr>\n<tr>\n<td>13 billion<\/td>\n<td>about 26 GB<\/td>\n<td>about 7 GB<\/td>\n<\/tr>\n<tr>\n<td>34 billion<\/td>\n<td>about 68 GB<\/td>\n<td>about 17 GB<\/td>\n<\/tr>\n<tr>\n<td>70 billion<\/td>\n<td>about 140 GB<\/td>\n<td>about 35 GB<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>These figures are for the weights only. Add about 20 to 40% for the context cache and the runtime, and more for long prompts or many users at once. A model that needs more memory than one GPU has can be split across several GPUs.<\/p>\n<h2>Step 3: estimate memory for fine-tuning<\/h2>\n<p>Full fine-tuning with mixed precision and the Adam optimiser needs about 16 bytes per parameter for weights, gradients and optimiser state: a 7B model needs roughly 112 GB before activations. Parameter-efficient methods such as LoRA, and QLoRA on a quantised base model, need far less, and a 7B model can often be tuned on a single 24 GB GPU. Larger models need more memory.<\/p>\n<h2>Step 4: workstation or server?<\/h2>\n<table>\n<thead>\n<tr>\n<th><\/th>\n<th>AI workstation<\/th>\n<th>GPU server<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Typical GPUs<\/td>\n<td>1 to 4 professional GPUs<\/td>\n<td>4 to 8 data-centre GPUs per server<\/td>\n<\/tr>\n<tr>\n<td>Best for<\/td>\n<td>Developers, data scientists, designers, small teams, proof of concept<\/td>\n<td>Shared use by many people, production inference, fine-tuning larger models<\/td>\n<\/tr>\n<tr>\n<td>Cooling and noise<\/td>\n<td>Tower, office-friendly<\/td>\n<td>Rack-mounted, needs a server room<\/td>\n<\/tr>\n<tr>\n<td>Memory and reliability<\/td>\n<td>Large VRAM on professional cards<\/td>\n<td>Highest memory and bandwidth, error-correcting memory, fast GPU-to-GPU links, built for 24\/7 duty<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2>Step 5: the rest of the system<\/h2>\n<ul>\n<li><strong>CPU and PCIe:<\/strong> each GPU needs a full-speed PCIe slot (x16), so the platform matters as much as the cards.<\/li>\n<li><strong>System memory:<\/strong> a common rule of thumb is at least as much RAM as total GPU memory, so models load quickly.<\/li>\n<li><strong>Storage:<\/strong> fast NVMe for datasets and model checkpoints, and shared storage plus backups for anything you cannot lose.<\/li>\n<li><strong>Network:<\/strong> 10 to 25 Gb\/s is usually enough for a single server serving users. Multi-server training usually needs 100 Gb\/s or faster.<\/li>\n<\/ul>\n<h2>Step 6: power and cooling, the part that surprises people<\/h2>\n<p>Data-centre GPUs commonly draw 300 to 700 W each, and a fully loaded 8-GPU server can draw 5 to 10 kW. Almost every watt becomes heat: 1 kW is about 3,400 BTU\/h, so an 8 kW server needs around 27,000 BTU\/h (about 2.3 tons) of cooling for that one machine.<\/p>\n<ul>\n<li>Check the power available to the rack, the circuit ratings and the redundancy of the power supplies.<\/li>\n<li>Size a UPS for the load and the minutes you need, and consider a generator for critical use.<\/li>\n<li>In the Gulf, do not rely on comfort air-conditioning. Plan precision cooling with redundancy, hot and cold aisle separation, and temperature monitoring.<\/li>\n<\/ul>\n<h2>Own the hardware or rent cloud GPUs?<\/h2>\n<p>Cloud GPUs suit short experiments and unpredictable peaks. Owned hardware often wins for steady daily use, for data that must stay on your premises, and for systems that sit next to your cameras and business applications. To compare, divide the three-year cost of hardware, power, cooling and support by the GPU hours you expect to use, and compare that with the cloud hourly price.<\/p>\n<h2>What to tell us before we quote<\/h2>\n<table>\n<thead>\n<tr>\n<th>Question<\/th>\n<th>Why it matters<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Which models or applications, and what size?<\/td>\n<td>Sets GPU memory<\/td>\n<\/tr>\n<tr>\n<td>Inference, fine-tuning or both?<\/td>\n<td>Sets memory, GPU count and interconnect<\/td>\n<\/tr>\n<tr>\n<td>How many users at the same time?<\/td>\n<td>Sets throughput and number of GPUs<\/td>\n<\/tr>\n<tr>\n<td>How much data, and how fast does it grow?<\/td>\n<td>Sets storage and backup<\/td>\n<\/tr>\n<tr>\n<td>Where will it be installed, and what power and cooling are there?<\/td>\n<td>Decides workstation, server or room works<\/td>\n<\/tr>\n<tr>\n<td>Budget and timeline<\/td>\n<td>Shapes the configuration<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<section class=\"mdt-guide-faq\">\n<h2>Frequently asked questions<\/h2>\n<h3>How much GPU memory do I need to run a 70B model?<\/h3>\n<p>About 35 to 45 GB with 4-bit quantisation and about 140 GB or more at 16-bit precision, so usually several GPUs or one very large GPU.<\/p>\n<h3>Do I need a GPU, or is a normal server enough?<\/h3>\n<p>Small models can run on a CPU, but slowly. GPUs deliver far higher throughput, which matters once several people use the model.<\/p>\n<h3>Do I need a fast GPU-to-GPU link?<\/h3>\n<p>Only when a model or training job is split across GPUs that exchange a lot of data. Independent inference copies on separate GPUs do not need it.<\/p>\n<h3>Can MDT install the power and cooling too?<\/h3>\n<p>Yes. We plan the server room, racks, power, UPS, cooling and cabling together with the servers, and support them afterwards.<\/p>\n<\/section>\n<p><strong>Next step:<\/strong> describe your workload in an <a href=\"https:\/\/mdtoman.com\/ar\/?mdt_type=ai#contact\">AI hardware quotation request<\/a>. Read more about our <a href=\"https:\/\/mdtoman.com\/ar\/services\/ai-gpu-infrastructure\/\">AI and GPU infrastructure service<\/a> and <a href=\"https:\/\/mdtoman.com\/ar\/services\/servers-server-rooms\/\">servers and server rooms<\/a>. We work in <a href=\"https:\/\/mdtoman.com\/ar\/cctv-oman\/\">Oman<\/a> and <a href=\"https:\/\/mdtoman.com\/ar\/cctv-uae\/\">the UAE<\/a>.<\/p>","protected":false},"excerpt":{"rendered":"<p>How to size GPU servers and AI workstations: memory for inference and fine-tuning, system design, power and cooling, and what to tell a supplier.<\/p>","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[312],"tags":[],"class_list":["post-443","post","type-post","status-publish","format-standard","hentry","category-guides"],"_links":{"self":[{"href":"https:\/\/mdtoman.com\/ar\/wp-json\/wp\/v2\/posts\/443","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mdtoman.com\/ar\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mdtoman.com\/ar\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mdtoman.com\/ar\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/mdtoman.com\/ar\/wp-json\/wp\/v2\/comments?post=443"}],"version-history":[{"count":0,"href":"https:\/\/mdtoman.com\/ar\/wp-json\/wp\/v2\/posts\/443\/revisions"}],"wp:attachment":[{"href":"https:\/\/mdtoman.com\/ar\/wp-json\/wp\/v2\/media?parent=443"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mdtoman.com\/ar\/wp-json\/wp\/v2\/categories?post=443"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mdtoman.com\/ar\/wp-json\/wp\/v2\/tags?post=443"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}