GPUs for LLM inference
As of — An 8B model at 16-bit: from $0.16 per GPU-hour on the NVIDIA V100 32GB (Geodd), 40 cards qualify; A 70B model at 8-bit: from $0.65 per GPU-hour on the NVIDIA RTX PRO 6000 Blackwell (Hostinger), 20 cards qualify; A 70B model at 16-bit on one GPU: from $1.71 per GPU-hour on the AMD Instinct MI300X (TensorWave), 10 cards qualify. Rates are each card’s cheapest live on-demand rate card across the providers we track, read hourly, every one linked to the page it was read from.
Serving a model keeps its weights in GPU memory: parameters times bytes per parameter - 2 bytes at 16-bit (FP16, BF16), 1 byte at 8-bit (FP8, INT8). The KV cache of every request being served comes on top, and grows with context length and concurrency, so the figures below are floors, not sizes.
An 8B model at 16-bit
8 billion parameters × 2 bytes = 16 GB of weights: a card with 24 GB or more.
| Card, memory | Cheapest on-demand | Provider, read | Median rate card | $ per GB-hour | Deploy |
|---|---|---|---|---|---|
| NVIDIA V100 32GB · 32 GB | $0.16 |  Geoddcaptured | $1.15 11 providers | $0.005 | Deploy ↗ |
| NVIDIA GeForce RTX 3090 · 24 GB | $0.18 |  OpenRelaycaptured | $0.42 10 providers | $0.007 | Deploy ↗ |
| NVIDIA A30 · 24 GB | $0.22 |  CloudGPUcaptured | $0.41 9 providers | $0.009 | Deploy ↗ |
| NVIDIA RTX A5000 · 24 GB | $0.27 from |  GPU.aicaptured | $0.58 10 providers | $0.011 | Deploy ↗ |
| NVIDIA L4 · 24 GB | $0.28 GPU only |  Hikubecaptured | $0.80 23 providers | $0.012 | Deploy ↗ |
| NVIDIA Quadro RTX 6000 · 24 GB | $0.31 GPU only |  centroncaptured | $0.51 4 providers | $0.013 | Deploy ↗ |
| NVIDIA A10 · 24 GB | $0.34 GPU only |  Intelion Cloudcaptured | $1.22 9 providers | $0.014 | Deploy ↗ |
| NVIDIA RTX 4090 · 24 GB | $0.35 | OpenRelay captured | $0.48 17 providers | $0.015 | Deploy ↗ |
| NVIDIA RTX A6000 · 48 GB | $0.35 |  Thunder Computecaptured | $0.59 21 providers | $0.007 | Deploy ↗ |
| NVIDIA A40 · 48 GB | $0.39 | CloudGPU captured | $0.64 8 providers | $0.008 | Deploy ↗ |
30 more cards qualify, dearer: NVIDIA GeForce RTX 3090 Ti $0.46, NVIDIA RTX 5090 $0.50, NVIDIA L40S $0.63, NVIDIA RTX 6000 Ada $0.65, NVIDIA RTX PRO 6000 Blackwell $0.65, NVIDIA A100 40GB $0.69, NVIDIA RTX PRO 4500 Blackwell $0.72, NVIDIA L40 $0.78, NVIDIA GB10 (DGX Spark) $0.79, NVIDIA RTX PRO 5000 Blackwell $0.80, NVIDIA RTX 5000 Ada Generation $0.83, NVIDIA A100 80GB $0.92, NVIDIA A10G $1.01, Intel Gaudi 2 $1.10, AMD Instinct MI300X $1.71, NVIDIA H100 80GB (form factor not stated) $1.72, NVIDIA H100 SXM $1.79, NVIDIA RTX PRO 6000 Blackwell Max-Q $1.80, NVIDIA H100 PCIe $1.98, AMD Instinct MI325X $2.25, NVIDIA GH200 $2.29, NVIDIA H200 $2.29, NVIDIA H100 NVL $2.92, AMD Instinct MI355X $2.95, NVIDIA H200 NVL $3.29, AMD Instinct MI350X $4.00, NVIDIA B200 $4.50, NVIDIA B300 $5.89, NVIDIA GB200 NVL72 $6.94, NVIDIA GB300 NVL72 $13.30.
A 70B model at 8-bit
70 billion parameters × 1 byte = 70 GB of weights: a card with 80 GB or more.
| Card, memory | Cheapest on-demand | Provider, read | Median rate card | $ per GB-hour | Deploy |
|---|---|---|---|---|---|
| NVIDIA RTX PRO 6000 Blackwell · 96 GB | $0.65 from |  Hostingercaptured | $2.20 42 providers | $0.007 | Deploy ↗ |
| NVIDIA GB10 (DGX Spark) · 128 GB | $0.79 | GPUwerk captured | $0.82 2 providers | $0.006 | Deploy ↗ |
| NVIDIA A100 80GB · 80 GB | $0.92 |  Yotta Labscaptured | $1.99 63 providers | $0.011 | Deploy ↗ |
| Intel Gaudi 2 · 96 GB | $1.10 |  Denvr Dataworkscaptured | $1.16 2 providers | $0.011 | Deploy ↗ |
| AMD Instinct MI300X · 192 GB | $1.71 from |  TensorWavecaptured | $2.65 7 providers | $0.009 | Deploy ↗ |
| NVIDIA H100 80GB (form factor not stated) · 80 GB | $1.72 |  Gcorecaptured | $3.75 28 providers | $0.022 | Deploy ↗ |
| NVIDIA H100 SXM · 80 GB | $1.79 GPU only |  Cudo Computecaptured | $3.29 41 providers | $0.022 | Deploy ↗ |
| NVIDIA RTX PRO 6000 Blackwell Max-Q · 96 GB | $1.80 |  EmpirioLabscaptured | $1.86 2 providers | $0.019 | Deploy ↗ |
| NVIDIA H100 PCIe · 80 GB | $1.98 from | GPU.ai captured | $2.82 10 providers | $0.025 | Deploy ↗ |
| AMD Instinct MI325X · 256 GB | $2.25 |  Bentauscaptured | $2.25 3 providers | $0.009 | Deploy ↗ |
10 more cards qualify, dearer: NVIDIA GH200 $2.29, NVIDIA H200 $2.29, NVIDIA H100 NVL $2.92, AMD Instinct MI355X $2.95, NVIDIA H200 NVL $3.29, AMD Instinct MI350X $4.00, NVIDIA B200 $4.50, NVIDIA B300 $5.89, NVIDIA GB200 NVL72 $6.94, NVIDIA GB300 NVL72 $13.30.
A 70B model at 16-bit on one GPU
70 billion parameters × 2 bytes = 140 GB of weights: a card with 141 GB or more (or the model split over two 80 GB cards).
| Card, memory | Cheapest on-demand | Provider, read | Median rate card | $ per GB-hour | Deploy |
|---|---|---|---|---|---|
| AMD Instinct MI300X · 192 GB | $1.71 from | TensorWave captured | $2.65 7 providers | $0.009 | Deploy ↗ |
| AMD Instinct MI325X · 256 GB | $2.25 | Bentaus captured | $2.25 3 providers | $0.009 | Deploy ↗ |
| NVIDIA H200 · 141 GB | $2.29 |  Dataoortscaptured | $4.56 56 providers | $0.016 | Deploy ↗ |
| AMD Instinct MI355X · 288 GB | $2.95 from | TensorWave captured | $6.14 4 providers | $0.010 | Deploy ↗ |
| NVIDIA H200 NVL · 141 GB | $3.29 from | GPU.ai captured | $3.62 7 providers | $0.023 | Deploy ↗ |
| AMD Instinct MI350X · 288 GB | $4.00 |  CloudRiftcaptured | $5.08 2 providers | $0.014 | Deploy ↗ |
| NVIDIA B200 · 192 GB | $4.50 from | Hostinger captured | $6.94 40 providers | $0.023 | Deploy ↗ |
| NVIDIA B300 · 288 GB | $5.89 |  Deep Infracaptured | $8.99 26 providers | $0.020 | Deploy ↗ |
| NVIDIA GB200 NVL72 · 186 GB | $6.94 from |  Bitdeer AIcaptured | $13.25 6 providers | $0.037 | Deploy ↗ |
| NVIDIA GB300 NVL72 · 288 GB | $13.30 |  Verdacaptured | $18.00 3 providers | $0.046 | Deploy ↗ |
How these figures are read
A card qualifies by its memory alone; speed, interconnect and software support are not ranked here. The rate is the card’s cheapest live on-demand rate card - each provider once, at its cheapest on-demand rate, community capacity and marketplace floors out, read within 48 hours - and the median is taken over the same rate cards, as on every card page. Memory figures are floors from the arithmetic above, not measurements: context length, batch size and the serving stack change them. Methodology.
Other workloads: GPUs for fine-tuning · GPUs for training · GPUs for image generation · Cheapest GPU memory per hour.
Data updated: