GPU and AI Workloads: Matching Hardware to What You Are Actually Doing

"AI workload" covers at least three things with entirely different hardware requirements. Buying capacity for the wrong one is the most expensive mistake available in this area.

The three profiles

Training from scratch needs very large amounts of memory and interconnect bandwidth, usually across multiple GPUs and often multiple nodes. It is bursty, long-running and expensive. Very few organisations do this.

Fine-tuning adapts an existing model. Memory requirements are much lower, particularly with parameter-efficient methods such as LoRA. Often a single GPU is enough. Runs are hours, not weeks.

Inference serves a trained model. It is latency-sensitive, runs continuously and is dominated by memory bandwidth and model size rather than raw compute. This is what most production AI workloads actually are.

GPU memory is the gate

Whether a model runs at all is decided by whether its weights fit in GPU memory. Roughly:

• At 16-bit precision, a model needs about 2 GB of GPU memory per billion parameters, plus overhead for activations and the KV cache.
• At 8-bit quantisation, about 1 GB per billion.
• At 4-bit quantisation, about 0.5 GB per billion, with some quality cost.

Serving concurrent requests adds KV cache consumption that scales with context length and batch size, and this is routinely underestimated. A model that fits comfortably at batch size one may not fit at batch size sixteen with long contexts.

The managed-API alternative

Before provisioning GPUs, ask whether you need them at all.

For inference against general-purpose foundation models, a managed API is usually cheaper, faster to adopt and entirely free of capacity management. You pay per token rather than per hour, which is dramatically better economics for intermittent workloads.

Self-hosting inference is justified when:

• Data cannot leave your environment for regulatory reasons.
• You are serving a model that is not available through an API.
• Volume is high and sustained enough that per-hour beats per-token.
• Latency requirements demand a model physically close to your application.

On AWS, Bedrock provides managed foundation-model access with per-region throughput quotas measured in requests and tokens per minute. Those quotas are the operative constraint — an AWS account with model access already granted and higher RPM limits removes a waiting period that can otherwise block a project. Confirm the specific models, the region and the actual RPM figure rather than treating access as binary.

On Google Cloud, Vertex AI covers the same ground and GPU quota is per region and per accelerator family — a Google Cloud account with approved accelerator quota in the region you need is meaningfully more useful than one without.

Quota is the real bottleneck

GPU capacity is constrained everywhere. Three things gate you:

1. Account quota for the accelerator family, per region. New accounts have zero.
2. Physical availability in that region at that moment.
3. Service limits on managed inference endpoints.

Quota increases are reviewed rather than automatic, and can take days. This is why environments with approved accelerator quota are described that way explicitly — the approval is the scarce part, not the money.

Controlling cost

GPU instances are the most expensive compute available, and idle ones are pure waste:

• Shut down between runs. A training instance left running overnight can cost more than the run.
• Use spot capacity for training with checkpointing, and recover from interruption. Discounts are large.
• Right-size the accelerator. A smaller GPU that fits the model beats a larger one at 15% utilisation.
• Batch inference requests to raise utilisation per GPU-hour.
• Quantise where quality permits. It reduces both memory and cost.
• Separate training from serving. They have opposite utilisation profiles and belong on different capacity.

Responsible operation

Two obligations come with this workload and are worth stating:

• Provider acceptable-use policies apply to model workloads. Accelerator capacity is a scarce, heavily monitored resource, and policies governing its use are enforced.
• Data protection obligations follow the data. Sending personal data to an inference endpoint is a processing activity, with the consent, transparency and residency requirements that implies.

Frequently asked questions

Do I need a GPU for inference? Not always. Small models and moderate throughput run acceptably on CPU, particularly quantised. Benchmark before assuming.

Training or fine-tuning? Almost certainly fine-tuning, or retrieval augmentation with no training at all. Training from scratch is rarely the right answer outside research.

How do I estimate GPU memory? Roughly 2 GB per billion parameters at 16-bit, halving per quantisation step, plus KV cache for your batch size and context length.

Why can I not launch a GPU instance? Almost always quota at zero for that family and region, or genuine regional unavailability. Both need addressing before capacity exists.