How many GPUs do I need for my AI workload?

The answer depends on the workload. For LLM fine-tuning on a 7B parameter model, 8 H100 GPUs is a reasonable starting point. For training a 70B parameter model from scratch, you need 64–512 GPUs depending on timeline. For inference serving a production LLM, the requirement depends on throughput and latency targets. The right answer requires a workload assessment, not a rule of thumb.

What is the difference between training and inference infrastructure?

Training infrastructure is optimized for throughput: moving as much data through the GPUs as quickly as possible. It requires high-bandwidth InfiniBand fabric, parallel storage, and GPUs with large HBM memory. Inference infrastructure is optimized for latency and cost-per-query: serving model responses quickly at low cost. Inference can use lower-cost GPUs (H100 PCIe vs. SXM5), standard Ethernet networking, and conventional storage. Many organizations run separate training and inference clusters.

Can I run AI workloads in my existing data center?

Possibly, but it requires a site readiness assessment first. The key constraints are power density (AI racks draw 10–30 kW vs. 3–5 kW for traditional IT), cooling capacity (liquid cooling is required above 20 kW), and structural load (AI servers are heavier than traditional servers). Many existing data centers can support a small AI deployment with upgrades, but cannot support a large cluster without significant infrastructure investment.

How long does it take to deploy AI infrastructure?

A purpose-built AI infrastructure facility takes 12–24 months from initial design to operational readiness. A smaller deployment in an existing facility can be completed in 6–12 months, depending on the extent of infrastructure upgrades required. GPU hardware lead times are currently 6–12 months, which is often the critical path.

What is the cost of AI infrastructure?

A small AI cluster (8 H100 GPUs) costs approximately $500K–$1M for hardware, plus $200K–$500K for infrastructure (power, cooling, networking, storage). A medium cluster (64 GPUs) costs $3M–$6M for hardware, plus $1M–$3M for infrastructure. A large cluster (512+ GPUs) costs $30M+ for hardware, plus $10M+ for infrastructure. These are rough estimates: actual costs depend on configuration, location, and vendor.

Should I build on-premises or use cloud?

For sustained, high-utilization AI workloads, on-premises infrastructure is typically 40–60% less expensive than equivalent cloud compute over a 3–5 year period. For variable workloads, prototyping, or organizations without the scale to justify on-premises infrastructure, cloud is appropriate. For regulated industries with sensitive data, on-premises is often required. The right answer depends on your specific workload, data, compliance requirements, and organizational scale.

What certifications should I require from an AI infrastructure vendor?

NVIDIA-certified engineers for GPU cluster design and deployment, PE-stamped electrical engineering for power infrastructure, NETA-certified commissioning for electrical systems, and Uptime Institute Tier Designer accreditation for data center design. These certifications indicate the depth of expertise required for mission-critical AI infrastructure.

What is InfiniBand and do I need it?

InfiniBand is a high-bandwidth, low-latency network fabric used to connect GPU nodes in AI training clusters. It provides 200–400 Gb/s per port with sub-microsecond latency: far beyond what standard Ethernet can provide. For large training clusters (64+ GPUs), InfiniBand is typically required to prevent network bandwidth from limiting GPU utilization. For smaller clusters and inference workloads, high-speed Ethernet with RoCEv2 is a viable alternative.

What cooling approach should I use for AI infrastructure?

The cooling approach depends on the planned rack density. For racks up to 15 kW, air cooling is sufficient. For racks up to 30 kW, rear-door heat exchangers (RDHx) are appropriate. For racks above 30 kW, direct-to-chip liquid cooling is required. For racks above 60 kW, full immersion cooling is the only viable option. Design the cooling system for the planned density, not the current density, retrofitting liquid cooling is expensive.

How do I protect against training job failures?

Implement a checkpoint strategy that saves model state at regular intervals (typically every 30–60 minutes for large training runs). Store checkpoints on fast NVMe storage co-located with the compute cluster. Define a recovery procedure that can restart training from the most recent checkpoint. Test the recovery procedure before running production training jobs.

AI Infrastructure FAQ

Executive Summary

These are the questions DCS Global engineers answer most frequently during initial client conversations. The answers are direct and specific, not hedged with 'it depends' without explaining what it depends on.

Key Takeaways

GPU requirements depend on workload type and scale, a workload assessment is required before sizing.
On-premises vs. cloud is a cost and compliance decision, not a technology preference.
Site readiness assessment is required before any infrastructure decision, power and cooling constraints are the most common project blockers.
GPU hardware lead times (6–12 months) must be factored into project timelines.
Vendor certification requirements (NVIDIA, PE stamp, NETA) are not optional for mission-critical deployments.

Questions and Answers

More AI Infrastructure Guides

Foundational

Beginner Overview

Plain-language introduction — what it is, why it matters, and how it fits into the broader infrastructure picture.

Strategic

Executive Brief

Business case, risk exposure, investment framing, and the three questions every executive should ask before approving a project.

Technical

Technical Overview

Architecture, components, design patterns, and the engineering decisions that determine long-term performance and reliability.

Decision

Buying Guide

Vendor evaluation criteria, RFP requirements, contract terms to negotiate, and the questions that separate qualified vendors from unqualified ones.

Implementation

Planning Checklist

Pre-project checklist covering site readiness, stakeholder alignment, compliance requirements, and the decisions that must be made before work begins.

Strategic

Common Mistakes

The ten most expensive mistakes organizations make — and the specific decisions that prevent each one.

Implementation

Implementation Roadmap

Phase-by-phase delivery plan with milestones, dependencies, go/no-go criteria, and the decisions that determine schedule performance.

Decision

Comparison Guide

Side-by-side comparison of approaches, vendors, and architectures — with the criteria that matter for enterprise procurement decisions.

Strategic

Related Categories

Compute Networking Power and Cooling Storage

Apply This Knowledge

Ready to move from research to decision?

DCS Global engineers can review your specific requirements and give you a direct assessment, not a sales pitch. Our infrastructure specialists have delivered a broad portfolio of projects across North America, Europe, the Middle East, and Asia-Pacific.

Schedule an Assessment View Solutions

AI Infrastructure FAQ

AI Infrastructure: Frequently Asked Questions

Questions and Answers

More AI Infrastructure Guides

Related Categories

Ready to move from research to decision?