Evaluation Framework
Enterprise AI infrastructure procurement is a complex, high-stakes decision. A structured evaluation framework prevents common mistakes and ensures the selected solution meets both current and future requirements.
Evaluation should proceed in three phases:
- Requirements definition: Document workload requirements, performance targets, compliance constraints, and budget parameters before engaging vendors
- Vendor qualification: Assess vendor capability, financial stability, reference customers, and support quality
- Technical evaluation: Benchmark performance on representative workloads, validate integration with existing infrastructure, and assess total system performance
Critical Compute Questions
Q1: What GPU generation and memory configuration is included?
Specify exact GPU model (H100 SXM5 vs. PCIe, H200, B200), memory capacity (80GB vs. 141GB HBM3e), and memory bandwidth. These specifications directly determine training throughput and the maximum model size that can be trained or served.
Q2: What is the system-level performance on representative workloads?
Request benchmark results on workloads similar to your use cases — not just theoretical GPU FLOPS. Ask for MLPerf Training and Inference benchmark results, and request performance data on specific model architectures (Llama 2/3, Mistral, etc.) if relevant.
Q3: What is the NVLink topology within the server?
For multi-GPU servers, NVLink bandwidth between GPUs determines all-reduce performance during distributed training. DGX H100 provides NVLink 4.0 at 900 GB/s bidirectional; some OEM servers use PCIe-only connectivity with significantly lower bandwidth.
Q4: What CPU and memory configuration accompanies the GPUs?
CPU-to-GPU PCIe bandwidth, CPU core count, and system memory capacity affect data preprocessing performance and model serving overhead. Verify these specifications match your workload requirements.
Critical Networking Questions
Q5: What inter-node interconnect is included?
Specify whether InfiniBand (HDR 200 Gb/s, NDR 400 Gb/s) or Ethernet (100GbE, 200GbE, 400GbE) is included, the switch topology (fat-tree, dragonfly), and whether the network is non-blocking. For large training clusters, non-blocking InfiniBand NDR is the performance standard.
Q6: What is the storage network bandwidth?
Storage network bandwidth must be sufficient to keep GPUs fed during training. Request the aggregate storage bandwidth specification and ask how it scales with cluster size.
Critical Storage Questions
Q7: What storage system is included, and what is its aggregate throughput?
Specify the storage system (parallel file system vs. NFS vs. object storage), aggregate read and write throughput, and IOPS. For GPU-dense clusters, parallel file systems (GPFS, Lustre, WEKA) are required — NFS is generally insufficient.
Q8: How does storage performance scale with cluster size?
Storage must scale with compute. Ask for performance benchmarks at your target cluster size, and understand the storage scaling model (additional storage nodes, additional drives, etc.).
Software & Support Questions
Q9: What software stack is included and what are the licensing terms?
Clarify what software is included (NVIDIA AI Enterprise, CUDA, cuDNN, NGC containers), what requires separate licensing, and the terms for software updates and support. NVIDIA AI Enterprise licensing adds significant cost — verify it is included or budget for it separately.
Q10: What are the support SLAs and escalation procedures?
For mission-critical AI infrastructure, require 4-hour or next-business-day hardware replacement SLAs. Ask specifically: How are hardware failures handled? What is the process for GPU replacement? What is the average time to resolution for critical issues?
Q11: What professional services are available for deployment and optimization?
Complex AI infrastructure deployments benefit from vendor professional services for rack integration, network configuration, storage tuning, and software stack deployment. Clarify what is included in the purchase price vs. separately priced.
Q12: What is the vendor's roadmap for next-generation hardware?
AI hardware generations change rapidly. Understand the vendor's upgrade path from current to next-generation hardware (H100 → H200 → Blackwell), and whether infrastructure investments (networking, storage, power) will be compatible with future GPU generations.
Vendor Assessment Framework
Financial Stability
AI infrastructure is a 3–5 year investment. Vendor financial stability is critical — a vendor that exits the market or is acquired mid-contract creates significant operational risk. Assess: revenue and profitability trends, customer concentration, and strategic backing.
Reference Customers
Request 3–5 reference customers with similar workloads and scale. Ask references specifically: What problems did you encounter during deployment? How responsive is the vendor's support team? Would you purchase from this vendor again?
Engineering Depth
Assess the vendor's AI infrastructure engineering expertise: Do they have certified NVIDIA engineers? Can they provide architecture design services? Do they have experience with your specific workload types?
Supply Chain Reliability
GPU supply chain disruptions have caused significant delays. Ask vendors about their GPU allocation agreements with NVIDIA, typical lead times, and how they handle supply shortages.
Common Procurement Mistakes
- Specifying GPU model without system context: A server with H100 GPUs but PCIe-only connectivity performs significantly worse than a DGX H100 with NVLink for distributed training
- Underspecifying storage: Storage is frequently the performance bottleneck in AI clusters — specify throughput requirements explicitly
- Ignoring power and cooling requirements: Discovering that the data center cannot support the power density after hardware delivery is an expensive mistake
- Accepting list price: AI infrastructure is negotiable — engage multiple vendors and use competitive pressure
- Not modeling software costs: NVIDIA AI Enterprise, MLOps platforms, and monitoring tools add 15–25% to hardware costs
- Insufficient lead time planning: GPU hardware lead times of 6–12 months require procurement planning well ahead of deployment targets
RFP Template Outline
A complete AI infrastructure RFP should include:
- Executive summary and procurement objectives
- Technical requirements (compute, networking, storage, power, cooling)
- Performance requirements (training throughput, inference latency/throughput)
- Software requirements (OS, drivers, AI software stack)
- Support requirements (SLAs, escalation procedures, on-site support)
- Professional services requirements (deployment, integration, training)
- Compliance requirements (security certifications, data sovereignty)
- Pricing requirements (itemized hardware, software, services, support)
- Vendor qualification requirements (financial statements, references, certifications)
- Evaluation criteria and weighting
- Timeline and delivery requirements
- Contract terms and conditions