Chip Is the Easy Part
Why I stopped comparing AI Chips on Spec Sheets
There’s a pattern I’ve noticed when teams choose AI hardware.
The conversation starts with spec sheets - memory, bandwidth, TFLOPS.
The chip with the better numbers gets picked.
Six months later, the team is fighting fires nobody saw coming.
Kernels that should exist, don’t.
Operations that should be fast, aren’t.
Engineers who should be productive, aren’t.
The chip met its spec sheet. The system didn’t perform.
This is the part of AI infrastructure that’s hardest to account for: the chip alone does nothing. What makes silicon useful is the ecosystem around it - the software layers, the libraries, and the tooling.
This post is about that ecosystem. What it actually consists of, and what an architect should evaluate when comparing options. If Part 1 was the framework and Part 2 was the chip tour, this is the part where you understand why picking chip is actually the easy part.
What the ecosystem actually is
When people say software stack in AI infrastructure - there are actually four layers, and each one matters differently.
Layer 1: The programming model. This is what you use to write code that runs on the chip. The Neuron SDK on AWS Trainium. CUDA on Nvidia. XLA-based stacks on Google TPU. ROCm on AMD. This is the layer most people think of when they think about software for a chip.
Layer 2: The kernel libraries. These are pre-tuned implementations of the operations every AI workload uses - matrix multiplication, attention, layer normalization, collective communication across chips. Most AI workloads spend 80%+ of their time inside these libraries, not in user code. The quality and coverage of these libraries determines real-world performance more than almost anything else.
Layer 3: The compilers and frameworks. PyTorch and JAX at the top. Below them, compilers like torch.compile, XLA, and Triton that take high-level model code and produce optimized kernels for specific hardware. This is the layer that lets researchers write models without thinking about chip-specific details.
Layer 4: The training and serving systems. vLLM, SGLang, TensorRT-LLM, and runtimes like AWS NeuronX for inference. DeepSpeed, FSDP, Megatron-LM for training. These handle the operational complexity - batching, KV cache management, parallelism strategies - that turns a model into a production service.
Each layer trades generality for performance.
A model written in pure PyTorch is the most flexible and the slowest.
A model compiled with a production inference runtime is the fastest and least flexible.
Most real systems use multiple layers from multiple sources.
The reason this matters: when you buy a chip, you’re inheriting all four layers’. A platform with a great Layer 1 but immature Layer 2 will not meet what’s on spec sheet.
Why the ecosystem matters
Ecosystems compound. Every layer reinforces the others.
Start with kernel libraries. When a new chip generation ships, someone has to rewrite the kernel libraries to exploit its new features - new precision formats, new memory hierarchies, new interconnect capabilities. This is lot of engineering work per generation. If the work happens after launch, the chip underperforms for months while the libraries catch up.
Then frameworks. PyTorch and JAX both support multiple hardware backends, but “support” is a spectrum. Common operations work everywhere. Uncommon operations work on the most-mature backend and may not exist on others.
Then tooling. Profilers, Debuggers, Memory inspectors and Compiler diagnostic tools. When a kernel runs at 20% of expected speed, you need tooling to figure out why.
A couple of examples
AWS has built out the Neuron stack - Neuron SDK as the programming model, Neuron Kernel Interface for low-level kernels, PyTorch and JAX integration, NeuronX for serving, and the Neuron Profiler for diagnostics - all co-evolving with Trainium and Inferentia silicon.
Nvidia’s stack follows the same shape: CUDA, cuDNN, cuBLAS, NCCL, TensorRT-LLM, Nsight.
What to actually evaluate
Spec sheet comparisons miss the ecosystem entirely. To evaluate platforms in 2026:
Run your actual workload, not the headline benchmarks.
Every platform optimizes its headline benchmarks aggressively - whatever the public leaderboards measure.The platform that wins on MLPerf may not win on your code.
Budget for what’s missing, not just what’s there.
When you adopt a platform, you’ll find gaps - a kernel that doesn’t exist, an operation that’s slow, a feature that’s documented but doesn’t work. Each gap costs your team time to work around.
Evaluate the operational tier, not just the programming tier.
Inference servers, training frameworks, observability, orchestration - Layer 4 - is where production headaches live. A chip with great Layer 1 and immature Layer 4 looks fine in a benchmark and breaks in production. Whatever platform you’re evaluating, look at how it integrates with vLLM or the equivalent inference runtime, how it works with PyTorch Lightning or DeepSpeed for training, what observability tools exist, and whether failure modes are debuggable.
What I’d take away
In a nutshell, when we speak of AI infrastructure, two chips with identical spec sheets can deliver radically different real-world performance, productivity, and operational pain - and the difference is almost always in the four layers around the silicon, not the silicon itself.
The chip is the easy part. The stack is the rest.
Next post: A tour of the physical ecosystem - the memory next to the chip, the network across the data center, the choreography of getting thousands of chips to act as one.If you want it in your inbox, subscribe.