Hardware

There is a quiet but very expensive bet being made across the AI compute layer right now, and I think it deserves more scrutiny than it is getting. The bet is that the best way to handle the growing demand for AI compute is to build silicon shaped around the architecture we have today. In practice, that means chips increasingly tuned for transformers. Etched is the clearest example, with hardware designed explicitly around transformer workloads. But the broader pattern shows up across the industry too: more memory bandwidth tuned for attention, more matrix throughput tuned for the operations LLMs actually use, and more interconnect tuned for the shapes of current models. ...

I came across this infographic and spent more time than I expected just reading through it. It’s a good snapshot of how much the anatomy of a server rack has shifted in the last few years. A rack used to be mostly about compute and storage. CPUs on top, drives somewhere in the middle, some networking at the top, and air blowing through the whole thing. The job of the infrastructure was to stay out of the way of the workload. Now the workload is the infrastructure. GPUs are the centre of gravity, and everything else, power distribution, cooling, interconnects, cable management, is designed around keeping them fed and cold enough to run flat out. ...