Maia 300: Microsoft Bets on Its Own Chips for AI Inference

Starting in 2027, Microsoft wants to run its AI services primarily on its own chips rather than on graphics processors from Nvidia and other vendors. On October 12, the company unveiled Maia 300, the third generation of its AI accelerator, and announced plans to scale the Maia family to several gigawatts of capacity. For users of Copilot, GitHub Copilot, and Azure, this is about more than hardware: whoever lowers the cost of every generated word can offer AI features more cheaply and more quickly.

Key takeaways

• Maia 300 is slated to enter production in 2027 and become the primary system for inference, meaning running finished AI models, in Microsoft’s own products.
• Each chip carries eight HBM4 memory stacks with a combined 288 GB and 18 TB/s of bandwidth within a 950-watt design; TSMC manufactures the compute dies on its 2-nanometer N2P process.
• Microsoft promises 40 percent more tokens per megawatt than leading GPU systems and up to seven times the throughput per server rack of its predecessor. Both are vendor projections without a disclosed comparison setup.
• The predecessor, Maia 200, already runs in the United States, Europe, and Australia, serving Microsoft Copilot and OpenAI models among others; GitHub Copilot and Microsoft Foundry are set to follow by year-end.

What Maia 300 does differently

The announcement came from Scott Guthrie, who runs Microsoft’s cloud and AI business. An accompanying technical report from the Azure hardware group provides the details. The biggest change: Maia 300 is no longer one large chip but is made up of several smaller building blocks known as chiplets. The compute engines sit on their own chiplets built on TSMC’s advanced 2-nanometer process, while the memory controllers and I/O live on separate dies. That leaves more room for compute; Microsoft puts the resulting gain in compute performance at 36 percent.

The real focus, though, is memory. The eight HBM4 stacks deliver 18 TB/s, roughly two and a half times the bandwidth of Maia 200, and capacity rises by a third to 288 GB. This is not an end in itself. Modern AI applications work with very long inputs, multistep agent workflows, and conversations in which the model has to keep a lot of intermediate state in memory, the so-called KV cache. On tasks like these, the chip is often waiting not for compute but for data. Very compact number formats such as MXFP4, which store weights and intermediate state in just a few bits and so save memory and bandwidth, fit the same goal.

A server rack holds 96 accelerators with a combined 27.6 TB of memory and a power draw of 167 kilowatts. Up to 128 of these racks can be linked over optical connections into a cluster of 12,288 chips. Microsoft uses its own transport protocol over Ethernet rather than proprietary interconnect technology. Liquid cooling removes more than 80 percent of the heat.

Why Microsoft counts tokens per megawatt

The most striking number in the announcement is not raw compute but an efficiency metric: 40 percent more tokens per megawatt than leading GPU-based systems. Tokens are the word and syllable fragments a language model generates. That Microsoft measures things this way says a lot about the industry’s bottleneck. New data centers today are held back less by money than by available power. Anyone who can get more answers out of the same grid connection can serve more customers without waiting for the next substation.

The figure is a projection, however. Microsoft names neither the specific GPU system it compared against nor the test conditions, as the trade publication Converge Digest explicitly points out. The sevenfold gain over Maia 200 also applies to a fully built-out rack with twice as many accelerators as before and is not a measurement from production. How Maia 300 will stack up against Nvidia’s upcoming generation, which we recently covered through the example of Vera Rubin at the cloud provider CoreWeave, cannot be read from these numbers.

What users and businesses get out of it

For most people, the chip stays invisible. It can still make a noticeable difference. Microsoft names specific services that are to run on Maia 300: multistep coding agents in GitHub, serving models from different providers in Microsoft Foundry, the MAI team’s own model development, and Dragon Copilot, an AI assistant that takes documentation work off physicians’ plates and needs fast responses. For Maia 200 alone, Microsoft says it runs AI features in Teams, Copilot, and Cowork with up to 1.7 times more tokens per watt than on leading GPU hardware.

For businesses in Germany and the rest of Europe, the key point is that Maia already runs in European data centers. Azure customers are likely to notice the effects through pricing and capacity rather than through a new product lineup: the announcement does not say whether or when customers will be able to rent Maia compute directly. More concrete details will probably come at Microsoft’s Ignite developer conference in November.

The strategic intent is clear. Like Google with its TPU chips and Amazon with Trainium, Microsoft wants to depend less on Nvidia, whose graphics processors are scarce and expensive. The industry publication The Information reported in August that Microsoft was negotiating with TSMC for manufacturing capacity for more than 300,000 Maia 300 chips in 2027; Microsoft has not officially confirmed any volume. This is not a complete break with Nvidia. Maia complements the GPU fleet rather than replacing it, especially since the chip is explicitly built for running finished models, not for training them.

Bottom line: the cost question is shifting to inference

With Maia 300, Microsoft is making a remarkably clear bet. It is not the training of ever larger models that its own chips are meant to make cheaper, but their day-to-day operation, which grows more expensive with every agent and every long context window. That the company built the chip around memory and power efficiency shows where it sees the limit of the coming years. Whether the math works out will only become clear in production in 2027, and the promised 40 percent will have to hold up against real Nvidia systems. If it succeeds, users will be the main beneficiaries: AI features that are still tightly budgeted today or locked behind expensive subscriptions will become cheaper for providers to run.