Shedding Light on the New AI Bottleneck

You can own every GPU on Earth and still leave most of them sitting idle.

That would have sounded absurd two years ago, but it’s now the central engineering problem in artificial intelligence (AI).

The AI race spent its first few years obsessed with semiconductors and GPUs. The next chapter is about the wire between the chips, and very few outside the networking world are watching it yet.

It all comes down to the fact that a frontier AI model is trained across hundreds of thousands of semiconductors optimized for AI. These chips are wired together so tightly that they have to behave as a single, enormous computer.

That’s the part that sometimes gets lost when we talk about AI compute.

Training AI models entails 100,000+ chips each operating in parallel – doing calculations together. At every step they stop, compare notes, and synchronize before moving on.

Thus, every GPU waits for the slowest message to arrive before the whole cluster can take its next step, meaning the entire machine runs at the speed of its slowest connection.

You can have the fastest processors ever built, but if they spend their time waiting to hear from each other, the technological edge is curtailed.

And the bigger the cluster, the tougher the problem.

More chips means the machine sprawls out across more physical space. And more space means the signals have to travel farther.

As modern AI workloads have grown from a room to a hall to multiple buildings on a campus… the distance a message must cross has quietly become one of the hard limits on how fast the whole system can operate.

The Copper Wall

A central question has been prevalent throughout the entire history of the computing era – how do we move data between chips?

For decades now, the answer has been the same: push electrons down a copper wire. It’s cheap, it’s proven, and it has worked well up to this point.

But that answer just doesn’t cut it anymore.

Copper has physical limits. Signals traveling too far, too fast across copper wires tend to degrade. Plus, the process loses energy through heat, which reduces efficiency.

Engineers call this the “interconnect wall,” and the AI buildout is running into it. Demand for bandwidth between chips is now rising faster than copper can physically deliver.

As such, the frontier of AI is no longer just the chip. It’s the conversation between chips. And that brings us to a hot topic right now – photonics.

Don’t Send Electrons – Send Light

Photonics is the technology of using light (photons) rather than electricity to generate, control, transmit, and detect signals. And photonics is especially advantageous for high-bandwidth, low-loss data movement between chips. We call these “optical interconnects.”

The beauty here is that light doesn’t degrade over distance the way electrical current does. Plus, it can carry vastly more information and burns far less energy on its journey.

We’ve moved data between buildings on fiber optic lines for decades now. That was an early form of photonics at work.

What’s new is that the industry is now putting the optics onto the semiconductor package itself, right beside the processor – co-packaged optics.

The idea is to convert the electrical signal to light at the last possible millimeter, so the data spends as much of its journey as possible traveling as light and as little as possible fighting through copper.

This is no longer theoretical.

NVIDIA has unveiled a line of silicon-photonics switches built expressly to wire together what it calls “million-GPU AI factories.”

Its own numbers make the stakes plain: fusing the optics directly into the switch delivers, by the company’s account, 3.5x better power efficiency, 63x better signal integrity, and four times fewer lasers than the conventional approach.

That power figure is the tell. In a world where electricity is the binding constraint on AI, an interconnect that does the same job on a third of the energy is vital.

And NVIDIA is far from alone.

The supporting cast reads like a who’s who of the physical technology economy – Taiwan Semiconductor (TSM) building the photonic chips, Corning (GLW) drawing the glass, and specialized optics firms like Lumentum (LITE), Coherent (COHR), and Fabrinet (FN) supplying the components… all with Broadcom (AVGO), Marvell (MRVL), and Cisco (CSCO) racing down parallel paths.

And as supporting evidence for how big the trend is, a startup called Lumilens just walked out of stealth with more than $900 million in funding for the single purpose of breaking these connectivity bottlenecks.

When that much capital and that many of the industry’s most serious players converge on one problem inside a single year, we can be sure that something big is in motion.

And here’s where it gets especially interesting…

The Bottleneck Beneath the Bottleneck

Every one of these silicon-photonics systems has to generate the light in the first place. But here’s the catch: silicon is great at steering light but lousy at producing it.

That being the case, each package needs a tiny laser, and those lasers are built from a specialty semiconductor called indium phosphide – a material that I imagine most folks have never heard of and will never think about.

The problem is, there isn’t enough of it.

The CEO of Lumentum – one of the key suppliers of indium phosphide – recently warned that production of this material already runs roughly 30% below what customers need. And he called the looming shortage potentially “worse than memory,” which suggests that a brutal supply crunch is coming.

It appears that a critical resource bottleneck is developing beneath the connectivity bottleneck.

The bottom line is this… every time compute gets abundant, the value migrates to whatever moves the data between the compute.

It happened inside the processor decades ago, when chips got so fast the wiring between their internal parts became the limit. Then it happened with memory, when processors outran the pipes feeding them data.

And it’s happening now at the scale of entire buildings, as the industry discovers that a hundred thousand bleeding-edge chips are only as useful as their ability to talk to one another.

So the AI story isn’t slowing down, but the frontier is moving. It’s moving off the chip and into the space between chips. And that space, increasingly, is going to be made of glass and light.

The companies that master the movement of light through an AI factory will collect a toll on every calculation today’s massive “AI factories” ever perform.

Regards,

Joe Withrow
Senior Analyst, Brownstone Research