Modern AI systems depend on more than powerful chips.
Inside the data centers used to train and run AI models, accelerators need a constant flow of data. If the network cannot move that data quickly enough, expensive chips can spend part of a workload waiting instead of computing.
That delay matters. A cluster of accelerators worth tens of millions of dollars can only run as fast as the network feeding it. As AI systems grow larger, network design is becoming a major factor in how much value companies get from their infrastructure.
One issue is especially important: moving the same data to many machines at once.
Vijayananda Jayaraman has spent more than 20 years working on routing, forwarding and overlay technologies for enterprise and service-provider networks. A senior technical leader at Cisco, he has worked on systems that help networks move data reliably at scale.
He is also the author of Automating the Data Center Network, a book about building and operating modern network fabrics as systems that can be version-controlled, validated and continuously improved.
Why AI changes the network problem
AI training and inference can create enormous data movement inside a data center.
A single server with eight accelerators can produce high aggregate network demand. Large training runs may coordinate thousands of accelerators, with machines exchanging data repeatedly during a workload.
To keep up, data center fabrics have moved from 100-gigabit links to 400-gigabit and 800-gigabit links. Even then, the network can become a constraint.
The challenge is not only how fast the links are. It is also how efficiently data is distributed.
Sending one copy or many
At a high level, there are two ways to send the same data to many machines.
One approach is to send a separate copy to every receiver. If 1,000 machines need the same data, the source sends it 1,000 times. That approach is simple, but it uses more bandwidth and puts more load on the source and the network as the number of receivers grows.
Another approach is to send the data once and let the network duplicate it only where paths split. In that model, one stream leaves the source and branches deeper in the network, closer to the machines that need it.
The first approach becomes more expensive with every receiver. The second is designed to scale more efficiently.
“The naive way feels free, because each copy is easy. The bill only shows up when you count them,” Jayaraman said. “Reaching 1,000 receivers by copying means you paid to send the same thing 1,000 times. Do that with AI-scale payloads and you are burning the exact bandwidth that decides whether your accelerators stay busy.”
Why the cloud moved away from multicast
Networking solved part of this problem decades ago with multicast, a family of technologies that can send one stream and duplicate it inside the network.
But cloud computing changed the assumptions.
Public cloud platforms often run on shared, multi-tenant virtual networks. Native network-layer multicast can be difficult to isolate safely between tenants and can create risks for shared infrastructure. As a result, major cloud providers have often restricted or avoided it.
That pushed many cloud-native systems toward application-level copying, where software sends separate copies to each destination.
Jayaraman made that argument in Send Once, Reach Many: Why One-to-Many Distribution Is Becoming a Bottleneck Again, saying the industry traded some network efficiency for cloud simplicity.
“A whole generation learned that you reach many machines by making many copies, because in the cloud that was the only option,” Jayaraman said. “It became invisible, just the way things are done. But it was a workaround, not a law of physics, and workarounds carry a cost that compounds quietly until one day it is your biggest line item.”
AI made the old problem expensive again
For years, sending extra copies may have been acceptable for many applications.
AI has changed that calculation.
Model checkpoints can be hundreds of gigabytes. Training systems may need to push parameters to many workers. Inference fleets may need the same model updates distributed across large numbers of servers. When the data is large and the audience is wide, inefficient distribution can become expensive quickly.
That cost can show up as wasted bandwidth, higher server load and idle compute.
“We spent a decade optimizing the chips and treating the network as plumbing,” Jayaraman said. “Now the plumbing is the bottleneck. When a single model update is hundreds of gigabytes and it has to reach thousands of machines, the difference between sending it once and sending it thousands of times shows up as idle accelerators and real money.”
Good distribution still needs control
Efficient distribution is not only a bandwidth problem. It is also a control problem.
A system needs to know which machines should receive a stream, when they should receive it and how the delivery tree should change as the workload changes. If membership information is wrong or outdated, the system can become inefficient or unreliable.
That kind of control-plane work is central to large-scale networking.
In AI data centers, the goal is to build a distribution tree so each network link carries the payload only when necessary. Copies should be made closer to the receivers rather than repeatedly from the source. For applications that need complete delivery, reliability also has to be added above the transport layer.
“Membership is the hard part, not the copying,” Jayaraman said. “Knowing exactly who needs this stream right now, and updating the delivery tree as membership changes, at the speed a modern system changes, is where the real engineering lives. Get that wrong and your efficient design quietly degrades back into sending copies.”
What comes next
AI has made the network a larger part of the infrastructure conversation.
Companies are spending heavily on data center fabrics because the network now affects how much of their expensive compute capacity they can actually use. Efficient one-to-many distribution is returning through overlay approaches, renewed standards work and AI workloads that make brute-force copying harder to justify.
For teams designing systems that move the same data to many machines, the question is becoming more practical than theoretical: how many copies are really being sent, and how many are actually needed?
Jayaraman said the answer will shape the next generation of AI infrastructure.
“The best trick the internet ever had was to send one copy and let the network do the duplicating,” Jayaraman said. “The cloud set it aside for simplicity, and for years that trade looked fine. It does not look fine anymore. The teams that remember how to do this will move data at a fraction of the cost of the teams that brute-force it, and at AI scale, that gap only widens.”