SC22 Nvidia says customers can now lengthen their 400Gbps InfiniBand networks even additional with the rollout of its MetroX-3 long-range system, which boosts the vary of its Quantum-2 switches to 25 miles or 40 kilometers.
Nvidia sees two principal use instances for the expertise: one is high-speed workload migration between bodily disparate compute facilities, whereas the opposite includes pooling these sources to handle bigger issues.
Whereas it was beforehand potential to interconnect two knowledge facilities over InfiniBand utilizing the MetroX-2 platform acquired from Mellanox, the equipment was restricted to at least one pair of 100 Gbps uplinks. By comparability, the third iteration of Nvidia’s MetroX platform provides help for a pair of 100Gbps dense wave division multiplex (DWDM) modules. The expertise permits considerably larger bandwidths by combining a number of 100Gbps indicators on a single fiber.
“This actually permits your massive campus clusters and your principal knowledge middle to behave like a single unit, a single knowledge middle,” stated Dion Harris, head of knowledge middle product advertising at Nvidia, throughout a press convention.
The method just isn’t with out compromises. MetroX-3 is clearly an ecosystem recreation. It’s designed to work with Nvidia’s InfiniBand ecosystem of Quantum-2 switches, ConnectX-7 NICs, and/or BlueField knowledge processing items (DPUs). Which means should you’re already utilizing one thing like HPE’s Ethernet-based Slingshot interconnects and even Nvidia’s Spectrum switches, MetroX-3 is not for you.
Assuming you reside inside Nvidia’s walled backyard or wish to broaden an current InfiniBand atmosphere to a brand new location, there are additionally efficiency concessions that have to be thought of. Whereas DWDM permits for enormous combination bandwidths on a single fiber, the expertise is proscribed to comparatively quick runs on the order of 40-80 kilometers. As is typical with optics, what you acquire in distance you quit in efficient bandwidth and vice versa.
Switching to DWDM has a price benefit. In line with Dell’Oro analyst Jimmy Yu, the extra you may pack onto a single strand of fiber, the much less you have to spend on fiber rental to get a given quantity of bandwidth.
And since DWDM is already used extensively by main telecommunications corporations AT&T and Lumen Know-how, Nvidia says clients can now faucet into current fiber infrastructures, somewhat than needing devoted fiber connectivity.
Slicing by way of the noise
MetroX-3 is a component of a bigger suite of {hardware} and software program introduced by Nvidia at this week’s Supercomputing occasion to handle the rising quantity of knowledge streaming to the sting.
“By creating extra analysis and high-fidelity instrumentation, which means you will have a way more environment friendly method of buying, analyzing and processing that knowledge,” Harris stated. Whenever you “produce 50 to 1,000 instances extra knowledge, how a lot do you retain? How a lot do you return to the core? How a lot do you analyze.”
Along with connecting knowledge facilities on InfiniBand, Nvidia is positioning MetroX-3, together with its Quantum Collection switches and BlueField DPUs, as a method to increase InfiniBand networks into lab environments the place nearly all of knowledge is generated. In doing so, the corporate says clients can use the Holoscan HPC framework operating on IGX, DGX, or HGX platforms on the edge to filter significant knowledge out of noise earlier than funneling that refined dataset into the first knowledge middle.
Initially launched alongside Nvidia’s IGX robotics and edge computing platform on this case, Holoscan’s AI inference was for medical imaging. Nevertheless, the platform has since been repurposed to be used on quite a lot of streaming knowledge codecs, together with non-image codecs. Holoscan has additionally been reworked to help the C++ and Python APIs, which Nvidia says researchers can use to construct customized knowledge pipelines round their workflows. ®
