Google dives into the ‘supercomputer’ game by knitting together purpose-built GPUs for large language model training

AI scientists and anybody with very huge computation wants will now be capable to flip to Google’s cloud to lease machines that will ship as a lot as 26 exaFLOPs. The brand new cloud choices, detailed at immediately’s keynote speech at Google I/O 2023, are choices that resurrect that Chilly Warfare period nomenclature of “supercomputers” due to their extraordinary capabilities and concentrate on very huge duties.

The brand new machines are constructed by combining Nvidia’s H100 GPUs with Google’s personal high-speed interconnections. The corporate expects that the mix of quick GPUs linked by quick information pathways will likely be very enticing for AI duties like coaching very giant language fashions. 

Very giant language fashions

The rise of those very giant fashions is reigniting curiosity in {hardware} that may effectively deal with very giant workloads. AI scientists have seen essentially the most jaw-dropping outcomes after they stretch the dimensions of the mannequin as giant as doable. New machines like it will make it simpler to push them larger and larger. 

Google’s new machines are enticing as a result of they’re in a position to speed up communications between the GPUs, which is able to, in flip, speed up the convergence of the mannequin as it’s skilled. The Nvidia GPUs will talk utilizing what Google describes as “custom-designed 200-Gbps IPUs” that supply “GPU-to-GPU information transfers bypassing the CPU host and flowing over separate interfaces from different VM networks and information visitors.” The corporate estimates that the information will movement between the GPUs 10 occasions sooner than a few of their earlier {hardware} with extra conventional communications paths. 

Most of the cloud providers supply some machines that ship the extremely parallel efficiency of the GPU or TPU. Amazon’s Internet Providers, for instance, offers a half-dozen completely different choices that mix a number of GPUs or a few of their new ARM-based Graviton chips. Google itself presents their very own chips, dubbed TPUs, in a variety of combos.

On the identical time, common GPUs have gotten commonplace. Even a few of the smaller clouds like Vultr have GPUs for lease, one thing that they provide at charges as little as 13 cents per hour for a fraction of a machine. 

Google is clearly aiming on the largest workloads with this announcement. Its new machines, labeled the A3, will bundle as much as 8 H100 GPUs from Nvidia constructed with the video processor producer’s HOPPER structure. Every machine may have as much as 2 terabytes of RAM for storing the coaching information. All of this will likely be synchronized by a fourth-generation Xeon processor.

Google is a part of an even bigger recreation

Google is just not the one firm headed down this path. In November, Microsoft introduced a partnership with Nvidia to provide their very own “supercomputer.” The corporate can even be utilizing chips just like the H100 as constructing blocks for interconnected “materials” or “meshes” optimized for coaching these very giant fashions. 

In February, IBM introduced it’s also constructing its personal model dubbed “Vela” that may practice very giant fashions for a few of its authorities prospects like NASA. These “basis fashions” will assist with many sciences like drug discovery or cybersecurity. 

One other huge objective for Google will likely be integrating this new {hardware} with its software program and cloud choices. OpenAI, for example, resells Azure’s computation by making it doable for its personal customers to fine-tune their very own foundational fashions. 

Google says the {hardware} will likely be out there by Vertex AI for patrons “seeking to develop complicated ML fashions with out the upkeep.” On the identical time, they’re additionally asserting expanded options and extra foundational fashions.