How to Buy a Workstation with NVLink

NVLink is a proprietary NVIDIA interconnect for high-speed communication between physically bridged video cards. Its use in desktops and workstations is something which a lot of folks have heard of but not everyone fully understands – thanks in no small part to sparse information from NVIDIA themselves and conflation with the similar, but older, SLI technology. If you are considering NVLink as a feature for your next Puget Systems workstation, here is how to go about getting it.

Step 1: Make sure your workflow can use NVLink

While having a high-speed connection between video cards sounds exciting, most major applications are not designed to take advantage of it for one reason or another. In many cases, software isn't even built to utilize multiple GPUs – so having support for NVLink is a rather moot point. In other programs, more than one GPU can be used but there may be no direct communication going on between the cards, rendering NVLink pointless. In still other cases, it could be helpful but the developers have not coded in support for it.

If you don't have any portions of your workflow where NVLink is properly detected and utilized, then there is no need to worry about including it as a feature of your next computer. In my experience, it is most commonly supported (on Windows) by GPU based rendering engines – but because it is a connection between just one pair of cards, it isn't always a good choice even for folks using such applications. If you want to have three or four video cards, just having one pair of them in NVLink isn't going to provide a material benefit. In fact, the most useful place for NVLink seems to be scientific computing on Linux – especially machine learning and AI development.

Step 2: Pick a platform which supports NVLink

In this regard, I am referring to the combination of core hardware and operating system as a "platform". There are several things involved in making sure that you pick a set of components that will work with NVLink:

• Two PCI Express slots large enough for video cards (so x16 size, physically) spaced 2, 3, or 4 slot widths apart
• A power supply with enough wattage to handle two high-end video cards, along with the rest of the hardware in the system
• Plenty of airflow to cool the cards under full load (the chassis we carry are great for this, and we usually upgrade case fans too)
• For Windows-based systems, a motherboard that is officially SLI compatible

That last one may seem odd – after all, we are talking about NVLink here, not SLI. However, the primary way to enable NVLink in Windows 10 is to turn on SLI. Remember how I mentioned NVIDIA being responsible for some of the confusion about NVLink? Here is a prime example! To ensure that you can enable SLI, and that it will behave as expected, you'll want a motherboard that is officially SLI compatible. You might be able to get away without that, but if something went amiss the motherboard maker would likely point out that you were trying to use a feature they had not tested or certified. Because of that, here at Puget we are limiting Windows configurations with NVLink to properly supported motherboards. As of the time I am writing this, the only SLI compatible motherboards in our line are for AMD's Threadripper and Threadripper Pro processors.

Linux has no such limitation, so far as I am aware, and NVLink "just works" on that operating system – as long as the physical bridge is installed correctly (and not defective). This allows us to offer NVLink on a much wider range of motherboards when using Linux.

Step 3: Select NVLink compatible video cards

Most NVIDIA graphics cards do not support NVLink – it is a feature they reserve for their top-end models, both on the consumer (GeForce) and professional side of things. In the current generation of Ampere-based desktop GPUs, only video three cards actually feature NVLink compatibility:

• GeForce RTX 3090 24GB
• RTX A5000 24GB
• RTX A6000 48GB

You'll note that there is no brand name listed for those last two cards. That is because NVIDIA dropped the "Quadro" branding that used to be associated with their professional graphics cards, and so far as I can tell they have decided not to replace it with anything. That makes referring to these products as a family rather difficult, but for the rest of this post I will call them the "A-series".

A pair of any of the above cards can be connected via a NVLink bridge, as long as they are spaced out appropriately on a motherboard. Bridges exist in 4-slot (with GeForce branding), 3-slot, and 2-slot sizes (both matching the appearance of A-series cards)… but all of those bridges work equally well across these three card models in my experience.

You can select the primary video card on our configure pages, and then pick a second one in the Accelerators category. But how do you get a NVLink bridge onto your quote?

Step 4: Talk to a consultant

We specifically chose to stop listing NVLink bridges publicly a while back, because we found that people were picking them even when their workflow couldn't make use of NVLink – or sometimes even when the motherboard or video cards themselves were not compatible with it. Instead, we now leave the ability to add a NVLink bridge to our consultants. Once you've got the rest of the hardware selected the way you want, simply save your configuration and reach out to us (or wait for us to contact out to you, as we do with every new and returning customer who saves a quote). You can then request that a NVLink bridge be added to your specs list, and we will make sure that the right size for your particular motherboard and video card combination is used. This will add some cost to your quote: NVIDIA charges a surprising amount for these, considering their size, and oddly the 2- and 3-slot versions are more expensive than the 4-slot.

If you want, you can also skip setting up the hardware configuration yourself and just contact us from the get-go. Our consultants will walk through our various workstation options with you, discuss your needs, and put together a recommendation with NVLink included. Here is a form you can use to get started:

Closing Thoughts

If you do decide to configure and purchase a workstation with NVLink – congratulations! I hope it serves you well for many years to come. Here are a couple closing thoughts that may help as you get started with it (for Windows – if you have Linux questions, ask Dr. Kinghorn):

• You may run into applications which don't like SLI / NVLink being enabled. If you get lower performance than you expected, or a multi-GPU aware program doesn't show your second video card, try toggling SLI off in the NVIDIA Control Panel and see if that helps. It is easy to turn on and off, though sometimes it prompts you to close out of running applications first.
• After updating NVIDA graphics drivers, double-check the SLI setting to make sure it is still enabled. I prefer doing "clean" driver installations, which remove the existing driver first – and in my experience those sometimes require re-enabling SLI afterward.