How to Enable and Test NVIDIA NVLink on Quadro and GeForce RTX Cards in Windows 10

Introduction

NVLink is a technology from NVIDIA for creating a high bandwidth link between two of their video cards. It can be used for many things, from basic SLI for faster gaming to potentially pooling GPU memory for rendering large and complex scenes. What NVLink can be used for depends on how software developers write their applications, and there is a lot of exciting potential for this technology.

However, as of the time we are publishing this, there is no easy way to tell if NVLink is enabled and functioning within Windows 10. Further, the way to enable NVLink is different depending on what sort of video cards you have. We have put together this guide to help: covering the different ways to enable NVLink, along with a small utility to easily test and make sure it is working as expected.

If you have already set up your system for NVLink, you can skip ahead to download and run the NVLinkTest utility.

Installing the Physical NVLink Bridge

Regardless of what cards you are using, for NVLink to function they must be connected by a physical NVLink bridge (or in some cases, two bridges). The bridge needs to be the right size to reach between the cards, which can potentially be 2, 3, or 4 PCI-E slots apart.

In addition to being the right size, the bridge you use also needs to be compatible with the video cards. For example, older Quadro GP100 bridges do not work on the newer GeForce RTX series cards. On the other hand, the GeForce bridges do seem to work on the Quadro GP100 cards – except that they are physically larger, so you can only fit one bridge instead of the two that the GP100 is designed to use. The safest option is to stick with bridges specifically designed for the video cards you are using. Some other combinations may work, but are not likely to be officially supported by NVIDIA.

Here are examples of what NVLink bridges can look like, as well as what it looks like to have them installed on cards:

Enabling NVLink on GeForce and Quadro RTX Video Cards

Once the physical link is securely installed, it is quite simple – though not obvious – how to enable NVLink on GeForce and Quadro RTX series cards. While NVLink itself is never mentioned in the NVIDIA Control Panel (so far as we could find) all you need to do is enable SLI. To do so, open the NVIDIA Control Panel, go to "Configure SLI, Surround, and PhysX" under "3D Settings" in the menu tree on the left, select "Maximize 3D performance" under "SLI configuration", and click Apply. Here is what that should look like, once SLI is enabled:

SLI itself is mostly used for gaming, and in the past we have recommended avoiding it for other applications – but if you want to use NVLink, this is the way to enable it on GeForce RTX 2080 and 2080 Ti cards, as well as the Titan RTX and Quadro RTX 5000, 6000, and 8000 models. Once enabled, you can test to make sure it is working.

Enabling NVLink on Quadro GP100 and GV100 Cards

NVLink originally debuted on the Quadro GP100 video cards, and is also found on the GV100. Both of these models require a pair of NVLink bridges for full performance, along with a more complex setup process than simply turning on SLI:

1) A third video card needs to be installed – ideally a Quadro from the same generation or newer than the cards you are bridging. This is required for video output, because enabling NVLink on the GP100 and GV100 cards will turn off their video outputs.

2) With both physical bridges installed and the third card connected to your monitor(s), open up the Windows command line. An easy way to do this is to right-click on the Start icon (not the normal left-click) and select Command Prompt.

3) Once that is open, navigate to "C:\Program Files\NVIDIA Corporation\NVSMI"

4) Run "nvidia-smi.exe -L" to see which numbers (starting with 0) are assigned to the two video cards you want to bridge.

5) For each of those cards, run "nvidia-smi.exe -i # -dm TCC", where # is the number of the GPU you wish to have in NVLink.

6) Once you have successfully run that command on both cards, reboot the system and test to see if NVLink is working. As mentioned before, TCC mode disables direct video output from those GPUs – so you need to have any monitors hooked up to additional cards.

This is how TCC is enabled on Quadro GP100s via the command line in Windows 10.

Download NVLinkTest to Verify NVLink Functionality

As of the publication of this article, there is no way to check NVLink status in the NVIDIA Control Panel. However, NVIDIA does supply some sample code in their CUDA Toolkit which can check for the peer-to-peer communication that NVLink enables and even measure bandwidth between video cards. You can download that toolkit, install Visual Studio, compile the sample code, and then run it – or…

We have compiled one of those sample programs and put together a simple GUI to make it easy to run in Windows 10. Just click the Download button below, save the linked ZIP file, and extract its contents. Inside you will find three files: a Readme.txt, NVIDIA’s p2pBandwidthLatencyTest.exe, and our NVLinkTest.exe. Run that last program and it will report whether NVLink is working or not.

9/16/2021 Update – We have added a CUDA 11 version of this utility, for use with newer NVIDIA drivers.

Download – CUDA 10 Download – CUDA 11

NVLinkTest Screenshots

Here are screenshots showing what NVLinkTest looks like while running. The first shows the intro screen, the second shows a successful run – where NVLink is enabled – and the remaining two show the errors you will get if CUDA is not available (which means you don't have a NVIDIA GPU or the NVIDIA drivers are not correctly installed) or if you do not have any active NVLink connections.

Additional Resources

If you want more info about NVLink in Windows, check out another article we published on the topic.

If you are interested in NVLink, Linux, and machine learning, check out Dr Kinghorn's HPC Blog.

For a list of all articles where we've talked about NVLink, use our website's search function.