Unreal 4.27 adds multi-GPU support for lightmass building

Always look at the date when you read an article. Some of the content in this article is most likely out of date, as it was written on September 9, 2021. For newer information, see our more recent articles.

Table of Contents

Introduction

The new 4.27 release of Unreal Engine offers several new features that will be useful to those in the Virtual Production and ArchViz spaces. One interesting feature that will have a big demand on hardware is multi-GPU support for GPU Lightmass. For anyone that relies on baked lighting, any way to speed up that process is something to get excited about. Let’s take a look at how this feature works, and what caveats exist.

Setting it Up

The first thing to note is that GPU Lightmass, with any number of video cards, is still a Beta feature, even though Unreal Engine has move version 4.27 to production-ready. Further, it is not as simple as installing version 4.27 and connecting two video cards. GPU light baking is not enabled by default, instead, you will need to install the GPU Lightmass plugin. Then, you will need to enable ray tracing in the project settings then, counter-intuitively, you will also need to disable real-time ray tracing features using the editor’s console. This seems odd, but it is telling the engine that it can use Windows DXR API for light baking, but that it doesn’t need to do any ray tracing in the real-time viewport.

That is how you get one GPU to work, to get multiple GPUs will need some extra tweaks. First, the DefaultEngine.ini needs to be edited to add a section about allowing multi-GPU in the editor. Then when you launch the editor, you will need to do so while also passing a command line argument with how many GPUs are in use. This must be done every time you launch the editor, or you can edit a shortcut to include that argument.

For specifics, check out Epic's documentation on enabling multiple GPUs.

Hardware Considerations

On the hardware side, you will need two video cards connected with an NVLink bridge, and SLI enabled in the NVIDIA control panel. That means two identical cards as NVLink does not support connecting different models, for example, you can’t connect an RTX A6000 with an RTX A5000 or last generation’s RTX 6000. NVIDIA has been making fewer and fewer cards with NVLink support. Of the current generation, only the professional line RTX A6000 and RTX A5000, or the GeForce RTX 3090 have NVLink connectors, making this a costly option.

The GPU's RAM will also need to be considered. For the light bake, the entire scene must be loaded into the GPU with the lowest level (highest detail) LOD. This can be highly problematic for large game levels, or projects that use Level Streaming. ArchViz and VP projects are typically smaller and should have no problems with the video cards listed above as they have at least 24GB. Older RTX 2080’s may be constrained by their 8GB VRAM.

Additionally, the CPU must have enough RAM to load all the generated lightmaps. As the GPU completes each map, they will be transferred to the CPU for encoding. However, this encoding does not happen until the entire bake process is completed. So while the GPU is able to offload some resources during calculation, the system's RAM needs to be up to the task. The amount of RAM needed will depend on the number of textures and their sizes. This shouldn’t be too concerning but is worth considering if you are upgrading an older system.

Another big aspect to consider is that all denoising and encoding of the final lightmaps will take place on the CPU once the full process is complete. If the project contains a significant number of maps, this could take a lot of time depending on the CPU, lessening the speed increase of the second GPU. The encoding process appears to be multi-threaded so higher core count CPUs should be faster.

Other limitations

There are several limitations in the current iteration of the tool that will likely be addressed in future updates. Many should have relatively easy workarounds. Things such as World Position Offset, Pixel Depth Offset, Stationary Sky Lights, and Lighting Channels are not currently supported. Slight changes to the workflow should be able to overcome this until official support arrives.

Volumetric Lightmap calculations are not yet supported on GPUs. This feature is mainly useful for characters and animated objects that wouldn’t have static shadows. So again, less of a demand for Virtual Production and ArchViz, but more for Game Developers. That is not to say you can’t use Volumetric Lightmaps, just that they will be calculated by the CPU. If your project relies heavily on this feature, Multi-GPU will not be much help.

Another big feature missing from GPU Lightmass is distributed baking via the Swarm Agent. This allows users to distribute the light baking to any number of available computers in the studio to speed up the process. GPU lightmass is already significantly faster than CPU, so they are using multi-GPU as an alternative to get even faster results. This shouldn’t be an issue for studios that only have a few systems as the speed increase of even one modern GPU could outperform most CPUs. But for larger game developers that may have access to hundreds of computers or even racks of servers, CPU lightmass may still be the way to go. Epic states in their documentation that there are no plans to bring GPU Lightmass to the Swarm Agent.

Is it worth it?

Exactly how much of an improvement you will see will vary from project to project, and on the specs of the rest of the system. According to Epic’s documentation, on a “medium-sized scene” with minimal volumetric Lightmap usage, they saw build times about 1.7x faster than a single card. That is already a pretty ideal scenario. If you have a larger scene, need more volumetric lightmaps or have an older 4 core CPU you may not see those kinds of improvements. We’ll need to do some more in depth testing with projects of various sizes, with different CPUs to get a better understanding.

With the current limitations and the fact that the GPU Lightmass plug-in is still in Beta, it is hard to recommend anyone build a system with multiple GPUs just for light baking. Outside of light baking, Unreal doesn’t have much use for multiple GPUs. The much-improved path tracer renderer does not support it, nor does the real-time viewport. If you are already using multiple GPUs for other rendering tasks, or for powering an LED volume, then this may be worth investigating. Outside of that, I’d recommend against building a new system just for this feature until it is out of beta.