Why cant my new workstation edit H.264 media smoothly?

Introduction

When you invest in a Puget Systems workstation, we spend considerable effort making sure that you are getting the best possible system for your workflow. And yet, even the most powerful workstations we can produce sometimes struggle with one of the most common codecs: H.264.

Is it the fault of the hardware? Are Adobe or Blackmagic developers just bad at optimizing their software? The answer is really neither. Instead, it comes down to the fundamentals of how H.264 works that makes it such a difficult codec to edit with.

Why is H.264 hard to edit?

In this post, we are going to keep things at a bit of a basic level since H.264 is extremely complex – especially once you start talking about different profiles, GOP length, etc. If you want to get into those fine details, we recommend reading the "Edit With" section of the Frame.io blog post How to Choose the Right Codec for Every Project or Adobe's Encoding options for H.264 video post. 

To understand why H.264 is so hard to edit, it is important to understand how it works. At its core, H.264 – also known as MPEG-4 AVC – is what is called an Inter-frame or Long GOP (group of pictures) codec. Unlike Intra-frame codecs like ProRes or DNx where each frame is self-contained, with a Long GOP codec (like H.264 or even H.265) the majority of the frames only record what has changed since the last frame.

In the image above, we are showing a simplified example of a red ball falling in front of a blue background. What would display on your screen if you were to watch this 5 frame video is exactly what you would expect. But if you could examine the actual frame-by-frame information in the encoded H.264 video file, what you would find is that the first frame looks normal, but the other frames would only include what has changed since that first frame. In this case, since the background is static, H.264 simply ignores it completely in order to reduce the final file size of the video. The end result is a video format that can pack a ton of information into a small file size, but requires additional processing power to decode.

The way H.264 does this is by separating your video into groups of frames with each frame becoming either an I-, P-, or B-frame.

• I-frames are used at the start of each group and is often called the "key frame". It is completely self-sufficient and acts as a reference frame for the rest of the group.
• P-frames contain only what has changed since either the last I-frame or the last P-frame (if there is one).
• B-frames are similar to P-frames except that they can either look back to the previous I/P-frame or they can look forward to the next I/P-frame.

These groups can range in size from just one frame long (referred to as "All-I" since every frame has to be an I-frame), to hundreds of frames in length. The longer the group, the smaller the file size, but the harder it is to decode.

In the example above, let's assume you have a GOP length of 9 frames. If you wanted to view frame 5 (which is a B-frame), the software first has to go to the closest P-frame, then follow a trail of P-frames (just one in this case) to get to the previous I-frame. At that point, it finally has all the information needed to actually turn frame 5 into a full image. Remember that this is very simplified example – these groups can be hundreds of frames long which results in dozens of jumps between B-,P-, and I-frames.

If you are watching an H.264 video in a normal video player, this isn't too big of a deal since the player knows that you are just going to watch the video at normal speed. Video editing, however, is much more complicated. Not only do you often jump around the video to random frames, but you can also play both backwards and forwards and at different speeds. Plus, it isn't good enough for the editing software to just display what has changed since the last I-frame, it has to completely render out each frame from scratch in order to properly apply effects like color correction.

Intra-frame codecs (such as ProRes, DNx, or even RAW) do not have this additional overhead since each frame is completely independent. The bitrate and file size may be much larger with ProRes versus H.264, but the overall load on your system is much, much less.

VFR – Variable Frame Rate

While not directly related to how H.264 works, many people also deal with especially poor performance due to what is called VFR or variable frame rate recording. This is a feature offered on some phones, drones, and other devices that allow it to dynamically reduce the frame rate of the recording if there is not much movement in the scene. This can drastically reduce the file size in some cases, but unfortunately, almost every editing software struggles with VFR.

The end result is a compounding of issues – your system will already struggle with H.264, then you are also throwing a variable frame rate at it. If this is an option on your recording device, we highly recommend disabling it. If you cannot do that, you should always transcode the footage so that it has a fixed frame rate before editing. Even if you just transcode to a fixed frame rate version of H.264, it should greatly improve editing performance. If you still have issues, then try one of the solutions listed in the next section.

How can you improve H.264 editing performance?

Since the issue with H.264 is due to how it works on a fundamental level, there is not an easy solution. Many video editing software has features for "Hardware accelerated H.264 decoding", but in our experience they are a mixed bag. The accelerated feature in DaVinci Resolve Studio seems to make at least a small difference, whereas a similar feature in Premiere Pro doesn't do much of anything from what we have seen.

In most cases, you only have a couple of options:

1) Get a more powerful computer

This is obviously a brute force solution that isn't always the right choice. If you are working on a laptop or low-end desktop, however, this may be a real option. Keep in mind that even the most powerful systems available today will struggle with higher bitrate H.264 media, so there is a limit to how effective this will be.

The best thing you can do is to ensure your system is able to utilize any hardware decoding that is available through the software. The latest versions of Premiere Pro (14.5+) and the Studio version of DaVinci Resolve.support hardware decoding via Intel Quick Sync or NVIDIA/AMD GPUs, which can result in massively better performance when working with H.264 media.

2) Dont record with H.264

Another obvious solution, but somewhat limited. Many cameras only offer H.264 as a recording option and if storage space is a concern you may not be able to afford jumping up to ProRes even if your camera has the ability to do so.

3) Transcode your footage to an intermediate codec before editing

This is similar to simply not recording with H.264, but at least removes the need to have a camera that supports ProRes or other Intra-frame codecs. While transcoding your footage to an intermediate codec like ProRes or DNx before editing should solve most performance issues (or at least greatly improve them), the downside is that you will still have to store the larger files that are the result of moving to a non-Long GOP codec. In addition, you have to wait for your files to transcode before you can begin editing.

There are a huge number of software packages that can be used for transcoding, but the two most popular tend to be Adobe Media Encoder and FFmpeg. If you decide to use Media Encoder, you might consider setting up a watch folder that will automatically transcode any footage you drop into that folder in order to streamline the process.

4) Adopt a proxy workflow

Proxies are not a new feature in most video editing applications, but in recent years they have become much, much easier to use. In a nutshell, proxies allow you to easily toggle between the original footage and a light-weight version that is optimized for editing. The advantage here is that you can use your original files when you need the best visual quality (color grading, masking, exporting, etc.), but switch over to the proxy when trimming or assembling your timeline.

In addition, with most editing applications these proxy files can be generated in the background and will auto-attach as they complete. The benefit here is that you can start your editing immediately, and as the proxies are automatically generated and attached you will simply notice a big improvement in playback and scrubbing performance. In fact, since you rarely need high quality video while trimming, you can reduce the resolution and quality of the proxies so that they are super quick to generate, minimizing the impact on your editing process.

This is becoming a common enough workflow that some high-end cameras even have the option to record lower-quality proxy files on the fly. Not to worry if you don't have that option, however, most editing software can either do it during ingest or you can manually tell it to generate and attach a proxy with the click of a button. While the process is different in every software application, here are a few guides to get you started:

• How to use Proxy Files to Edit SUPER FAST in Premiere Pro!
• Faster Resolve Editing Using Optimized Media – DaVinci Resolve 12.5 Tutorial (applies to newer versions as well).

Final words

Whether your system will struggle with H.264 media or not depends on a wide range of factors including what software you are using, how powerful your system is, and the bitrate and other settings used to create your H.264 media. You may never run into any issues, or your system may chunk along since you are trying to work with 4K 150mbps H.264 media. If you are having issues, however, hopefully this post has given you an idea of why H.264 is not considered an editing codec as well as giving you a number of solutions to help improve your editing experience.