Star Citizen Benchmark: Arena Commander v0.8

Posted on by

Introduction

 
The long-awaited dogfighting module for Star Citizen, dubbed Arena Commander, has been released to backers of the game in its pre-beta form. It only has a few ships, a pair of maps, and a handful of game modes at this point… but is it the first part of the Star Citizen game to really be playable. It is also the first chance we have to fly ships, seeing how the physics handle and how they look in their natural environment. A lot of people have been waiting many months for this, and many will be looking at performance of this module as a gauge to see whether their computers can handle Star Citizen.
 
On the one hand this is a bit dangerous: the game is not optimized for performance at all yet, so as the developers improve things we could see a massive boost in frame rates. At the same time, the finished game will have the ability to encounter larger battles and more ships being on-screen at a time… so performance could also be worse. Since even the beta of the full game is likely a year or two away, though, the debut of Arena Commander is at least a good chance to gather some rough data on how the game performs. That is what this article will aim to do, but instead of testing dozens of varying combinations I want to approach this from the standpoint of questions that supporters of the game have expressed. Here are some of the questions I wanted to cover:
 
– How do some of AMD and NVIDIA's top-end video cards perform compared to eachother? (dual Titans vs the R9 295X)
 
– How well does the game scale going from one to two video cards? (testing both SLI and Crossfire)
 
– Does the amount of video memory on a graphics card impact game performance?
 
– How does framerate scale with core count (6 / 4 / 2 cores), and does Hyperthreading have any impact?
 
– What resolution and quality settings are best for an average gaming computer as of now?
 
Unfortunately, current game conditions have prevented me from gathering some of this data, but even in those cases what we found may still be of interest to some folks. Each of the topics above will have a page devoted to it, whether to show benchmark results or to explain issues currently found in Arena Commander. If you want to skip ahead to a specific area of interest to you, please use the navigation menu below.

Benchmark System Specs

It is always a good idea to list off what hardware is used when testing, so here is the testbed we used for most of this article:
 
Asus P9X79 Deluxe
Intel Core i7 4960X (six core @ 3.6GHz)
Kingston HyperX 16GB (4x4GB) 1600MHz CL9 memory
Samsung 840 Pro 128GB SSD
Corsair H60 cpu cooler
Seasonic SS-1200XP3 1200W power supply
Fractal Design Arc Midi chassis
 
The default video card used was a NVIDIA GeForce GTX Titan 6GB, but in some tests a pair of those or an AMD Radeon R9 295X 8GB was used; such instances are noted on the graphs.
 
The testing on the last page of the article was done on my home computer instead, to get a better feel for what an average gaming rig might be like. It has the following specs:
 
Asus Gryphon Z87
Intel Core i5 4670 (quad core @ 3.4GHz)
Asus GeForce GTX 670 2GB
Mushkin Silverline 32GB (4x8GB) 1333MHz CL9 memory
Intel 520 series 240GB SSD
Silverstone NT06e cpu cooler
Seasonic G 550W power supply
Silverstone SG01 chassis

Benchmark Procedure

The tests on the following pages were all conducted in the same manner. Resolution and quality settings were selected within Star Citizen, and no special settings were configured in the video card drivers. Unless otherwise noted, tests were done at Very High quality and 1920×1080.

Each test was performed in the Hornet F7C fighter, on the Broken Moon map, and in Vanduul Swarm mode. I did not yet have access to the multi-player module, and wanted the most demanding option possible. The Swarm mode puts an added load on the system compared to just using the Free Flight mode: there are additional ships and weapon effects for the video card to deal with, as well as the AI for the other pilots (both friendly and hostile) to push the CPU harder.

I had FRAPS running in the background to measure frame rates, and ran its benchmark mode for 300 seconds (5 minutes) starting when the "Game Begins" countdown reached 0. During this time I would engage in combat, and while it is impossible to exactly repeat the same movements during each run I was at least trying to keep the style of play consistent. Here is a video showing what the benchmark segment looks like, along with a couple minutes of the gameplay prior to getting to that point:

I should note that the minimum frame rate numbers on the following pages are somewhat tricky, and I thought about leaving them out entirely. This is because those dips all occurred in non-critical situations. The noticable pauses in gameplay (resulting from momentary frame rate dips) were either when loading a new wave of enemy ships or when crashing… which was usually fatal. The average frame rates definitely convey how playable and smooth the game was in each hardware configuration far more than the minimums, so please keep that in mind as you look over the results.

AMD vs NVIDIA

One thing people almost always ask about when gaming computers come up in discussion is “AMD or NVIDIA?”. These are the two companies that have been making graphics processors for decades now (AMD having bought ATI) and there are fans on both sides of the fence, as well as users who just prefer whatever works best or is the most cost effective at the time.
 
Here are results from some of the top-end cards that each company has to offer currently. We’ve got the GeForce GTX Titan 6GB from NVIDIA, which is technically not quite as fast as the newer GTX 780 Ti and Titan Black models… but is still very, very fast. Facing off with that we have the AMD Radeon R9 295X, which has two GPUs on a single card. Each has 4GB of memory and is about equal to the R9 290X single-GPU cards. Since it is up against a single GPU card from NVIDIA, we turned off the second GPU in this initial test. Here are the results:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
At 1920×1080 the NVIDIA GTX Titan clearly wins on average frame rate, by a pretty wide margin. Minimum frame rates are pretty close between the two cards, though, so at least the AMD graphics chip isn't dropping too low at any point. I should note that AMD's loss here does not appear to be related to video memory. I checked GPUz, and it was showing about 3GB of the Radeon's memory in use during this test.
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
At 4K both cards struggle, and while the game was still surprisingly more fluid than I would have expected for such low results I could definitely feel the difference, and it made combat difficult compared to the 30fps+ average at the lower resolution. I will say, though, that this game looks *amazing* at 4K. Hopefully as it is optimized during development and as newer video cards come out we will see single-GPU solutions capable of playing it at higher frame rates.

Multi-GPU Scaling

When the performance of a single GPU isn’t enough, adding a second one can sometimes help. NVIDIA and AMD both offer this sort of option, though they use different names: SLI for NVIDIA, and Crossfire for AMD. Both companies also offer solutions with two GPUs on a single physical card, which is effectively the same as having two cards but can save on space. The R9 295X mentioned on the previous page is an example of that.
 
However, a second card usually doesn’t mean a full doubling of performance… and if a game is limited by CPU performance instead then it may not improve things at all. To see how NVIDIA and AMD cards scale in this initial release of Arena Commander, we tested a pair of GeForce GTX Titans against a single Titan (using the results from the previous page), and also the R9 295X in its normal Crossfire mode versus the same card with the second GPU turned off (again, as on the previous page). Here is what we found:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
As with the single GPU results on the page before, the Titan wins out in performance at 1080P. However, I observed some light graphical corruption at this resolution. On the AMD card, it would show up as occassional flashes of odd colors and shapes – fleeting, but noticable. On the NVIDIA card it was more like every other frame was being rendered with a slightly different color pallet, more pinkish than the colors should have been. Both were definitely distracting, but it was playable. This seemed quite odd until I ran the 4K resolution testing.
 
When the resolution was moved up to 3840×2160, the graphical corruption increased dramatically. Both cards were now showing weird colors and strange geometric shapes all over the place, almost completely obscuring the screen. It was absolutely unplayable, though underneath it all you could tell the game was running. I thought about trying to get results anyway, but I couldn't bring myself to play for five minutes to give the same length of testing that I had on other settings.

From other reports on the Star Citizen forums, it appears that most people with dual GPU – either SLI or Crossfire – are seeing similar corruption issues. As such I don't think that it is a hardware problem so much as either drivers that need optimizing for this game or game code that needs to be changed a bit to work properly with dual GPUs. This is one of the pitfalls of benchmarking during the alpha stage of game production, but hopefully the findings help the game developers fix this well before the Star Citizen's full release.

GPU Memory

The more complex a game is, and the higher the resolution it is being played at, the more memory a video card needs in order to keep track of all the data involved. More specifically, the memory is per GPU (graphics processing unit), not card. For example, the Radeon R9 295X is a single video card with two GPUs. While it is advertised as having 8GB of memory, that is more accurately put as 4GB of memory for each GPU; it is effectively the same thing as having two separate cards, each with a single GPU and that much video memory.
 
With Star Citizen being a very graphically complex game – ship models with polygon counts in the hundreds of thousands, and the potential for multiple ships on-screen at once – the amount of data needed to display it could be a big factor for performance. To find out, we have pitted a GeForce GTX 780 Ti 3GB card against a GeForce GTX Titan 6GB. The 780 Ti is technically a slightly faster card, so if it outperforms the Titan then 3GB of memory is sufficient… but if the Titan wins, then 3GB is not sufficient. Many cards have even less than 3GB of memory, of course, but lets start here and see how it goes.

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
At 1920×1080, it looks like the cards are neck and neck for average frame rates… but the 780 Ti never dips as low on the minimum FPS. That may be an anomoly – as I have mentioned previously, the minimum frame rates in this testing are not incredibly reliable – but it could also mean that the slightly faster GPU on that card keeps it from ever dropping quite as low. Since the 780 Ti is also cheaper than the Titan, though, this is definitely the card to get if you want great performance in Arena Commander and will be playing at this resolution.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
As expected, scaling the resolution up to 4K reduces performance on both cards… but the GTX 780 Ti manages to take the lead in both average and mininum frame rates, despite only having half the amount of video memory. Looking at GPUz during the testing showed that virtually all of the 780 Ti's memory was being used, so I had expected it to take a performance hit from that… but it comes out ahead, showing that even at higher resolutions the 780 Ti may be the better choice for gaming. Of course neither individual GPU was able to maintain smooth gameplay: it looked amazing at 4K, but actually playing at these lower frame rates was definitely frustrating.

CPU Core Scaling

Some of the Star Citizen developers have stated that the game is largely CPU limited at this point. To test that theory, we ran through benchmarks with various settings on the Core i7 processor in our test rig. It is a six-core CPU, so we ran it with all cores on, then four, and then just two. That should simulate other Core i7 and even Core i3 processors. We also ran tests at those different core counts with Hyperthreading turned off, to see if that Intel feature had any impact on performance. Here is what we found:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
I should start by explaining the 2-core score. With just two cores enabled and Hyperthreading turned off, the game would consistenly freeze part-way into loading the Star Citizen hangar (where you select a ship to play Arena Commander with). Sometimes the game would pause for a few seconds at that point with any set of hardware, as graphics and such were loading into view, but here it would always hang and then crash as soon as I tried to alt-tab out to see what was going on. This makes me wonder if some part of the game is just expecting to see more than two cores as being available.

With Hyperthreading enabled, the 2-core configuration would show up to Windows as actually being 4 cores… and at that point the game was playable, though with noticeably lower performance. Once I reached 4 cores, either with or without Hyperthreading enabled, performance pretty much leveled out. You see some differences above, but they are all reasonably close together. The only thing I might be able to pull out of this is that average frame rates were higher with Hyperthreading disabled, both at 4- and 6-core settings. I have seen similar results in some other games as well, and this is one of the reasons I often recommend a Core i5 (quad-core without Hyperthreading) for gamers, rather than the Core i7 (which adds Hyperthreading).

Recommended Settings for Playing Arena Commander Today

All of the previous testing was done on a high-end testbed setup at our offices, which would run $4,000 to $5,500 depending on the video card setup. Most of us gamers don’t have that much to spend – at least I know I don’t! – so I wanted to see what a more ‘average’ gaming system today can handle when it comes to Star Citizen. My home gaming computer should be a great example of such a configuration: a Core i5 quad-core processor at 3.4GHz with a GeForce GTX 670 2GB video card. I also have a 2560×1440 resolution monitor, allowing me to test for the best balance of performance and quality at that resolution as well as the more common 1920×1080 and even 1280 x 720 to see if lowering resolution a lot improves performance. Needless to say, most gaming rigs out there (mine included) would be brought to their knees if you try to play at 4K or across multiple monitors; for those inclined to do so, the previous pages will be much more applicable.
 
With that in mind, here are the results I found with Medium, High, and Very High quality settings at each resolution:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
There was very little difference in frame rate at this resolution, leading me to think that my Core i5 is probably CPU limited at this setting. I did notice that at this low of a resolution some parts of the HUD (the interface for seeing ship info) were not as legible as they were at higher settings. I imagine few people have monitors at this low of a resolution, outside of perhaps lower-end laptops, so it is probably not anything to worry about.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Once I moved up to 1080P there was a noticable drop in the minimum frame rates between quality settings, though the averages were basically all the same still. This is likely in part due to CPU limitations again, and as I mentioned in the benchmark procedure section the minimum frame rates are somewhat decieving. Those low dips were only reached at non-critical times in the gameplay – like when a new wave of enemies spawned, or when a fatal collision occurred. As such, I would consider even Very High to be just fine at this resolution.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
At 2560×1440 there is finally some difference in the average frame rates, with the Very High setting dropping below my personal threshold of 30fps. Since this is the native resolution of my monitor, I will liklely run the game at this resolution and High settings, since the frame rate at that level was definitely better and the graphics still look quite good. This is the first game I've played where my GTX 670 struggles at max quality, though, so I will probably be eying a video card upgrade when the GeForce GTX 800-series comes out.

Conclusion

Hopefully the information on the previous pages is helpful and informative! Remember that Star Citizen is in development, and even this Arena Commander dogfighting module is only in early alpha stages. Performance will change over time. However, folks looking to get a new system before the finished game is released, or to upgrade an existing computer, could still use the data here to make an informed decision.

For anyone wanting to purchase a system, rather than build one themselves, we here at Puget Systems would be honored to help design and build such a computer – just call or email us! Oh, and I'll "see you in the 'Verse!"

Tags: , , , , , , , , ,