Benchmarking

Benchmarking with Enemy Territory QUAKE Wars

1. Benchmarking

Enemy Territory QUAKE Wars (ETQW) provides several means to benchmark the system performance. However, anyone using ETQW for benchmarking should be careful in properly interpreting the results.

ETQW uses several threads of execution for instance for the game logic, texture streaming, texture transcoding, sound processing etc. However, by default the renderer is not running in a separate thread. When a system has multiple CPUs or a CPU with multiple cores the ETQW performance can be improved by enabling the multi-threaded renderer by setting the console variable r_useThreadedRenderer to 2. However, when a system does neither have multiple CPUs nor a CPU with multiple cores the performance may not improve if this multi-threading is enabled.

At any time one of the many components in a PC can become a bottleneck and limit the performance. As such correct interpretation of the results requires identifying the bottlenecks in the system.

At higher resolutions the ETQW performance quickly becomes limited by the graphics card. This means that graphics card cannot keep up with the speed of the CPU and memory. At lower resolutions and with high end graphics cards the ETQW performance is more likely to be dependent on the CPU speed, but also very much dependent on the memory speed. Increasing the memory speed can sometimes have close to the same effect on the performance as increasing the CPU speed. ETQW may not utilize all the available memory bandwidth but increasing the memory speed also lowers the memory access time which affects the performance.

2. Multiplayer Performance using 'timeNetDemo'

A network demo is a recording of the network traffic during a multiplayer game. A network demo can be recorded at the a client connected to a multiplayer server. Such network demos are referred to as client netdemos. A network demo can also be recorded at the server. These network demos are referred to as server netdemos.

When playing back a client netdemo, ETQW will behave exactly as if playing a live multiplayer game. Server netdemos record more information than client netdemos, and when playing back a server demo the view can be moved freely through the world or the view can be attached to any of the players that were playing on the server at the time of the recording.

ETQW allows playback of client netdemos as fast as possible where the engine continuously runs one game frame and one render frame. Playing back a network demo as fast as possible is referred to as a time net demo. Time Net demos can be used to rate the overall performance of a system configuration including CPU speed, memory speed and GPU speed.

The ETQW net time demos are meant to be used to rate the overall system performance. However, when using ETQW to benchmark the speed of graphics cards always make sure the system has the fastest CPU and the fastest memory available to make sure the CPU and memory subsystem have as little effect on the performance as possible.

When benchmarking the speed of CPUs always make sure the system has the fastest memory available to make sure the effect of the memory speed on the performance is as small as possible. Furthermore use a low resolution with the fastest graphics card solution available and disable features like anti-aliasing.

A net time demo can be started with the following console command:

timeNetDemo <demoName>

After playing the net demo the average framerate (FPS) is shown. Furthermore, the number of MegaTexture tiles that were processed is shown as well. While the player moves through an ETQW environment, terrain texture data is streamed in from the harddisk on a per tile basis. This texture data needs to be processed before it can be displayed. This process uses multiple threads such that a slow harddisk won't slow down the game. However, processing the texture data does take up CPU time. In other words the number of tiles that were processed during the demo playback may change the average FPS. To get consistent results, and/or when benchmarking only the graphics card performance set r_megaTilesPerSecond to 0. This way, parts of the terrain may become blurry during demo playback but the harddisk and CPU performance won't affect the measured average FPS.

3. Single Player Performance using 'timeServer'

Single player games with bots are typically more CPU intensive because the game does not just have to keep track of things that are happening in the player's view, but has to keep track of everything in the world. This may include things that happen behind the player or happen on the other side of a map. Furthermore, the game has to run the Artificial Intelligence (AI) for the bots which requires CPU time as well.

To benchmark the CPU and memory speed of a system it is best to measure the performance of a single player game. This can be done with the following console command:

timeServer <mapName> <number of game frames> <number of bots>

This is similar to running a time net demo except that this command runs a live game. It basically runs the game as fast as possible for the specified number of game frames with the specified number of bots. Each game frame it also renders one frame and it also runs one bot AI think frame.

With the bots running in a separate thread the game is not deterministic because the bots may query the collision state of the game world at arbitrary times. As such the 'timeServer' test is best run multiple times, where the median of the runs is taken as the performance result.