Quake 2 Engine Robustness Testbed

A collection of 46 ‘hand-made’ BSP files is provided. The first BSP file (testbox00.bsp) is a tiny, but otherwise complete and valid Quake 2 map. Every engine should load it correctly. Remaining 45 files are variations of the same map corrupted in different ways to evaluate the robustness of various routines responsible for parsing, detecting collisions and rendering BSP files.

All testing is performed on top of default Quake 2 installation with 3.20 patch applied. Original Quake 2 client is tested with its default OpenGL renderer (‘ref_gl’). R1Q2 client is tested with R1GL renderer (‘ref_r1gl’). AprQ2 and Q2PRO are tested with their built-in OpenGL renderers.

Both client and server parts of code are tested. Testing method for each testcase number NN > 0 is as follows. Engine is started in windowed mode with default configuration. Listen server is created on a known valid map (for example, ‘testbox00’). After local client enters the map, ‘gamemap testboxNN’ command is executed. If engine didn’t crash upon loading the map, player moves around the map for a while and shoots. If engine still didn’t crash, it is exited by typing ‘quit’ command.

Two-stage map loading process is needed to check if server handles transitions to malformed maps robustly: it should validate the map before ‘gamemap’ command completes. Dropping all clients at map change with error message is unacceptable for multiplayer servers. Server should print a warning message and stay on the previous map instead, like it does for maps that are merely missing.

Client, of course, is permitted to drop the connection if it encounters a malformed map, since it can’t enter the server without a map!

Total number of various responses observed per engine is provided in the table below. There, unstable responses include undefined behaviour (crashes, hangs) and general instability (graphics corruption, infinite reconnect loops). Error responses include both hard and soft errors that cause server to be killed.

Clearly, original Quake 2 client is the most vulnerable one being unstable in 66% of cases. However, more recent engines like R1Q2 and AprQ2 are hardly any more stable. AprQ2 includes a few more validity checks compared to R1Q2. Q2PRO is the most stable engine, and the only one capable of robustly detecting malformed BSP files without crashing the server.

It is out of scope of this document to evaluate the actual exploitability and severeness of discovered flaws. Surely, undefined behaviour signifies at least denial of service vulnerability. In some cases, it may signify arbitrary code execution vulnerability. For example, testcase #06 triggers a reliable stack-based buffer overflow in vulnerable engines.

Developers of Quake 2 engine forks are welcome to use the provided collection of testcases to check, debug and fix their engines. Testcases for other file formats supported by Quake 2 may be added later.

Until discovered flaws are fixed, it is recommended that users of vulnerable clients double check what server they are connecting to and avoid connecting to unknown/untrusted servers. Blindly copying and pasting server addresses from IRC and using commands like ‘followip’ is particularly dangerous. It might be desirable to recursively make the Quake 2 directory read-only to prevent the server from uploading arbitrary files (remember, setting ‘allow_download’ console variable to 0 is not enough).

Server operators must ensure that they obtain game data from trusted sources. In particular, allowing players to upload arbitrary maps or other files to the server (for example, via anonymous FTP) is unacceptable.