Separate debug symbols, just like Windows

Currently we're working heavily on Linux. You might ask why, but for now you'll have to wait to find out.

Having become more familiar with heavy debugging under Linux we'd like to share with you a little tip about being able to ship binaries in a title that are still useful for debugging problems that are discovered out there in the wild.

This is achievable under Linux by shipping debug binaries that have the debugging symbols separated from the binaries.

Being able to do this under Windows is well known, in fact it's the default. Under Linux it's equally possible by using the less well know gcc debug-link functionality.

This functionality is particularly useful when a distributed application dumps core on a user. One can get the core file, use the separate debugging information and see exactly where the application crashed. All you need to do when you make a build is put aside the separate debug files.

Generally you don't want to distribute the debug symbols, for most people it's just a waste of space, and on the other hand it makes it easier for nefarious types to reverse engineer you code, or otherwise manipulate your software.

This is potentially handy to many others, game developers or otherwise who are working under Linux.

The How

Separating debug symbols from the main binary is achieved with using objcopy which is part of the bintools package found on many Linux systems.

We're particularly interested in the command line arguments --only-keep-debug/--add-gnu-debuglink

What do these command line flags do?

--add-gnu-debuglink adds a .gnu_debuglink section to the binary. In that section is stored the name of debug file to look for.

Below is a short shell transcript of how this is achieved:

$ gcc -g -shared -o libtest.so libtest.c
$ objcopy --only-keep-debug libtest.so libtest.dbg
$ objcopy --add-gnu-debuglink=libtest.dbg libtest.so
$ objdump -s -j .gnu_debuglink libtest.so
 
libtest.so:     file format elf32-i386
 
Contents of section .gnu_debuglink:
 0000 6c696274 6573742e 64656275 67000000  libtest.debug...
 0010 52a7fd0a                             R... 

The first part is the name of the file, the second part is a check-sum of debug-info file for later reference.

Build ID

Did you know that binaries also get stamped with a unique id when they are built? The ld --build-id flag stamps in a hash near the end of the link.

$ readelf --wide --sections ./libtest.so  | grep build
  [ 1] .note.gnu.build-id NOTE            000000d4 0000d4 000024 00   A  0   0  4
$ objdump -s -j .note.gnu.build-id libtest.so 
 
libtest.so:     file format elf32-i386
 
Contents of section .note.gnu.build-id:
 00d4 04000000 14000000 03000000 474e5500  ............GNU.
 00e4 a07ab0e4 7cd54f60 0f5cf66b 5799b05c  .z..|.O`.\.kW..\
 00f4 2d43f456                             -C.V            

Although the actual file may change (due to prelink or similar) the hash will not be updated and remain constant.

Finding the debug info files

The last piece of the puzzle is how gdb attempts to find the debug-info files when it is run. The main variable influencing this is the command debug-file-directory.

After starting gdb, one can ...

(gdb) show debug-file-directory 
The directory where separate debug symbols are searched for is "/usr/lib/debug".

The first thing gdb does, which you can verify via an strace, is
search for a file called [debug-file-directory]/.build-id/xx/yyyyyy.dbg; where xx is the first two hexadecimal digits of the hash, and yyy the rest of it:

$ objdump -s -j .note.gnu.build-id /bin/ls
 
/bin/ls:     file format elf32-i386
 
Contents of section .note.gnu.build-id:
 8048168 04000000 14000000 03000000 474e5500  ............GNU.
 8048178 c6fd8024 2a11673c 7c6a5af6 2c65b1b5  ...$*.g<|jZ.,e..
 8048188 d7e13fd4                             ..?.            
 
... [running gdb /bin/ls] ...
 
access("/usr/lib/debug/.build-id/c6/fd80242a11673c7c6a5af62c65b1b5d7e13fd4.debug", F_OK) = -1 ENOENT (No such file or directory)

Next it moves onto the debug-link info filename. First it looks for the filename in same directory as the object being debugged. After that it looks for the file in a sub-directory called .debug/ in the same directory.

Finally, it prepends the debug-file-directory to the path of the object being inspected and looks for the debug info there. This is why the /usr/lib/debug directory looks like the root of a file-system; if you're looking for the debug-info of /usr/lib/libfoo.so it will be looked for in /usr/lib/debug/usr/lib/libfoo.so.

Interestingly, the sysroot and solib-search-path don't appear to have anything to do with these lookups. So if you change the sysroot, you also need to change the debug-file-directory to match.

Remember to keep the debug files for every build that gets distributed and you can load up the binary, core file and debug file all together and see exactly what happened.