GDB, or how I learned to stop worrying and love the debugger

While compiled languages are typically faster, this comes at the cost of increased obfuscation. If you’re familiar with an interpreted language you are probably used to errors getting caught at runtime and being handled by the language interpreter. Since an interpreter completely contains the execution of the software it is able to do things like inspect stack frames and variable types. In addition there is typically some language support for complex error handling. While these things can be accopmlished in C it is typically less intuitive, and often requires the use of additional tools. GDB, the GNU Debugger is one such tool. From the GDB man page

“GDB can do four main kinds of things (plus other things in support of these) to help you catch bugs in
the act:
· Start your program, specifying anything that might affect its behavior.
· Make your program stop on specified conditions.
· Examine what has happened, when your program has stopped.
· Change things in your program, so you can experiment with correcting the effects of one bug and go on to learn about another.”

To quickly demonstrate some of GDBs ability we will need a program as contrived as it is broken. Fortunately Wikipedia provides.

#include <stdlib.h>;

void fault();

int main() {
    int i = 0;

void fault() {
    *i = 4;
    char *s = "segfaults";
    *s = 'H';

If you compile and run this program you should see something along the lines of “segmentation fault(core dumped)”. To use GDB we need two things, a binary with debugging symbols, and a working GDB install. All of the examples in this post use the program listed above, if you wanna try it out you can create a binary with debug symbols using the -g flag with clang or gcc e.g.

gcc -g -o program program.c

It is imperative to compile a version of your binary with -g on to get the most out of GDB. It is also worth noting that gcc was used to generate the binary used to make this and clang’s output will be minimally but noticeably different. Next, if you don’t have GDB installed you can download a copy of the latest release from the GNU repos, currently 7.10, and build it as follows

tar -Jxvf gdb-7.10.tar.xz && cd gdb-7.10
make && make install

With GDB installed one can start using it to debug programs. Begin by entering ‘gdb [program name]’ in a terminal. If everything is working you should recieve a prompt. If you enter ‘run’ the program will run and GDB will print something along the lines of

(gdb) run
Program received signal SIGSEGV, Segmentation fault.
0x00000000004004db in fault (i=0x7fffffffe95c) at toasted.c:11
11          *s = 'H';

GDB already provides some useful information. We’re able to see the line that causes the segfault, as well as the function it’s in and the arguments to that function. However, when code reuse is nontrivial, it isn’t always clear where a function is being called. If you want to inspect the full series of function calls that led to an error you can enter ‘backtrace’. backtrace will print the entirety of the current call stack along with the address, line number, and arguments of each function call.

(gdb) backtrace
#0  0x00000000004004db in fault (i=0x7fffffffe95c) at toasted.c:11
#1  0x00000000004004c4 in main () at toasted.c:6

GDB also supports breakpoints. Breakpoints are a tool for stopping execution within the debugging environment, making it possible to inspect and alter a programs state upon the attempted execution of an arbitrary section of code. This allows you to preempt faulty code and study the source of errors. To insert a breakpoint enter ‘breakpoint [LOCATION]’, where LOCATION can be a function name, a line number, or a memory address. Once you’ve set a breakpoint, when you run your program using ‘run’ within GDB, its execution will be halted when the program counter reaches the specified address. While breakpoints can be implemented in software, it is interesting to note that x86 has several registers which implement breakpoints at the hardware level. Breakpoints are numbered, and to remove a breakpoint simply enter ‘delete [NUMBER]’

(gdb) break fault
Breakpoint 1 at 0x4004e0: file toasted.c, line 11.
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y

Starting program: /root/warez/program 

Breakpoint 1, fault (i=0x7fffffffe95c) at toasted.c:11
11          *i = 4;

Above a breakpoint is created, breakpoint 1, and the program restarted, only to be paused at the breakpoint. From this point we can do a variety of interesting things. One useful command is ‘print’, which prints memory and register values. The syntax is ‘print [FORMAT] [VALUE]’. FORMAT is optional, and specifies the base VALUE will be printed in, for example /d is decimal, /t is binary, and /x is hex. VALUE is the register, variable, or location you wish to print. To print registers, prefix the register name with ‘$’, variables can be optionally annotated with a C style type cast

(gdb) print fault
$2 = {void (int *)} 0x4004d8 <fault>;
(gdb) print /x $rip
$3 = 0x4004e0
(gdb) print /x 0x4004d8
$4 = 0x4004d8(gdb)
print /d (int) *i
$5 = 0

We can see now that the program counter, stored in register rip, is 8 bytes higher than the address of our function. Since our function has a label at the beginning which is just the 64-bit address of that location it makes sense that we see the disparity we do. To print the value of pointers, C style dereferencing is supported. In addition to inspecting variables, GDB can also change their values. This can be done using ‘set [VARIABLE]=[VALUE]’. It is important that the type of value matches that of variable. When in doubt ‘whatis [VARIABLE]’ should provide some type information.

(gdb) whatis i
type = int *
(gdb) set *i=5
(gdb) print *i
$6 = 5

Having inspected and modified program state, it is time to resume execution. Two ways to do this are ‘step’ and ‘continue’. step, and the variant stepi move the program forward one line, and one instruction respectivly, stopping again after executing the appropriate number of instructions. You can step forward repeatedly to inspect how a program changes state line by line, or, if you’re not using a high level language like C, instruction by instruction. continue will continue executing your program normally, stopping only at the next breakpoint.

(gdb) step
12          char *s =this is the  "segfaults";
//next line to be executed
(gdb) print *i
$6 = 4

All of these things can be quite helpful, but we still haven’t used GDB to discover the cause of the segfault. This warrants a closer inspection of what the program is doing. To get this disassemble the function in question using ‘disas [FUNCTION]’

(gdb) disas fault
Dump of assembler code for function fault:
   0x00000000004004d8 <+0>:     push   %rbp
   0x00000000004004d9 <+1>:     mov    %rsp,%rbp
   0x00000000004004dc <+4>:     mov    %rdi,-0x18(%rbp)
   0x00000000004004e0 <+8>:     mov    -0x18(%rbp),%rax

   0x00000000004004e4 <+12>:    movl   $0x4,(%rax)
=> 0x00000000004004ea <+18>:    movq   $0x400584,-0x8(%rbp)
   0x00000000004004f2 <+26>:    mov    -0x8(%rbp),%rax
   0x00000000004004f6 <+30>:    movb   $0x48,(%rax)

   0x00000000004004f9 <+33>:    nop
   0x00000000004004fa <+34>:    pop    %rbp
   0x00000000004004fb <+35>:    retq   
End of assembler dump.

Since the program counter currently resides in this function, GDB is kind enough to annotate exactly which address the program counter points at.

And now for a brief digression about assembly. The broken out section constitutes the actual body of the C function. The other two sections are boilerplate the compiler adds to functions. The first instruction in the relevant block can be ignored as it deals with setting the value of i. The pointed instruction, which we know is the first instruction in the assignment ‘char* s = “segfault”;’, can be read ‘move 0x400584 as a 64-bit value to the address 8 bytes lower than the base of the current stack frame’.The offset is because the stack grows down in memory. Everything is still pretty reasonable, 0x400584 makes sense typewise as a value for our pointer s, which should be stored on the stack, it’s a static value because the compiler layed it out as binary data when it was compiled. You can confirm this by opening another terminal, leaving your precious GDB session untouched, and running

strings [program name] | grep segfaults

. strings is a tool to print the various ascii strings present in a binary file, and piping its output through grep should show the presence of the string “segfaults” in the binary itself. Running step executes the pointed instruction, and the one after that, moving the pointer’s address into a register for later use. The next instruction seems sane, moving 0x48 (ascii H) to the beginning of the string’s location in memory. The only instructions left are a nop and some stuff common to most functions, so this instruction must constitute the entirety of the offending line.

While the next instruction will trigger the segfault, the key lies in the address being written to, so it’s worth finding out a little more about it. It so happens that by looking at the section information the error reveals itself, and while printing out the section layout is rarely a thing I find myself doing in GDB it is certainly within the capabilities of the tool. The rather unclear syntax to get this information is ‘maint info section’, which when run gives you a summary of each of the individual sections in the binary.

(gdb) maint info section
Exec file:
    `/root/warez/program', file type elf64-x86-64.
 [0]     0x00400200->0x0040021c at 0x00000200: .interp ALLOC LOAD READONLY DATA HAS_CONTENTS
 [1]     0x0040021c->0x0040023c at 0x0000021c: .note.ABI-tag ALLOC LOAD READONLY DATA HAS_CONTENTS
 [2]     0x0040023c->0x00400260 at 0x0000023c: ALLOC LOAD READONLY DATA HAS_CONTENTS
 [3]     0x00400260->0x0040027c at 0x00000260: .gnu.hash ALLOC LOAD READONLY DATA HAS_CONTENTS
 [4]     0x00400280->0x004002c8 at 0x00000280: .dynsym ALLOC LOAD READONLY DATA HAS_CONTENTS
 [5]     0x004002c8->0x00400300 at 0x000002c8: .dynstr ALLOC LOAD READONLY DATA HAS_CONTENTS
 [6]     0x00400300->0x00400306 at 0x00000300: .gnu.version ALLOC LOAD READONLY DATA HAS_CONTENTS
 [7]     0x00400308->0x00400328 at 0x00000308: .gnu.version_r ALLOC LOAD READONLY DATA HAS_CONTENTS
 [8]     0x00400328->0x00400340 at 0x00000328: .rela.dyn ALLOC LOAD READONLY DATA HAS_CONTENTS
 [9]     0x00400340->0x00400370 at 0x00000340: .rela.plt ALLOC LOAD READONLY DATA HAS_CONTENTS
 [10]     0x00400370->0x0040038a at 0x00000370: .init ALLOC LOAD READONLY CODE HAS_CONTENTS
 [11]     0x00400390->0x004003c0 at 0x00000390: .plt ALLOC LOAD READONLY CODE HAS_CONTENTS
 [12]     0x004003c0->0x00400572 at 0x000003c0: .text ALLOC LOAD READONLY CODE HAS_CONTENTS
 [13]     0x00400574->0x0040057d at 0x00000574: .fini ALLOC LOAD READONLY CODE HAS_CONTENTS
 [14]     0x00400580->0x0040058e at 0x00000580: .rodata ALLOC LOAD READONLY DATA HAS_CONTENTS

Looking at the section data one can see that 0x400584 is in the read only data section. Writing to read only memory causes a segfault, but this is hard to determine as the cause without a bit of knowledge about how compilers lay out binaries or a trusty debugger.

One final command worth mentioning is ‘watch’, like break it will pause the execution of the program under certain conditions, but it does so when a piece of data changes. That data must exist, so it is sometimes necessary to use watch in conjunction with break. For example

(gdb) break main
Breakpoint 1 at 0x4004be: file toasted.c, line 6.
(gdb) run
Starting program: /root/warez/program 

Breakpoint 1, main () at toasted.c:6
6           int i = 0;
(gdb) watch i
Hardware watchpoint 2: i
(gdb) delete 1
(gdb) continue
Hardware watchpoint 2: i

Old value = 0
New value = 4
fault (i=0x7fffffffe95c) at toasted.c:12
12          char *s = "segfaults";
(gdb) bt
#0  fault (i=0x7fffffffe95c) at toasted.c:12
#1  0x00000000004004d1 in main () at .c:7

GDB has a load of additional functionality, and I highly reccomend checking out the official documentation in all of its Javascript free wonderment.


Catching Bugs in C with Lint

As you all have probably realized by now, C is not known for being a friendly language. In the best case, development is slow for even the simplest of tasks, and in the worst case… well… you’ve probably never been more tempted to through your laptop out the window. A lot of the frustrations with C come from the difficulty of fixing bugs in your code. The error messages can be cryptic unless you have a thorough understanding of the inner workings of your machine. Even then, the error messages may lead you down the wrong path of how to fix your code.

One tool that helps identify issues in your code in between making your change and compiling your code is lint. A linter is a static code checker which evaluates your code without compiling it. You can think of it as an extra set of eyes on your code – a set of eyes that doesn’t miss silly mistakes like memory leaks or uninitialized variables.

Linters exist for pretty much every language out there, and functionality between different tools range from language specific gotchas to styling. The original C lint program has had most of its functionality wrapped into the various C compilers over the years, but other tools have come and improved upon its legacy. One highly recommended option for C is cppcheck (works for both C and C++).

sudo apt-get install cppcheck

To check a file, simply run cppcheck on the file.

cppcheck myfile.c

As an example of the power of cppcheck, here’s a short program with a memory leak.

#include <stdlib.h>

void foo(int size) {
    int *array_pointer;
    array_pointer = (int *)malloc(size*sizeof(int));
    int i;
    for (i = 0; i < size; ++i)
        array_pointer[i] = i;

int main() {

What does cppcheck say about this program?

jgibson@ubuntu:~$ cppcheck debugging_exercise.c 
Checking debugging_exercise.c...
[debugging_exercise.c:13]: (error) Memory leak: array_pointer

Cppcheck tells you exactly where the memory leak is, making it quick and simple to fix.

Cppcheck is not perfect. For example, if you add a printf statement at the end of foo(), it tricks cppcheck into not finding the memory leak, even though it is still there.

#include <stdlib.h>
#include <stdio.h>

void foo(int size) {
    int *array_pointer;
    array_pointer = (int *)malloc(size*sizeof(int));
    int i;
    for (i = 0; i < size; ++i)
        array_pointer[i] = i;
    printf("Array at index 6 is %i!\n", array_pointer[6]);

int main() {
jgibson@ubuntu:~$ cppcheck debugging_exercise.c 
Checking debugging_exercise.c...

Cppcheck won’t catch all of the problems with your code, but it will help catch a large portion of them before you have to deal with C’s atrocious error messages.

For more info on cppcheck and how to use it, check out their documentation. You can look through their complete list of what they check for as well as some of the more advanced features such as ignoring certain warnings.