Well, the segfault is gone, but now the code just runs in a continuous loop, which is arguably worse. Run the invader program from GDB and press Ctrl-C to halt execution. View a backtrace by typing bt to see where we are in the code (screenshot below). Note that in a larger, multi-threaded program, when you Ctrl-C into a loop, it may stop at different places each time, so it is often worth trying it more than once and examining the backtrace of each.
As you can see, GDB has listed an impressive selection of function calls.
This is the call stack, and it shows you how your program got to its current point. The top line is where the program stopped when you interrupted it, and the last line is the original function call.
This call stack actually contains almost all system library calls, which are not that interesting, as it is unlikely that the bug is in anyone’s code but ours. To examine the frame you are interested in, select it using the framecommand:
(gdb) f 9
GDB prints out the line about to be executed, which is the printf call. A look at the local variables with i lo doesn’t show anything untoward (you can also display the local variables for every stack frame in a backtrace with the command bt full), so what’s going on here?
At least some output is appearing on the screen when the program is run, so it must be the outer forloop (the one that contains the printf statement), that is running repeatedly. It is set to run no more than 36 times, so you can assume that count is never getting high enough to exit the loop. Put a watch on count again to see if’s behaving as expected i.e. incrementing by 2 each time:
(gdb) b 31 (gdb) c (gdb) watch count (gdb) c
As soon as GDB stops you can see what the problem is – count is being changed from 2 back to 0. The line reported is the for loop, but this is yet to be executed, so what was the previous statement that just changed count? That’s right – the last statement in the loop resets count to zero. The programmer resets the buffer after printing to the screen and has been over-zealous with their tidying up.
Exit GDB, remove the count = 0; statement at line 47, recompile the program and hurrah! Everything works as expected.
Luckily this program only had a couple of errors in it, but the techniques that you’ve used to find them will enable you to track down all sorts of lovely bugs, from simple typos to subtle mistakes in pointer handling.
Debugging is very much like being a detective, and GDB is a great tool to aid you in confirming your suspicions about what might be going wrong.
Some other topics worth mentioning
1. Reversible debugging
One of the more exciting new features in GDB 7.0 onwards, is reversible debugging. This is still in its early stages of development, but you can see it at work as follows (type gdb –version to check if you have a recent enough version number). Start the invader program with GDB. Set a breakpoint at line 44. Type run and when the breakpoint is hit, type rec (or target record). Now use the ncommand twice and print out the buffer:
(gdb) p buffer $1 = ' ' <repeats 45 times> (gdb)
The buffer has been reset – oops! But we wanted to see what was in it – lets turn back time with rn(or reverse-next). The memset is ‘undone’, and now if we print out the buffer, we can see that it contains the characters that were printed to the screen. Amazing! Examples of other commands you can use to step backwards are rc (reverse-continue), and rs (reverse-step).
Reversible debugging is still in its infancy, and it does make your program run slower (because it is recording and storing so much extra information), but it has great potential and it is fun to see it at work.
2. Automate GDB startup
Each time GDB is run, it checks the local directory for a .gdbinit file and runs any commands it finds there. This is a really useful way to save configuration information that you don’t want to type in each time, such as any runtime arguments or GDB settings. For example, in Part 1 of the tutorial, you can see the following line is output by GDB in Figure 1:
Missing separate debuginfos, use: debuginfo-install glibc-2.11.1-6.i686
This is because more recent versions of GDB will warn you if you do not have the debug information installed for system libraries that are called by your application. For most of us, this is an unnecessary distraction. You can hide these reports by typing in
set build-id-verbose 0
when you start up GDB, or, you can add this line to your .gdbinit file. Just open any text editor, type in your commands (one per line) and save the file as .gdbinit. This file needs to be saved in the directory that you run GDB from, which is usually the one that contains your executable. Remember that files that start with a dot are hidden on Linux, so you won’t be able to see it in the graphical browser unless you enable viewing of hidden files, but running ls -a on the command line will display it. You can set up a separate .gdbnit file for each application you are debugging, as long as your applications don’t all live in the same folder!
3. Attach to a process
You don’t have to start your application with GDB – you can also use GDB to attach to a running program. In a new terminal window, look up the process id (pid) by running ps and note down the number. Then start GDB in a separate terminal window and type attach. GDB will halt the program’s execution and display the now familiar (gdb) prompt, allowing you to debug as if you had run the application directly. To let the program continue as normal, use the detach command to quietly slip away.
4. Core dumps
Traditionally, GDB has been used to examine core dump files. When a program crashes, the Linux kernel can create a “core dump” of data about what was happening in the program at that time. GDB can read this file to give you valuable information about a crash after the event. Many Linux distributions have this ability disabled by default – you need to type ulimit -c unlimited to allow the creation of the files. On Fedora you may also need to update the abrt package withyum update abrt, as it can interfere with core file creation. Once you’ve done that, when a crash occurs, a file called core.pid should be created in the same directory as the executable, where pid is the process id.
To examine a core file, just pass it to gdb with gdb core.pid (obviously, using your core file’s actual name). GDB will load up all the info and it will look as though you have just run the program and seen the error. You’ll be able to view a backtrace and a range of other information, but it will be a frozen “snapshot” of execution. The advantage of using a core dump is that you don’t have to wait around for the crash to happen, and users can send core files back to the programmer to illustrate a problem, but beyond that there is no substitute for debugging a program live if you are able to.
The end of the beginning
GDB has hundreds of configuration settings and many more commands, so your next stop is the GDB documentation. There is also a slightly out-of-date, but still useful, GDB quick ref PDF reference card floating around online, (just Google for GDB quick ref), which you can hide under some books on your desk, thus fooling people that you really have committed 500 slightly obscure commands to memory.
There should be nothing stopping you now, so enjoy using GDB and long may it aid you in your quest for wonderfully bug free code.
Appendix: GDB Tutorial Quick Ref
b <line num/function> breakpoint r <args> run/restart, with program arguments n next : step over next line s step : step into next line finish step out of function c continue l list next 10 lines of source l - list previous 10 lines of source i lo info locals – display all variables in the current stack frame i b info breakpoints – display all watchpoints and breakpoints watch <var> watchpoint d <num> delete breakpoint/watchpoint d delete all breakpoints/watchpoints attach <pid> attach to running process detach detach from running process p print bt backtrace bt full backtrace and print all local variables for each frame frame <num> select frame in stack rec target record (for reversible debugging) rs reverse step rc reverse continue rn reverse next