View A Backtrace For All Threads With GDB

Debugging multi-threaded programs can be really tricky. GDB however, will always do its best to rescue you from whatever horrible bug you’re currently looking at. Today I wanted to show you a really nice command for viewing all the threads in your program.

This particular command is especially helpful if you ever need to diagnose a deadlock, because it will give you a complete overview of the entire system and allow you to see which locks are currently being requested.

I’m going to use the C++11 threads program from my thread tutorial to illustrate, with one minor modification – I’ve added a sleep command in the thread subroutine. The only reason I’ve done this is because it keeps my threads alive for long enough to show them all to you in the debugger. On a commercial multi-threaded system, your threads will be doing something most of the time, so my sleep command is essentially simulating my threads going off and doing important jobs ūüėČ

Okay, here’s the code:

#include <iostream>
#include <string>
#include <thread>
#include <mutex>
#include <unistd.h>
#include <vector>

using namespace std;

#define NUM_THREADS 5

#define BLACK   "\033[0m"
#define RED     "\033[1;31m"
#define GREEN   "\033[1;32m"
#define YELLOW  "\033[1;33m"
#define BLUE    "\033[1;34m"
#define CYAN    "\033[1;36m"

std::mutex colour_mutex;

void PrintAsciiText(int id)
{
    string colour;

    colour_mutex.lock();
    switch((long)id)
    {
    case 0:
        colour = RED;
        break;
    case 1:
        colour = GREEN;
        break;
    case 2:
        colour = YELLOW;
        break;
    case 3:
        colour = BLUE;
        break;
    case 4:
        colour = CYAN;
        break;
    default:
        colour = BLACK;
        break;
    }

    sleep(1);
    cout << colour << "I'm a new thread, I'm number " << (long)id << BLACK << endl;
    colour_mutex.unlock();
}

int main()
{
    vector threads;

    for(int i = 0; i < NUM_THREADS; ++i)
    {
       threads.push_back(std::thread(PrintAsciiText, i));
    }

    for (vector::iterator it = threads.begin() ; it != threads.end() ; ++it)
    {
        it->join();
    }

    return 0;
}

And in case you need a refresher, here’s the output:

threadsc11_output

Now what we need to do is run this under the debugger. Make sure you’ve compiled with the -g option:

g++ -Wall -g -pthread -std=c++11 threadsc11.cpp -o threadsc11

And then run it under GDB (type this command in the same directory as your executable):

gdb threadsc11

Now, let’s add a breakpoint in our thread subroutine at line 49:

(gdb) b 49

And then run the program:

(gdb) r

Log GDB Output To A File

Now, even from my fairly short piece of code, full thread output will be more than a screenful. On a program running dozens of threads you’ll get a huge amount of information. The best thing to do is save this output to a file, so you can examine it more clearly. To do this, we need to enable logging (just like we saw last week):

(gdb) set logging on
Copying output to gdb.txt

And now, we can use the amazing gdb command:

(gdb) thread apply all bt

What this does is say:

Hey gdb, you know those threads you’ve got running? Well, there’s something I want you to apply to all of them right now, and that is a backtrace. Go!

And then we get a whole lot of output in our terminal window.

You can turn logging off at this point if you like:

(gdb) set logging off
Done logging to gdb.txt

And now, the exciting part. Let’s take a look at the file.

What I’ve done here is make a screenshot of the entire file so I can talk you through it. I’ve also highlighted parts of the file – the original was just in black and white:

gdb-thread-apply-all-bt

So, what can learn from this?

1. It’s handy to know that threads are always listed in order of creation, so thread 1 was created first (during execution), and thread 6 was created last.

2. You can see that each thread, highlighted at the top in orange, is listed separately with a backtrace underneath it. The threads are numbered from 1-6, which is gdb’s numbering.

3. The thread id that appears on the console, as specified by our program and printed to the output, is passed to the subroutine as an argument, and you can see this on the yellow highlighted lines, in brackets, after the function name (PrintAsciiText). GDB always displays the parameters passed to a call listed in a stack trace.

4. We actually have four different numbering systems for our threads here: If we look at gdb thread 2, it has a pthread id of 0x7ffff6fd3700, and a Linux system id (Light Weight Process) of 32691, as well as our own output id (which we created in the main function) of 0.

4. I’ve highlighted in yellow¬†the current function that each thread is in,¬†with regards to our own program. What I mean by this is, I’ve ignored standard library calls. You can see that five of the threads are in the PrintAsciiText function, and the first thread (at the bottom, highlighted in purple), is in the main function.

5. I’ve highlighted in blue the ‘wait’ for the mutex. Four threads are currently suspended and waiting – and the wait is at the top of each of their stacks. The fifth thread (gdb thread 2) is the one that has triggered the breakpoint at line 49. He already has the lock – note that you can’t actually tell this from his stack trace. In fact, creating separate function calls to lock and unlock mutexes will pay dividends at debug time, because it will allow you to see when a lock has been taken without having to trawl through the code each time.

6. I’ve highlighted the mutex that each thread is waiting for in green. Since it is passed as a parameter to the std:mutex::lock call, gdb tells you the name of it in brackets after listing the call.

7. You can confirm, with the multiple references to calls in the libpthread.so.0 library, that C++11 threads are in fact, just a wrapper around pthreads when compiled with gcc on Linux.

So there you go. Awesome, all-in-one debug information for all of your threads, and a breakdown of what your stack trace is telling you.