• DebuggingCrashes

Debugging crashes

If your bitlbee is crashing, we need the following information:

• A backtrace, ideally with debug symbols

• Bitlbee version (/ctcp root version from your irc client)

• Distro and distro version
• How it was installed

  • If you built it yourself, running bitlbee -V in a shell shows the configure parameters in recent versions

• List of plugins installed or protocols used, particularly third party ones.

Debug symbols

This is optional but recommended: Debug symbols add extra information that helps a lot when understanding a backtrace.

If you're using the debian nightly apt repo, you already have debug symbols.

If you're building bitlbee yourself, just add --debug=1 to ./configure:

• ./configure --debug=1

For everyone else, your distro might have a way to install separate debug symbol files. Or you might have to rebuild bitlbee.

Getting a backtrace

There are a few tools to get backtraces.

1. gdb - The most generic method. The crash needs to be reproducible.
2. valgrind - Better for some kinds of bugs. The crash needs to be reproducible.
3. coredumpctl - Included only in linux distros that use systemd, collects previous crashes, but doesn't always keep them.

With gdb

1. Kill all the currently running bitlbee processes
2. If needed, switch to the correct user first with something like "sudo -u bitlbee -s"
3. Start bitlbee with the following commandline

   BITLBEE_DEBUG=1 gdb -ex 'handle SIGPIPE nostop noprint pass' -ex run -ex bt --args bitlbee -Dnv

   This will run bitlbee until the first crash, ignoring SIGPIPE (which is harmless), and dumping a backtrace at the end.
4. Connect with your irc client and make it crash
5. Copy the output
6. Pastebin it, ensure there are no passwords or other private information.
7. Post to irc (preferred) or open a bug report (slower)

If possible, keep gdb open, since you may be asked to run a few extra commands for extra information.

Otherwise, if you happen to have a coredump file, you can open it in gdb like this (assuming it's called core.1234)

• gdb -ex bt bitlbee core.1234

With valgrind

Valgrind is a wonderful tool that can provide extra information about many bugs. It's slower but accurate, and sometimes even prevents crashes.

Repeat the same steps as above, but use the following commandline instead:

• valgrind --log-file=valgrind.log bitlbee -Dnv

And see if anything interesting appeared in valgrind.log, even if it didn't crash.

With coredumpctl

If you're using a distro with systemd, you might be able to get information from previous crashes with the coredumpctl tool. When this works, it can be very useful since you don't need to reproduce the crash again to get debugging information.

Note that using changing the "User" setting of /etc/bitlbee/bitlbee.conf will disable coredump creation for security reasons.

1. Run coredumpctl to see if it caught anything. If bitlbee runs as a different user, you might have to run coredumpctl as root.

   $ sudo coredumpctl
   TIME                            PID   UID   GID SIG PRESENT EXE
   Sun 2016-03-20 03:30:57 UTC    3477  1001  1001   6   /usr/sbin/bitlbee
   Sat 2016-05-07 17:52:16 UTC    3045  1001  1001  11 * /usr/sbin/bitlbee

   The last entry has a * in the PRESENT column, which means it's recent enough to keep the core dump around.

2. Start a gdb session with that coredump.

   $ coredumpctl gdb

3. Type "bt" to get the backtrace

   (gdb) bt

4. Report that.

Fun stuff to do inside gdb after a crash

All the irc connections are in irc_connection_list. It's a linked list, so if its ->next is something other than null, you have more than one connection.

(gdb) p irc_connection_list
$27 = 0x5555567ca600 = {0x5555567c1030}
(gdb) p irc_connection_list->next
$28 = 0x0
(gdb) p irc_connection_list->data
$29 = (gpointer) 0x5555567c1030

Assuming the simpler case of only one irc connection you can get the object like this:

(gdb) set $irc = (irc_t *)irc_connection_list->data

Otherwise add one or more ->next before ->data

One interesting part is the sendbuffer (irc messages that haven't reached the irc client yet)

(gdb) p $irc->sendbuffer
$30 = 0x5555568fc040 ":root!root@localhost.localdomain PRIVMSG &bitlbee :mastodon - Error: Stream closed (200 OK)\r\n:root!root@localhost.localdomain PRIVMSG &bitlbee :mastodon - Signing off"...

If the process is still alive (not in a coredump) and the crash didn't mess things up too much, you can try saving the user config from here. Make a backup of your /var/lib/bitlbee, this is risky.

(gdb) p cmd_save($irc, 0)
$33 = void