= Debugging crashes =

If your bitlbee is crashing, we need the following information:

 * A backtrace, ideally with debug symbols
 * Bitlbee version ({{{/ctcp root version}}} from your irc client) 
 * Distro and distro version
 * How it was installed
    * If you built it yourself, running {{{bitlbee -V}}} in a shell shows the configure parameters in recent versions
 * List of plugins installed or protocols used, particularly third party ones.

== Debug symbols ==

This is optional but recommended: Debug symbols add extra information that helps a lot when understanding a backtrace.

If you're using the [[Packages|debian nightly apt repo]], you already have debug symbols.

If you're building bitlbee yourself, just add --debug=1 to ./configure:

 {{{
./configure --debug=1
}}}

For everyone else, your distro might have a way to install separate debug symbol files. Or you might have to rebuild bitlbee.

== Getting a backtrace ==

There are a few tools to get backtraces.

 1. gdb - The most generic method. The crash needs to be reproducible.
 2. valgrind - Better for some kinds of bugs. The crash needs to be reproducible.
 3. coredumpctl - Included only in linux distros that use systemd, collects previous crashes, but doesn't always keep them.

=== With gdb ===

 1. Kill all the currently running bitlbee processes
 2. If needed, switch to the correct user first with something like "sudo -u bitlbee -s"
 3. Start bitlbee with the following commandline

 {{{
BITLBEE_DEBUG=1 gdb -ex 'handle SIGPIPE nostop noprint pass' -ex run -ex bt --args bitlbee -Dnv
}}}

 This will run bitlbee until the first crash, ignoring SIGPIPE (which is harmless), and dumping a backtrace at the end.

 4. Connect with your irc client and make it crash
 5. Copy the output
 6. Pastebin it, ensure there are no passwords or other private information.
 7. Post to irc (preferred) or open a bug report (slower)

If possible, keep gdb open, since you may be asked to run a few extra commands for extra information.

Otherwise, if you happen to have a coredump file, you can open it in gdb like this (assuming it's called core.1234)

 {{{
gdb -ex bt bitlbee core.1234
}}}

=== With valgrind ===

Valgrind is a wonderful tool that can provide extra information about many bugs. It's slower but accurate, and sometimes even prevents crashes.

Repeat the same steps as above, but use the following commandline instead:

 {{{
valgrind --log-file=valgrind.log bitlbee -Dnv
}}}

And see if anything interesting appeared in valgrind.log, even if it didn't crash.

=== With coredumpctl ===

If you're using a distro with systemd, you might be able to get information from previous crashes with the {{{coredumpctl}}} tool. When this works, it can be very useful since you don't need to reproduce the crash again to get debugging information.

Note that using changing the "User" setting of {{{/etc/bitlbee/bitlbee.conf}}} will disable coredump creation for security reasons.

 1. Run coredumpctl to see if it caught anything. If bitlbee runs as a different user, you might have to run coredumpctl as root.

 {{{
$ sudo coredumpctl
TIME                            PID   UID   GID SIG PRESENT EXE
Sun 2016-03-20 03:30:57 UTC    3477  1001  1001   6   /usr/sbin/bitlbee
Sat 2016-05-07 17:52:16 UTC    3045  1001  1001  11 * /usr/sbin/bitlbee
}}}

 The last entry has a {{{*}}} in the PRESENT column, which means it's recent enough to keep the core dump around.

 2. Start a gdb session with that coredump.

 {{{
$ coredumpctl gdb
}}}

 3. Type "bt" to get the backtrace

 {{{
(gdb) bt
}}}

 4. Report that.


== Fun stuff to do inside gdb after a crash ==

All the irc connections are in `irc_connection_list`. It's a linked list, so if its `->next` is something other than null, you have more than one connection.

{{{
(gdb) p irc_connection_list
$27 = 0x5555567ca600 = {0x5555567c1030}
(gdb) p irc_connection_list->next
$28 = 0x0
(gdb) p irc_connection_list->data
$29 = (gpointer) 0x5555567c1030
}}}

Assuming the simpler case of only one irc connection you can get the object like this:

{{{
(gdb) set $irc = (irc_t *)irc_connection_list->data
}}}

Otherwise add one or more `->next` before `->data`

One interesting part is the sendbuffer (irc messages that haven't reached the irc client yet)

{{{
(gdb) p $irc->sendbuffer
$30 = 0x5555568fc040 ":root!root@localhost.localdomain PRIVMSG &bitlbee :mastodon - Error: Stream closed (200 OK)\r\n:root!root@localhost.localdomain PRIVMSG &bitlbee :mastodon - Signing off"...
}}}

If the process is still alive (not in a coredump) and the crash didn't mess things up too much, you can try saving the user config from here. Make a backup of your /var/lib/bitlbee, this is risky.

{{{
(gdb) p cmd_save($irc, 0)
$33 = void
}}}