Differences between revisions 5 and 6
Revision 5 as of 2016-05-07 21:04:13
Size: 882
Editor: dx
Comment:
Revision 6 as of 2016-05-07 22:32:41
Size: 3725
Editor: dx
Comment: rewrite the whole damn page. now *this* is something i can link to people
Deletions are marked like this. Additions are marked like this.
Line 2: Line 2:
How to get backtrace (on Linux)?
Line 4: Line 3:
The backtrace is useful when the application crashes or freezes. You might use the following steps: == Debugging crashes ==
Line 6: Line 5:
1. Rebuild BitlBee with --debug=1 (if you're using the Debian packages from http://code.bitlbee.org/debian/ you don't need to do this) If your bitlbee is crashing, we need the following information:
Line 8: Line 7:
2. Start the debugger with the binary and log the output:  * A backtrace, ideally with debug symbols
 * Bitlbee version ({{{/ctcp root version}}} from your irc client)
 * Distro and distro version
 * How it was installed
    * If you built it yourself, running {{{bitlbee -V}}} in a shell shows the configure parameters in recent versions
 * List of plugins installed or protocols used, particularly third party ones.
Line 10: Line 14:
{{{
gdb --args ./bitlbee -v -n -D 2>&1 | tee /tmp/gdb.log
== Debug symbols ==

This is optional but recommended: Debug symbols add extra information that helps a lot when understanding a backtrace.

If you're using the [[Packages|debian nightly apt repo]], you already have debug symbols.

If you're building bitlbee yourself, just add --debug=1 to ./configure:

 {{{
./configure --debug=1
Line 14: Line 26:
3. Inside the debugger, start the application: For everyone else, your distro might have a way to install separate debug symbol files. Or you might have to rebuild bitlbee.
Line 16: Line 28:
{{{
run
== Getting a backtrace ==

There are a few tools to get backtraces.

 1. gdb - The most generic method. The crash needs to be reproducible.
 2. valgrind - Better for some kinds of bugs. The crash needs to be reproducible.
 3. coredumpctl - Included only in linux distros that use systemd, collects previous crashes, but doesn't always keep them.

=== With gdb ===

 1. Kill all the currently running bitlbee processes
 2. If needed, switch to the correct user first with something like "sudo -u bitlbee -s"
 3. Start bitlbee with the following commandline

 {{{
BITLBEE_DEBUG=1 gdb -ex 'handle SIGPIPE nostop noprint pass' -ex run -ex bt --args bitlbee -Dnv
Line 20: Line 46:
4. Do the steps to reproduce the application crash or freeze. If the freeze is the case, you need to press CTRL-C in gdb to get the gdb commandline back.  This will run bitlbee until the first crash, ignoring SIGPIPE (which is harmless), and dumping a backtrace at the end.
Line 22: Line 48:
5. Produce the backtrace:  4. Connect with your irc client and make it crash
 5. Copy the output
 6. Pastebin it, ensure there are no passwords or other private information.
 7. Post to irc (preferred) or open a bug report (slower)
Line 24: Line 53:
{{{
bt
If possible, keep gdb open, since you may be asked to run a few extra commands for extra information.

Otherwise, if you happen to have a coredump file, you can open it in gdb like this (assuming it's called core.1234)

 {{{
gdb -ex bt bitlbee core.1234
Line 28: Line 61:
6. Quit the debugger === With valgrind ===
Line 30: Line 63:
{{{
quit
Valgrind is a wonderful tool that can provide extra information about many bugs. It's slower but accurate, and sometimes even prevents crashes.

Repeat the same steps as above, but use the following commandline instead:

 {{{
valgrind --log-file=valgrind.log bitlbee -Dnv
Line 34: Line 71:
7. Attach the whole gdb.log to the bug report. Remember to include the ''exact'' !BitlBee version used to create your backtrace. And see if anything interesting appeared in valgrind.log, even if it didn't crash.

=== With coredumpctl ===

If you're using a distro with systemd, you might be able to get information from previous crashes with the {{{coredumpctl}}} tool. When this works, it can be very useful since you don't need to reproduce the crash again to get debugging information.

Note that using changing the "User" setting of {{{/etc/bitlbee/bitlbee.conf}}} will disable coredump creation for security reasons.

 1. Run coredumpctl to see if it caught anything. If bitlbee runs as a different user, you might have to run coredumpctl as root.

 {{{
$ sudo coredumpctl
TIME PID UID GID SIG PRESENT EXE
Sun 2016-03-20 03:30:57 UTC 3477 1001 1001 6 /usr/sbin/bitlbee
Sat 2016-05-07 17:52:16 UTC 3045 1001 1001 11 * /usr/sbin/bitlbee
}}}

 The last entry has a {{{*}}} in the PRESENT column, which means it's recent enough to keep the core dump around.

 2. Start a gdb session with that coredump.

 {{{
$ coredumpctl gdb
}}}

 3. Type "bt" to get the backtrace

 {{{
(gdb) bt
}}}

 4. Report that.

Debugging crashes

If your bitlbee is crashing, we need the following information:

  • A backtrace, ideally with debug symbols
  • Bitlbee version (/ctcp root version from your irc client)

  • Distro and distro version
  • How it was installed
    • If you built it yourself, running bitlbee -V in a shell shows the configure parameters in recent versions

  • List of plugins installed or protocols used, particularly third party ones.

Debug symbols

This is optional but recommended: Debug symbols add extra information that helps a lot when understanding a backtrace.

If you're using the debian nightly apt repo, you already have debug symbols.

If you're building bitlbee yourself, just add --debug=1 to ./configure:

  • ./configure --debug=1

For everyone else, your distro might have a way to install separate debug symbol files. Or you might have to rebuild bitlbee.

Getting a backtrace

There are a few tools to get backtraces.

  1. gdb - The most generic method. The crash needs to be reproducible.
  2. valgrind - Better for some kinds of bugs. The crash needs to be reproducible.
  3. coredumpctl - Included only in linux distros that use systemd, collects previous crashes, but doesn't always keep them.

With gdb

  1. Kill all the currently running bitlbee processes
  2. If needed, switch to the correct user first with something like "sudo -u bitlbee -s"
  3. Start bitlbee with the following commandline
    BITLBEE_DEBUG=1 gdb -ex 'handle SIGPIPE nostop noprint pass' -ex run -ex bt --args bitlbee -Dnv
    This will run bitlbee until the first crash, ignoring SIGPIPE (which is harmless), and dumping a backtrace at the end.
  4. Connect with your irc client and make it crash
  5. Copy the output
  6. Pastebin it, ensure there are no passwords or other private information.
  7. Post to irc (preferred) or open a bug report (slower)

If possible, keep gdb open, since you may be asked to run a few extra commands for extra information.

Otherwise, if you happen to have a coredump file, you can open it in gdb like this (assuming it's called core.1234)

  • gdb -ex bt bitlbee core.1234

With valgrind

Valgrind is a wonderful tool that can provide extra information about many bugs. It's slower but accurate, and sometimes even prevents crashes.

Repeat the same steps as above, but use the following commandline instead:

  • valgrind --log-file=valgrind.log bitlbee -Dnv

And see if anything interesting appeared in valgrind.log, even if it didn't crash.

With coredumpctl

If you're using a distro with systemd, you might be able to get information from previous crashes with the coredumpctl tool. When this works, it can be very useful since you don't need to reproduce the crash again to get debugging information.

Note that using changing the "User" setting of /etc/bitlbee/bitlbee.conf will disable coredump creation for security reasons.

  1. Run coredumpctl to see if it caught anything. If bitlbee runs as a different user, you might have to run coredumpctl as root.
    $ sudo coredumpctl
    TIME                            PID   UID   GID SIG PRESENT EXE
    Sun 2016-03-20 03:30:57 UTC    3477  1001  1001   6   /usr/sbin/bitlbee
    Sat 2016-05-07 17:52:16 UTC    3045  1001  1001  11 * /usr/sbin/bitlbee

    The last entry has a * in the PRESENT column, which means it's recent enough to keep the core dump around.

  2. Start a gdb session with that coredump.
    $ coredumpctl gdb
  3. Type "bt" to get the backtrace
    (gdb) bt
  4. Report that.

BitlBee Wiki: DebuggingCrashes (last edited 2018-02-06 17:11:06 by 2800:810:46b:747:76d4:35ff:feeb:fa02)