Guidelines and HOWTOs/Debugging/How to create useful crash reports

From KDE Community Wiki
Revision as of 14:59, 15 March 2023 by Ngraham (talk | contribs) (Update coredumpctl section to be a bit more user-friendly.)

Note

This whole page is irrelevant if you're using Windows, MacOS, or Android, or AppImages on Linux.


A good crash report at Bugzilla consists of two parts: a description of how to reproduce the crash and a backtrace of the crash. With one of those elements missing, it is much harder (if not impossible) for developers to tackle the problem.

A description should consist of more than only "it crashed": try to describe everything you did prior to the crash. Did you click on a button or open a particular website or file? That little detail which may look useless to you may be useful for the developer, so just write it down.


Backtraces

Backtraces are essential. They may look meaningless to you, but they might actually contain a wealth of useful information. A backtrace describes which functions were called prior to the crash, so that developers may track down in which function the mess started. Having good backtraces has a downside: libraries and executables occupy much more disk space than their optimized counter parts. That's the reason why many distros choose to install stripped files which result in useless backtraces. But don't worry; with some modifications you can create full blown useful backtraces for KDE applications.

Note

Don't attach the backtrace to the bug report. Instead, simply paste it. This way it is much easier for developers to search for duplicate reports, because attachments will not be searched.


Install debugging packages

If your distribution has debugging-enabled packages, install them.

It is easy to see which debug packages you are missing from looking at the backtrace. For example, take the following line from a backtrace:

#6  0xb7975bdc in ?? () from /usr/lib/libkmailprivate.so.4

The ?? indicates that the library libkmailprivate.so.4 does not have debug information, which might be available in separate debug packages. In this case, it is pretty easy to guess that you need to install debug packages for KMail to get a better backtrace.

Sometimes, you need to install more than one debug package to get a good backtrace. This depends on how the distribution splits up the packages. For example, for some distributions it is enough to install the debug package for kdepim to get enough debugging information for a crash in KMail, for other distributions there is an additional debug package just for KMail.

Here's a list of how to obtain debug packages for some distributions:

Arch Linux

Arch Linux provides debug packages via its debuginfod server. They should work out of the box in gdb and drkonqi. See their informative document Getting Traces.

Debian

Recent versions of Debian (e.g. Debian testing/Bookworm) have debuginfod. See https://wiki.debian.org/Debuginfod .

You will need to set the environment variable

DEBUGINFOD_URLS="https://debuginfod.debian.net"

E.g. run the command below, then restart the computer:

echo export DEBUGINFOD_URLS=\"https://debuginfod.debian.net\" | sudo tee -a /etc/profile

If your version of Debian does not have debuginfod, see below.

Debian offers -dbg packages to easy create useful backtraces. Just install the corresponding -dbg package. e.g. kdepim-dbg for KMail crashes. The dependencies of -dbg makes sure to pull in the other needed packages. See also the Debian document on How to get a Backtrace.

Fedora

Recent versions of Fedora (e.g. Fedora 37) have debuginfod. See https://fedoraproject.org/wiki/Debuginfod .

If your version of Fedora does not have debuginfod, see below.

In Fedora you need to open a terminal and enter :

  1. dnf provides "*/libkmailprivate.so.4"
    to find out which package provides that file. The reason to replace the path /usr/lib with * is that some installation paths are symbolic links to other directories; dnf provides should resolve the wildcard.
  2. sudo dnf debuginfo-install kmail-libs
    to install (as root) the debug packages for that package, and its dependencies.

StackTraces is a complete and detailed guide for Fedora describing how to proceed. Fedora uses a separate debuginfo repository that has to be enabled, `dnf debuginfo-install` should do this.

FreeBSD ports

You need to recompile the concerned packages with the flag WITH_DEBUG. Please refer to the KDE on FreeBSD FAQ.

Gentoo

Gentoo has its own web page describing how to proceed.

Ubuntu-based distros (Ubuntu, Kubuntu, KDE Neon, Linux Mint)

(K)Ubuntu has debuginfod starting with 22.10. See https://ubuntu.com/server/docs/service-debuginfod https://ubuntu.com/blog/debuginfod-is-now-available-in-ubuntu .

For Linux distributions based on versions of Ubuntu previous to 22.10, see below.

Procedure (K)Ubuntu gdb without debuginfod

This is the procedure "On (K)Ubuntu one KDE process crashes, you can reproduce that crash, and you know the name of the KDE process/executable > gdb > dbgsym ddeb packages > file with good backtrace ~/backtrace.txt".

If your (K)Ubuntu Linux distribution does not have debuginfod.

For example the name of the KDE process is "kcalc" and we can crash it by sending the signal "SIGSEGV" to the process "kcalc".

Do replace "kcalc" with the actual name of the KDE executable. Do replace sending the "SIGSEGV" signal with you actually reproducing the issue and making the KDE process crash.

As per https://wiki.ubuntu.com/DebuggingProgramCrash and https://wiki.ubuntu.com/Backtrace

Add dbgsym ddeb repository

Open one terminal and run:

echo -e "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse\ndeb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse\ndeb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/ddebs.list
sudo apt install ubuntu-dbgsym-keyring
sudo apt update
Generate core
gdb kcalc
# If it asks "Enable debuginfod for this session? (y or [n])", answer "n".
(gdb) run

Wait for the kcalc window to be shown and finish loading.

Reproduce the issue and crash the KDE process. In this example, in another terminal we run:

psaux | grep kcalc
# Get the process ID (PID) of the kcalc process. E.g. 2282.
kill -SIGSEGV 2282
# Or: killall -SIGSEGV kcalc -v

In the terminal with gdb:

Thread 1 "kcalc" received signal SIGSEGV, Segmentation fault.
0x00007ffff5afe32f in poll () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) set use-coredump-filter off
(gdb) set dump-excluded-mappings off
(gdb) generate-core-file ~/core.core
(gdb) quit
Install all dbgsym ddeb packages for core
sudo apt install debian-goodies
sudo find-dbgsym-packages --all ~/core.core
sudo find-dbgsym-packages --install ~/core.core

If it fails to install the dbgsym ddeb packages with an error like "The following packages have unmet dependencies: libexpat1-dbgsym : Depends: libexpat1 (= 2.4.8-2) but 2.4.8-2ubuntu0.22.10.1 is to be installed E: Unable to correct problems, you have held broken packages.".

Take the command line displayed e.g. "apt install --no-install-recommends kcalc-dbgsym kde-style-breeze-dbgsym kimageformat-plugins-dbgsym etc.". Remove the package "libexpat1-dbgsym" from that command line and run the resulting command line.

Generate backtrace from core
gdb kcalc ~/core.core
# If it asks "Enable debuginfod for this session? (y or [n])", answer "n".
(gdb) set trace-commands on
(gdb) set logging overwrite on
(gdb) set pagination 0
(gdb) set logging file ~/backtrace.txt
(gdb) set logging enabled on
(gdb) backtrace full
(gdb) info registers
(gdb) info locals
(gdb) x/16i $pc
(gdb) thread apply all backtrace
(gdb) quit

Or, on one line:

gdb -batch -ex 'set trace-commands on' -ex 'set logging overwrite on' -ex 'set pagination 0' -ex 'set logging file ~/backtrace.txt' -ex 'set logging enabled on' -ex 'backtrace full' -ex 'info registers' -ex 'info locals' -ex 'x/16i $pc' -ex 'thread apply all backtrace' kcalc ~/core.core

Apply this procedure when one KDE process crashes, you can reproduce that crash, and you know the name of the KDE process/executable. Add the contents of the ~/backtrace.txt file to the bug report.

Uninstall all dbgsym ddeb packages

In Ubuntu there are few *-dbg deb packages and they are mostly for language runtimes. Do not install or uninstall those. E.g. on Kubuntu 22.10 only 4 *-dbg deb files are installed by default: libc6-dbg, libkrb5-dbg, libxdmcp6-dbg, libxxf86vm1-dbg.

If you really want, you can remove all the ddebs and the ddebs repositories. See https://wiki.ubuntu.com/DebuggingProgramCrash

KDE neon

Neon has the debug symbols in its regular repository.

Mageia

Mageia offers -debug packages in a separate repository. See their wiki page.

openSUSE

openSUSE Tumbleweed x86_64 has experimental support for debuginfod. See https://debuginfod.opensuse.org/

If not using debuginfod, you should only install the -debuginfo packages, for example: kdepimlibs4-debuginfo. You can find these packages in KDE repositories. There is also a dedicated openSUSE debugging page.

Solus

Solus separates debugging symbols by default but they are included in the official repository and are easy to install. For instance if you need symbols for knotifyconfig you simply install knotifyconfig-dbginfo via the Software Center or via eopkg.

Finding the package that provides the symbol is done like so:

#6  0xb7975bdc in ?? () from /usr/lib64/libkwin.so.5

Use the filename in this case libkwin.so.5 in eopkg search-file:

eopkg search-file libkwin.so.5

This gives kwin as the package name, you can then search for kwin and find the debug symbols package:

eopkg search kwin | grep dbginfo
kwin-dbginfo          - Debug symbols for kwin

Compile from sources

If your distribution doesn't have debugging-enabled packages for KDE, you'll have to compile the software from source code. See Get_Involved/development

Debuginfod

You can determine if you Linux distribution has debuginfod installed and set up correctly.

The environment variable DEBUGINFOD_URLS should be be defined and have the correct value for your Linux distribution.

$ echo $DEBUGINFOD_URLS
https://debuginfod.debian.net

When you run gdb it should ask "Enable debuginfod for this session?". Reply "y" to this question.

$ gdb kcalc
GNU gdb (Debian 12.1-3) 12.1
...
Reading symbols from kcalc...

This GDB supports auto-downloading debuginfo from the following URLs:
https://debuginfod.debian.net
Enable debuginfod for this session? (y or [n])

In order not to reply to this question each time you run GDB. Append to ~/.gdbinit

set debuginfod enabled on

Procedure gdb and debuginfod

This is the procedure "one KDE process crashes, you can reproduce that crash, and you know the name of the KDE process/executable > gdb > debuginfod > file with good backtrace ~/backtrace.txt".

If your Linux distribution has debuginfod and debuginfod is configured correctly.

For example the name of the KDE process is "kcalc" and we can crash it by sending the signal "SIGSEGV" to the process "kcalc".

Do replace "kcalc" with the actual name of the KDE executable. Do replace sending the "SIGSEGV" signal with you actually reproducing the issue and making the KDE process crash.

Open one terminal and run:

$ gdb kcalc
GNU gdb (Debian 12.1-3) 12.1
...
Reading symbols from kcalc...

This GDB supports auto-downloading debuginfo from the following URLs:
https://debuginfod.debian.net
Enable debuginfod for this session? (y or [n])

y
(gdb) run

Wait for debug symbols to finish downloading if needed. Wait for the kcalc window to be shown and finish loading.

Reproduce the issue and crash the KDE process. In this example, in another terminal we run:

psaux | grep kcalc
# Get the process ID (PID) of the kcalc process. E.g. 2282.
kill -SIGSEGV 2282

In the terminal with gdb:

Thread 1 "kcalc" received signal SIGSEGV, Segmentation fault.
0x00007ffff5afe32f in poll () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) set logging file ~/backtrace.txt
(gdb) set logging enabled on
Copying output to /home/n/backtrace.txt.
Copying debug output to /home/n/backtrace.txt.
(gdb) thread apply all bt full
...
--Type <RET> for more, q to quit, c to continue without paging--
c
(gdb) set logging enabled off
Done logging to /home/n/backtrace.txt.
(gdb) quit

Apply this procedure when one KDE process crashes, you can reproduce that crash, and you know the name of the KDE process/executable. Add the contents of the ~/backtrace.txt file to the bug report.

kcrash and drkonqi

In order for drkonqi to use debuginfod, we need to wait for https://bugs.kde.org/show_bug.cgi?id=454063 to be available in Linux distributions.

Retrieving a backtrace using the KDE Crash Report tool

Do whatever you did to make the application crash, and crash it again! The KDE Crash Report dialog should appear right after the crash, which shows the "Developer Information" tab.

KDE Crash Dialog
KDE Crash Dialog

Click that tab and wait for a minute. This process may take quite some memory, so things may go sluggish all of a sudden. But the result should look much better.

Note

You need GDB installed to get the backtrace of a crash. Please read the next section to know what GDB is, and how to install it.


Retrieving a backtrace with GDB

In some cases, it is not possible to create a backtrace with the KDE Crash Dialog. This may be caused by an application which entered an infinite loop, or the crash dialog did not appear at all for some reason. You can try to grab a backtrace with gdb, the GNU Debugger. GDB is widely available through distribution packages.

You can run an application from inside gdb, or attach gdb to an already running process. The latter may be useful when an application already has entered an infinite loop. But we will first start with running an application inside gdb. From the shell, run:

$ gdb someKDEapp

The GDB prompt will appear. Note that this does not start the application itself, so you should immediately run it by invoking the run command:

(gdb) run

This will run the application like you are used to, and you can work with it like normal (it only consumes far more memory and may feel sluggish). Now it's time to reproduce your crash. When you succeed, the application just closes and you should return to your GDB prompt.

Note

Some KDE applications (such as plasma-desktop, kmix, rekonq, kmail, kontact, amarok, juk and ktorrent) have special code to ensure that there is only one running instance of the application at a time. For these applications you should type in "run --nofork" at the (gdb) prompt instead of "run" because otherwise gdb will try to debug the wrong process. If you are unsure as to whether to use --nofork just try it. If the application says it's an unknown option you can remove --nofork.


Now it's time to run the 'backtrace' command:

(gdb) backtrace

This should give a good backtrace which can be posted at the KDE Bugzilla.

In case you want to attach to an existing process, run the following command in the shell:

$ gdb someKDEapp pid

where pid is the process ID of the process you want to attach to. Once attached, and the process is in an infinite loop, after using the 'backtrace' command again a useful backtrace will appear. You can use 'continue' command to let the application run again and press Ctrl+C in gdb to be able to again enter commands.

Retrieving a backtrace when an uncaught exception is causing a crash

If your backtrace looks like this:

 Thread 1 (Thread 0xb77be730 (LWP 5794)):
 [KCrash Handler]
 #7  0x00e9b416 in __kernel_vsyscall ()
 #8  0x002b1c8f in raise () from /lib/i386-linux-gnu/libc.so.6
 #9  0x002b52b5 in abort () from /lib/i386-linux-gnu/libc.so.6
 #10 0x001c14ed in __gnu_cxx::__verbose_terminate_handler() () from
 /usr/lib/i386-linux-gnu/libstdc++.so.6
 #11 0x001bf283 in ?? () from /usr/lib/i386-linux-gnu/libstdc++.so.6
 #12 0x001bf2bf in std::terminate() () from
 /usr/lib/i386-linux-gnu/libstdc++.so.6
 #13 0x001bf464 in __cxa_rethrow () from /usr/lib/i386-linux-gnu/libstdc++.so.6
 [...]

then only GDB can help to get a useful backtrace. A few more commands are needed. From the shell, run:

 $ gdb someKDEapp

Then start loading the application:

 (gdb) start

The GDB prompt will reappear. Turn logging on:

 (gdb) set logging file ~/gdblog.txt
 (gdb) set logging on

We must now tell gdb to stop just before the exception is thrown. enter:

 (gdb) break __cxa_throw
 (gdb) break __cxa_rethrow

and finally:

 (gdb) cont

Now reproduce the crash. When you succeed, the application window will look frozen.

Don't worry and switch back to the shell window. If you see the gdb prompt, type:

 (gdb) backtrace

press enter until the (gdb) prompt reappears.

All done! you can now quit gdb:

 (gdb) kill

The kill command will stop the program and now enter:

 (gdb) quit

Copy/paste the content saved in the ~/gdblog.txt file to your bug report.


Retrieving a backtrace using coredumpctl

If your application crashed in the past and you cannot get it to crash again, it is often possible to retrieve the backtrace anyway using the coredumpctl tool.

Run coredumpctl --reverse. You will see output like this:

Fri 2020-12-18 13:47:18 MST   30930  1000   100  11 present   /home/nate/kde/usr/bin/neochat
Fri 2020-12-18 12:17:45 MST   30742  1000   100  11 present   /home/nate/kde/usr/bin/dolphin
Fri 2020-12-18 09:50:54 MST   30913  1000   100  11 present   /home/nate/kde/usr/bin/systemsettings5
Thu 2020-12-17 20:11:19 MST   21396  1000   100  11 present   /home/nate/kde/usr/bin/baloo_file

Find the table row that corresponds to the crashing app; the name of its executable file will be listed in the last column. The second column lists a number; copy that number to the clipboard, then run the following command:

$ coredumpctl gdb [paste the number]

It may take some time for something to happen. Eventually you will see a "(gdb)" prompt. Type bt at the prompt and hit the Enter/Return key. If it says "--Type <RET> for more, q to quit, c to continue without paging--" at the bottom of the terminal window, then press the Enter/Return key. Do this until there's nothing more. Then copy the text in the terminal window and paste it into the bug report in question.

Retrieving a backtrace with Valgrind

When it comes to crashes, Valgrind is also a useful tool to create a backtrace. It's not a substitution for GDB, but rather a supplement.

When you run an application in valgrind, every piece of memory read or written by the application is being checked. Valgrind will report erroneous memory operations in the standard output or in a log file. Since most crashes are due to an invalid memory read, valgrind can be useful to track down where the problem occurs.

Note

Valgrind consists of several tools in order to check or profile an application. For this article, we only use memcheck, the default tool when valgrind is being invoked.


Like GDB, Valgrind makes running an application much slower, while consuming a lot more resources.

Start the application within valgrind:

$ valgrind --log-file=someKDEapp someKDEapp

Now reproduce the crash. As soon as this happens, the application and valgrind will terminate. What's left is a file named someKDEapp.pid where pid is replaced by the process ID of the valgrind process. The file may list more errors than the one causing the crash. Here's the bit causing the crash which corresponds to the GDB backtrace above:

==23292== Invalid read of size 4
==23292==    at 0x806BD9E: TreeMapItem::parent() const (treemap.h:285)
==23292==    by 0x8065FB9: TreeMapItemList::compareItems(void*, void*) (treemap.cpp:720)
==23292==    by 0x50AC618: QGList::operator==(QGList const&) const (in /usr/qt/3/lib/libqt-mt.so.3.3.8)
==23292==    by 0x806D3BF: QPtrList<TreeMapItem>::operator==(QPtrList<TreeMapItem> const&) const (qptrlist.h:74)
==23292==    by 0x8062DE7: TreeMapWidget::mousePressEvent(QMouseEvent*) (treemap.cpp:1840)
==23292==    by 0x4E2FA62: QWidget::event(QEvent*) (in /usr/qt/3/lib/libqt-mt.so.3.3.8)
==23292==    by 0x4D96CA6: QApplication::internalNotify(QObject*, QEvent*) (in /usr/qt/3/lib/libqt-mt.so.3.3.8)
==23292==    by 0x4D97A87: QApplication::notify(QObject*, QEvent*) (in /usr/qt/3/lib/libqt-mt.so.3.3.8)
==23292==    by 0x4809AC3: KApplication::notify(QObject*, QEvent*) (kapplication.cpp:550)
==23292==    by 0x4D36FD1: QETWidget::translateMouseEvent(_XEvent const*) (in /usr/qt/3/lib/libqt-mt.so.3.3.8)
==23292==    by 0x4D368AF: QApplication::x11ProcessEvent(_XEvent*) (in /usr/qt/3/lib/libqt-mt.so.3.3.8)
==23292==    by 0x4D46760: QEventLoop::processEvents(unsigned) (in /usr/qt/3/lib/libqt-mt.so.3.3.8)
==23292==  Address 0x2C is not stack'd, malloc'd or (recently) free'd

But to be sure, just attach the whole log file to the crash report.

Good backtrace vs. bad backtrace

Bad backtrace:

Thread 1 (Thread 0x7ffff451f340 (LWP 5169) "kcalc"):
#0  0x00007ffff6122faf in poll () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007ffff502f33d in g_main_context_iterate.constprop () from /lib64/libglib-2.0.so.0

Good backtrace:

Thread 1 (Thread 0x7ffff451f340 (LWP 5617) "kcalc"):
#0  0x00007ffff6122faf in __GI___poll (fds=0x5555559391f0, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
        sc_ret = -516
        sc_cancel_oldtype = 0
        sc_ret = <optimized out>
#1  0x00007ffff502f33d in g_main_context_poll (priority=<optimized out>, n_fds=2, fds=0x5555559391f0, timeout=<optimized out>, context=0x7fffe0005010) at ../glib/gmain.c:4543
        ret = <optimized out>
        errsv = <optimized out>
        poll_func = 0x7ffff4fdf280 <g_poll>
        poll_func = <optimized out>
        ret = <optimized out>
        errsv = <optimized out>
#2  g_main_context_iterate.constprop.0 (context=0x7fffe0005010, block=1, dispatch=1, self=<optimized out>) at ../glib/gmain.c:4233
        max_priority = 2147483647
        timeout = -1
        some_ready = <optimized out>
        nfds = 2
        allocated_nfds = <optimized out>
        fds = 0x5555559391f0
        begin_time_nsec = 2890812067963

Add application-specific details

For some KDE software, it's helpful to include specific details of how you were using it inside the bug report:

  • Plasma Desktop: Plasmoids you have in your desktop (both official and unofficial), desktop settings (wallpaper plugin, themes), activities and dashboard configuration. For more information, see the Plasma debugging page.
  • KWin (the window manager): state of compositing (desktop effects): enabled / suspended / disabled, kind of effects enabled, window decoration. For more information, see the KWin debugging page.
  • Dolphin: file view mode, grouping and sorting settings, preview settings, directory you were browsing.
  • KMail: Mail protocols and account-types you use.
  • KWrite/Kate: Type of the document you were editing.
  • Juk/Dragon/Amarok (and other multimedia players): Type of media (extension and format) you were watching and/or listening to.
  • Discover: If it crashed during a search, include what you were searching for and what page you were on when it crashed.