Guidelines and HOWTOs/Debugging/Shared Memory Usage in KDE: Difference between revisions

From KDE Community Wiki
(Typo)
m (MB)
 
(16 intermediate revisions by 4 users not shown)
Line 1: Line 1:
It is unfortunate that a lot of people don't understand the UNIX memory model. Unfortunately this is helped by tools like ps which are not able to provide accurate information about memory usage. In UNIX a process uses basically three kinds of memory segments: '''shared memory segments''', '''code segments''' and '''data segments'''.
This article helps understanding UNIX memory management. In UNIX a process uses basically three kinds of memory segments: '''shared memory segments''', '''code segments''' and '''data segments'''.


= Terms =
'''Shared memory''' is used by shared libraries. This memory is shared
'''Shared memory''' is used by shared libraries. This memory is shared
by all processes which use a certain library. Unfortunately there is no
by all processes which use a certain library. Unfortunately there is no
easy way to determine how much shared memory is used by how many processes.
easy way to determine how much shared memory is used by how many processes.
So a process can use 10Mb of shared memory, but you don't know whether this
So a process can use 10MB of shared memory, but you don't know whether this
memory is shared with 1, 2 or 10 processes. So if you have 10 processes who
memory is shared with 1, 2 or 10 processes. So if you have 10 processes who
each use 10Mb of shared memory this actually requires 10Mb in the best case
each use 10MB of shared memory this actually requires 10MB in the best case
and 100Mb in the worst case.
and 100MB in the worst case.


'''Code segments''' contain the actual executable code of your program.
'''Code segments''' contain the actual executable code of your program.
Line 19: Line 20:
segments are 5 times in memory.
segments are 5 times in memory.


The size reported by ps is typically just the numbers for shared, code and
The size reported by [http://man-wiki.net/index.php/Ps ps auxf] is typically just the numbers for shared, code and
data added. This is not a very accurate representation of the memory usage
data added. This is not a very accurate representation of the memory usage
of an application.
of an application.
Line 34: Line 35:
  cat /proc/<pid-of-process>/status.
  cat /proc/<pid-of-process>/status.


== Further Information ==
= Example program =
To demonstrate this, let's write a memory leaking program:


More information about this topic can be found in the article about [http://ktown.kde.org/~seli/memory/analysis.html memory usage analysis].
'''main.cpp'''
<syntaxhighlight lang="cpp-qt">
#include <KAboutData>
#include <KApplication>
#include <KCmdLineArgs>
#include <KMessageBox>
 
int main (int argc, char *argv[])
{
    KAboutData aboutData( "tutorial1", 0, ki18n("Tutorial 1"), "1.0",
                          ki18n("Displays a KMessageBox popup") );
 
    KCmdLineArgs::init( argc, argv, &aboutData );
    KApplication app;
 
    for ( int i=0; i<100000; i++ ) new QString();
 
    KMessageBox::questionYesNo( 0, i18n( "Hello World" ) );
    int* i;
    return 0;
}
</syntaxhighlight>
'''CMakeLists.txt'''
<syntaxhighlight lang="cmake">
project (tutorial1)
find_package(KDE4 REQUIRED)
include (KDE4Defaults)
include_directories(${KDE4_INCLUDES})
set(tutorial1_SRCS main.cpp)
kde4_add_executable(tutorial1 ${tutorial1_SRCS})
target_link_libraries(tutorial1 ${KDE4_KDEUI_LIBS})
install(TARGETS tutorial1  ${INSTALL_TARGETS_DEFAULT_ARGS})
</syntaxhighlight>
 
Compile and link this program:
 
<syntaxhighlight lang="bash">
cmake . && make -j4
</syntaxhighlight>
 
Run it:
<syntaxhighlight lang="bash">
./tutorial1 &
[3] 22733
</syntaxhighlight>
 
In this case the program gets the process ID 22733. We look at its memory consumption:
<pre>
cat /proc/22733/status
VmRSS:    19772 kB
VmData:    5776 kB
VmStk:        84 kB
VmExe:        8 kB
VmLib:    26804 kB
</pre>
If we change the "100000" in the program code to "1", we get a different picture:
<pre>
VmRSS:    16624 kB
VmData:    2652 kB
VmStk:        84 kB
VmExe:        8 kB
VmLib:    26804 kB
</pre>
So we see the heap is counted to <tt>VmData</tt> and contained in <tt>VmRSS</tt>.
 
== Why is it so big ==
Probably VmLib is so big because it contains all library code in memory needed for KDE. Let's see what the loader thinks are our program's dependencies:
# ldd tutorial1
        linux-vdso.so.1 =>  (0x00007fff739f3000)
        libkdeui.so.5 => /usr/local/lib64/libkdeui.so.5 (0x00007fdb6448c000)
        libkdecore.so.5 => /usr/local/lib64/libkdecore.so.5 (0x00007fdb63f31000)
        libQtDBus.so.4 => /usr/local/lib64/libQtDBus.so.4 (0x00007fdb63cb5000)
        libQtCore.so.4 => /usr/local/lib64/libQtCore.so.4 (0x00007fdb6380b000)
[...]
We gonna remove the KDE and Qt stuff, so let's write a new main.cpp:
int main (int argc, char *argv[])
{
    for ( int i=0; i<100000; i++ ) new int;
    while (true);
    return 0;
}
And compile and link it with the C++ libraries:
# g++ -o tutorial1 main.cpp
Why do we need the C++ libraries (g++ is basically gcc -lstdc++)? Because we have a call to the new keyword in the program. What are the dependencies now?
# ldd tutorial1
        linux-vdso.so.1 =>  (0x00007fffd3bff000)
        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007fe2437a5000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fe24354e000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fe243338000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fe242fcb000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fe243aae000)
that's all.
 
Note that this program runs until you terminate it with CTRL_C as we are not using KDE's messagebox any more.
# cat /proc/21028/status | grep VmLib
VmLib:      2912 kB
You see - not using libraries save place in memory, but as libraries are shared, it does not make sense to deny using libraries that are in memory anyway.
 
== disassemble it ==
Now let's disassemble the small program using the command
# objdump -d tutorial1
[...]
00000000004005b4 <main>:
  4005b4:      55                      push  %rbp
  4005b5:      48 89 e5                mov    %rsp,%rbp
  4005b8:      48 83 ec 20            sub    $0x20,%rsp
  4005bc:      89 7d ec                mov    %edi,-0x14(%rbp)
  4005bf:      48 89 75 e0            mov    %rsi,-0x20(%rbp)
  4005c3:      c7 45 fc 00 00 00 00    movl  $0x0,-0x4(%rbp)
  4005ca:      eb 0e                  jmp    4005da <main+0x26>
  4005cc:      bf 04 00 00 00          mov    $0x4,%edi
  4005d1:      e8 ea fe ff ff          callq  4004c0 <_Znwm@plt>
  4005d6:      83 45 fc 01            addl  $0x1,-0x4(%rbp)
  4005da:      81 7d fc 9f 86 01 00    cmpl  $0x1869f,-0x4(%rbp)
  4005e1:      0f 9e c0                setle  %al
  4005e4:      84 c0                  test  %al,%al
  4005e6:      75 e4                  jne    4005cc <main+0x18>
  4005e8:      eb fe                  jmp    4005e8 <main+0x34>
  4005ea:      90                      nop
  4005eb:      90                      nop
  4005ec:      90                      nop
  4005ed:      90                      nop
  4005ee:      90                      nop
  4005ef:      90                      nop
[...]
You see this is the main function in real [http://www.staerk.de/thorsten/Tutorials/Assembler_Tutorial assembler code].
 
== find out its symbols ==
Let's find out what symbols it contains:
<pre>
# nm tutorial1
0000000000600e10 d _DYNAMIC
0000000000600fe8 d _GLOBAL_OFFSET_TABLE_
00000000004006e0 R _IO_stdin_used
                w _Jv_RegisterClasses
                U _Znwm@@GLIBCXX_3.4
0000000000600df0 d __CTOR_END__
0000000000600de8 d __CTOR_LIST__
0000000000600e00 D __DTOR_END__
0000000000600df8 d __DTOR_LIST__
00000000004007a8 r __FRAME_END__
0000000000600e08 d __JCR_END__
0000000000600e08 d __JCR_LIST__
0000000000601020 A __bss_start
0000000000601010 D __data_start
0000000000400690 t __do_global_ctors_aux
0000000000400520 t __do_global_dtors_aux
0000000000601018 D __dso_handle
                w __gmon_start__
0000000000600de4 d __init_array_end
0000000000600de4 d __init_array_start
0000000000400680 T __libc_csu_fini
00000000004005f0 T __libc_csu_init
                U __libc_start_main@@GLIBC_2.2.5
0000000000601020 A _edata
0000000000601030 A _end
00000000004006c8 T _fini
0000000000400488 T _init
00000000004006d8 t _real_fini
00000000004004d0 T _start
00000000004004fc t call_gmon_start
0000000000601020 b completed.5939
0000000000601010 W data_start
0000000000601028 b dtor_idx.5941
0000000000400590 t frame_dummy
00000000004005b4 T main
</pre>
Now according to [http://man-wiki.net/index.php/Nm nm's man page] T stands for text which stands for code segment, and D stands for the data segment.
 
= See also =
* http://www-archive.mozilla.org/projects/footprint/footprint-guide.html

Latest revision as of 02:02, 17 April 2023

This article helps understanding UNIX memory management. In UNIX a process uses basically three kinds of memory segments: shared memory segments, code segments and data segments.

Terms

Shared memory is used by shared libraries. This memory is shared by all processes which use a certain library. Unfortunately there is no easy way to determine how much shared memory is used by how many processes. So a process can use 10MB of shared memory, but you don't know whether this memory is shared with 1, 2 or 10 processes. So if you have 10 processes who each use 10MB of shared memory this actually requires 10MB in the best case and 100MB in the worst case.

Code segments contain the actual executable code of your program. This memory is shared by all processes of this same program. If you start your program 5 times, it needs to load the code segment of your program only once.

Data segments contain the data of your program. This kind of memory is very important because the data segments of a process are not shared with other processes. Starting the same program 5 times makes that the data segments are 5 times in memory.

The size reported by ps auxf is typically just the numbers for shared, code and data added. This is not a very accurate representation of the memory usage of an application.

KDE applications tend to be reported as quite large because the numbers reported include the size of the shared memory segments. This size is added to the size of each KDE application while in practice the shared memory segments appear in memory only once. This is rather illusive, imagine how the output of ps would look like if it included the size of the UNIX kernel for each process!

Instead of looking at the output of ps you get a better idea of the actual memory usage of an application by looking at the output of

cat /proc/<pid-of-process>/status.

Example program

To demonstrate this, let's write a memory leaking program:

main.cpp

#include <KAboutData>
#include <KApplication>
#include <KCmdLineArgs>
#include <KMessageBox>

int main (int argc, char *argv[])
{
    KAboutData aboutData( "tutorial1", 0, ki18n("Tutorial 1"), "1.0",
                          ki18n("Displays a KMessageBox popup") );

    KCmdLineArgs::init( argc, argv, &aboutData );
    KApplication app;

    for ( int i=0; i<100000; i++ ) new QString();

    KMessageBox::questionYesNo( 0, i18n( "Hello World" ) );
    int* i;
    return 0;
}

CMakeLists.txt

project (tutorial1)
find_package(KDE4 REQUIRED)
include (KDE4Defaults)
include_directories(${KDE4_INCLUDES})
set(tutorial1_SRCS main.cpp)
kde4_add_executable(tutorial1 ${tutorial1_SRCS})
target_link_libraries(tutorial1 ${KDE4_KDEUI_LIBS})
install(TARGETS tutorial1  ${INSTALL_TARGETS_DEFAULT_ARGS})

Compile and link this program:

cmake . && make -j4

Run it:

./tutorial1 &
[3] 22733

In this case the program gets the process ID 22733. We look at its memory consumption:

cat /proc/22733/status
VmRSS:     19772 kB
VmData:     5776 kB
VmStk:        84 kB
VmExe:         8 kB
VmLib:     26804 kB

If we change the "100000" in the program code to "1", we get a different picture:

VmRSS:     16624 kB
VmData:     2652 kB
VmStk:        84 kB
VmExe:         8 kB
VmLib:     26804 kB

So we see the heap is counted to VmData and contained in VmRSS.

Why is it so big

Probably VmLib is so big because it contains all library code in memory needed for KDE. Let's see what the loader thinks are our program's dependencies:

# ldd tutorial1
       linux-vdso.so.1 =>  (0x00007fff739f3000)
       libkdeui.so.5 => /usr/local/lib64/libkdeui.so.5 (0x00007fdb6448c000)
       libkdecore.so.5 => /usr/local/lib64/libkdecore.so.5 (0x00007fdb63f31000)
       libQtDBus.so.4 => /usr/local/lib64/libQtDBus.so.4 (0x00007fdb63cb5000)
       libQtCore.so.4 => /usr/local/lib64/libQtCore.so.4 (0x00007fdb6380b000)
[...]

We gonna remove the KDE and Qt stuff, so let's write a new main.cpp:

int main (int argc, char *argv[])
{
   for ( int i=0; i<100000; i++ ) new int;
   while (true);
   return 0;
}

And compile and link it with the C++ libraries:

# g++ -o tutorial1 main.cpp

Why do we need the C++ libraries (g++ is basically gcc -lstdc++)? Because we have a call to the new keyword in the program. What are the dependencies now?

# ldd tutorial1
       linux-vdso.so.1 =>  (0x00007fffd3bff000)
       libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007fe2437a5000)
       libm.so.6 => /lib64/libm.so.6 (0x00007fe24354e000)
       libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fe243338000)
       libc.so.6 => /lib64/libc.so.6 (0x00007fe242fcb000)
       /lib64/ld-linux-x86-64.so.2 (0x00007fe243aae000)

that's all.

Note that this program runs until you terminate it with CTRL_C as we are not using KDE's messagebox any more.

# cat /proc/21028/status | grep VmLib
VmLib:      2912 kB

You see - not using libraries save place in memory, but as libraries are shared, it does not make sense to deny using libraries that are in memory anyway.

disassemble it

Now let's disassemble the small program using the command

# objdump -d tutorial1
[...]
00000000004005b4 <main>:
 4005b4:       55                      push   %rbp
 4005b5:       48 89 e5                mov    %rsp,%rbp
 4005b8:       48 83 ec 20             sub    $0x20,%rsp
 4005bc:       89 7d ec                mov    %edi,-0x14(%rbp)
 4005bf:       48 89 75 e0             mov    %rsi,-0x20(%rbp)
 4005c3:       c7 45 fc 00 00 00 00    movl   $0x0,-0x4(%rbp)
 4005ca:       eb 0e                   jmp    4005da <main+0x26>
 4005cc:       bf 04 00 00 00          mov    $0x4,%edi
 4005d1:       e8 ea fe ff ff          callq  4004c0 <_Znwm@plt>
 4005d6:       83 45 fc 01             addl   $0x1,-0x4(%rbp)
 4005da:       81 7d fc 9f 86 01 00    cmpl   $0x1869f,-0x4(%rbp)
 4005e1:       0f 9e c0                setle  %al
 4005e4:       84 c0                   test   %al,%al
 4005e6:       75 e4                   jne    4005cc <main+0x18>
 4005e8:       eb fe                   jmp    4005e8 <main+0x34>
 4005ea:       90                      nop
 4005eb:       90                      nop
 4005ec:       90                      nop
 4005ed:       90                      nop
 4005ee:       90                      nop
 4005ef:       90                      nop
[...]

You see this is the main function in real assembler code.

find out its symbols

Let's find out what symbols it contains:

# nm tutorial1
0000000000600e10 d _DYNAMIC
0000000000600fe8 d _GLOBAL_OFFSET_TABLE_
00000000004006e0 R _IO_stdin_used
                 w _Jv_RegisterClasses
                 U _Znwm@@GLIBCXX_3.4
0000000000600df0 d __CTOR_END__
0000000000600de8 d __CTOR_LIST__
0000000000600e00 D __DTOR_END__
0000000000600df8 d __DTOR_LIST__
00000000004007a8 r __FRAME_END__
0000000000600e08 d __JCR_END__
0000000000600e08 d __JCR_LIST__
0000000000601020 A __bss_start
0000000000601010 D __data_start
0000000000400690 t __do_global_ctors_aux
0000000000400520 t __do_global_dtors_aux
0000000000601018 D __dso_handle
                 w __gmon_start__
0000000000600de4 d __init_array_end
0000000000600de4 d __init_array_start
0000000000400680 T __libc_csu_fini
00000000004005f0 T __libc_csu_init
                 U __libc_start_main@@GLIBC_2.2.5
0000000000601020 A _edata
0000000000601030 A _end
00000000004006c8 T _fini
0000000000400488 T _init
00000000004006d8 t _real_fini
00000000004004d0 T _start
00000000004004fc t call_gmon_start
0000000000601020 b completed.5939
0000000000601010 W data_start
0000000000601028 b dtor_idx.5941
0000000000400590 t frame_dummy
00000000004005b4 T main

Now according to nm's man page T stands for text which stands for code segment, and D stands for the data segment.

See also