Jeff Li

Be another Jeff

Install SystemTap in Ubuntu 14.04

About 2 months ago, I happened to read a book about how to learn OS principles by hands-on experience on Solaris with MDB and DTrace. After reading the chapters about memory management, I was totally impressed by MDB and DTrace. They acts like a microscope to help you look into the kernel. For example, you can walk through the data structures used by Solaris kernel to manage the process. You can even observe the memory related data structures and emulate the translation between virtual address and physical address manually. Then I became curious about if Linux has similar tools because I am pretty sure that it will be very helpful to understand the design and implementation of Linux kernel.

Since I have heard about SystemTap before, so after googling “systemtap ubuntu”, I found several articles about how to install SystemTap in Ubuntu. However, those are kind of outdated even the one posted in the SystemTap’s homepage. For example, some complicated steps are not needed anymore while some are not applicable when working with latest kernel. So I decided to write down this experience. The environment is a Ubuntu 14.04(trust) box running in DigitalOcean with 3.13.0-36-generic kernel image.

Install SystemTap

SystemTap 2.3 is available in the apt source thus it can be installed simply with apt-get. Gcc is also required since the SystemTap script will be translated into C source code which will be compiled as kernel module finally.

1
2
3

sudo apt-get install systemtap
sudo apt-get install gcc
sudo apt-get install linux-headers-$(uname -r)

Test the installation with the HelloWorld example:

1

sudo stap -e 'probe begin { printf("Hello, World!\n"); exit() }'

You should see Hello, World! is emitted in the console.

Install debug file

SystemTap is a powerful trace tool. Let’s see another example which will print hello world to the console when the system call sys_call is called.

1

sudo stap -e 'probe kernel.function("sys_open") {log("hello world") exit()}'

It is likely to end with the error information:

1
2
3
4
5
6

semantic error: while resolving probe point: identifier 'kernel' at <input>:1:7
        source: probe kernel.function("sys_open") {log("hello world") exit()}
                      ^

semantic error: missing x86_64 kernel/module debuginfo [man warning::debuginfo] under '/lib/modules/3.13.0-36-generic/build'
Pass 2: analysis failed.  [man error::pass2]

The debug symbols are required to solve this problem. You can following the instruction in the Ubuntu Wiki.

You can also install the symbols manually if you failed by following above instructions. Goto ddeb and find a symbol file matched your kernel. If you are not able to find a matched symbol file, I suggest you upgrade your linux kernel or you will have to build it by yourself.

1
2

wget http://ddebs.ubuntu.com/pool/main/l/linux/linux-image-3.13.0-36-generic-dbgsym_3.13.0-36.63_amd64.ddeb
sudo dpkg -i  linux-image-3.13.0-36-generic-dbgsym_3.13.0-36.63_amd64.ddeb

Troubleshooting

build directory not found

Error Information:

1

Checking "/lib/modules/3.13.0-36-generic/build/.config" failed with error: No such file or directory

/lib/modules/3.13.0-36-generic/build is actually a soft link to the linux header files directory. This error is not supposed to appear because when you install the headers by sudo apt-get install linux-headers-$(uname -r), the soft link will be created automatically. But if you are as unlucky as me, please install the linux headers and make sure the build links to /usr/src/linux-headers-3.13.0-36-generic/.

disktop.stp does not work

SystemTap is also shipped with some examples. disktop.top is a SystemTap script used to get the status of reading/writing. If you use the SystemTap provided by the apt software repository, the following error is likely to occur.

1
2
3
4
5
6
7
8
9

/tmp/stapvSQlYC/stap_46806491689887ce5beb12ed5b6f2994_27689_src.c: In function ‘function_uid’:
/tmp/stapvSQlYC/stap_46806491689887ce5beb12ed5b6f2994_27689_src.c:3459:16: error: incompatible types when assigning to type ‘int64_t’ from type ‘kuid_t’
  STAP_RETVALUE = current_uid();
                ^
make[1]: *** [/tmp/stapvSQlYC/stap_46806491689887ce5beb12ed5b6f2994_27689_src.o] Error 1
make: *** [_module_/tmp/stapvSQlYC] Error 2
WARNING: kbuild exited with status: 2
Pass 4: compilation failed.  [man error::pass4]
Tip: /usr/share/doc/systemtap/README.Debian should help you get started.

This is probably a bug. One way to fix it is building SystemTap from source. 

1
2
3
4
5
6
7
8
9

sudo apt-get remove systemtap
wget https://fedorahosted.org/releases/e/l/elfutils/0.160/elfutils-0.160.tar.bz2
wget https://sourceware.org/systemtap/ftp/releases/systemtap-2.5.tar.gz
tar jxf elfutils-0.160.tar.bz2
tar zxf systemtap-2.5.tar.gz
cd systemtap-2.5
./configure --with-elfutils=../elfutils-0.160
make
sudo make install

pass 4 compile error

This is the most weird issue I have met. Almost all the examples can’t work and SystemTap keeps complaing pass 4: compilation failed. [man error::pass4]. That means the SystemTap script can be translated into C source code which can also be compile as a kernel module. However, it fails to load the kernel module. After downgrading the kernel from 3.13.0-36 to 3.13.0-32, I found ever thing is ok. After I switched back to 3.13.0-36, however, nothing is wrong. If you meet similar issue, try to clean the symbols file, linux headers and switch to other kernel version with matched debug symbol files and linux headers.

Further Reading

  1. Getting Started with SystemTap: A introduction with simple examples. It is simple and thus good for beginners.

  2. SystemTap Beginners Guid: SystemTap guide from Red Hat planned for beginners

  3. SystemTap on CentOS: Detail how to install SystemTap in CentOS. I followed the step on a CentOS 6.5 box. The whole process was very smooth. Every thing is fine including the disktop example.

  4. SystemTap examples: Examples from SystemTap’s homepage

Comments