strace cheat sheet

Nov 14, 2015 · 4 min read

linux-how-to

TL;DR

strace traces system calls and signals and is an invaluable tool for gathering context when debugging.

This cheat sheet will show a few useful ways of using strace, how to filter the output, and summarize some of the more useful commands line arguments that strace accepts.

strace PTRACE_TRACEME EPERM (Operation not permitted)

You should run strace as root. If you get this message when running as root, it means that strace is not allowed to attach to processes on your system.

You can enable strace to attach to processes by changing a setting in proc:

sudo bash -c 'echo 0 > /proc/sys/kernel/yama/ptrace_scope'

To make this change permanent between reboots first check your system for a file named: /etc/sysctl.d/10-ptrace.conf.

If your system has that file, modify it to read:

kernel.yama.ptrace_scope = 0

If your system does NOT have that file:

Open /etc/sysctl.conf in your favorite
Find the setting kernel.yama.ptrace_scope
If it exists, modify it to set the value to 0
If it does not exist, add a line with kernel.yama.ptrace_scope = 0

strace pid

strace a running process

strace -p [pid]

strace a running process and threads

strace -fp [pid]

strace a running process and print strings

strace -s 80 -fp [pid]

This will print the first 80 characters of every string.

strace program

strace a program

strace ./program

strace a program and threads

strace -f ./program

strace a program and print strings

strace -s 80 -f ./program

This will print the first 80 characters of every string.

strace futex

While using strace, you may see many futex system calls, for example:

[pid  2082] futex(0x7f4c4d3bff30, FUTEX_WAKE_PRIVATE, 1) = 0 <0.000003>
[pid  2081] futex(0x7f4c4d3bff60, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid  2082] futex(0x7f4c4d3bff60, FUTEX_WAKE_PRIVATE, 1) = 0 <0.000003>
[pid  2082] futex(0x7f4c4d3bff64, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished ...>

The futex system call is commonly used by threading implementations, like libpthread, to implement higher level locking primitives, like mutexes, semaphores, and more.

filter out futex calls

If your application is making a particularly large number of futex calls and you don’t want to trace them, you can either:

Exclude just the futex calls by using grep:

strace strace -Tf ./program 2>&1 | grep -v futex

Or, trace a specific set of system calls you do care about. In this example, trace open, read, and write this program and its threads:

strace -Tfe trace=open,read,write ./program

Other useful options

There are many useful options that be added when using strace. Some of the more useful options for general purposes are:

-f Follow threads and child processes that are created. Useful option because many programs will spawn additional processes or threads to do work.
-T Print time spent in system call. This can be useful if you are trying to determine if a particular system call is taking a lot of time to return.
-t Print the time of day at the start of each line.
-s [size] Print [size] characters per string displayed. This is useful if you are trying to trace what a program is writing to a file descriptor.
-c Print a histogram of the number of system calls and the time spent at the termination of strace.
-e trace=open,close Trace only the open and close system calls.

For a complete list of command line arguments, see the strace man page.

Conclusion

strace is an essential tool for debugging anything on a Linux system. You should use strace anytime you want to understand what a process is doing. Reaching for strace as the first line of defense when debugging anything is a great way to quickly gather context about a problem.

strace accepts many command line arguments and readers are encouraged to consult the strace man page to learn about all the arguments.