How to Debug Running Processes on Linux Using Strace to Trace System Calls, Filter Output, and Diagnose Failures

Learning how to debug running processes on Linux using strace to trace system calls, filter output, and diagnose failures is one of the most valuable skills a server administrator can develop. When a process crashes silently, hangs indefinitely, or behaves unexpectedly, standard logs often don’t tell you enough. Strace fills that gap. It intercepts every system call a process makes , file opens, network connections, memory requests , and prints them to your terminal in real time. This tutorial walks you through installing strace, attaching it to live processes, filtering noisy output, and reading results to pinpoint the root cause of failures. Whether you manage a WordPress server, a database, or a custom application, strace gives you a clear view of what your software is actually doing at the kernel level.

Prerequisites and Requirements for Tracing System Calls with Strace

Before you start tracing system calls with strace, make sure your environment meets these requirements.

Required access and software:

  • A Linux server running Ubuntu 20.04, 22.04, or a similar Debian-based distribution
  • Root or sudo privileges on the server
  • SSH access to the machine
  • Basic familiarity with the Linux command line
  • A running process you want to inspect (Apache, Nginx, MySQL, or any application)

Assumed knowledge level: You should be comfortable running commands in a terminal and understand what processes and PIDs are. You don’t need to know C or kernel internals.

Estimated time to complete: 30–45 minutes, depending on how complex your debugging scenario is.

Strace is available in the default repositories of most Linux distributions. You won’t need to compile anything from source. The official strace man page is an excellent reference to keep open while you work through this tutorial.

How to Debug Running Processes on Linux Using Strace: Step-by-Step Guide

For a related walkthrough, see: How to Set Up an Openssh Certificate Authority for Scalable User and Host Authentication on Ubuntu Server

Follow these steps in order. Each one builds on the last.

Step 1: Install Strace

First, update your package list and install strace.

sudo apt update
sudo apt install strace -y

Confirm the installation worked by checking the version.

strace --version

You should see output like strace -- version 5.x. If you’re on a Red Hat or CentOS system, use sudo yum install strace -y instead.

Step 2: Find the PID of Your Target Process

You need the process ID (PID) before you can attach strace to a running process. Use one of these commands to find it.

pgrep nginx
ps aux | grep apache2
pidof mysqld

Pick the PID from the output. For example, if pgrep nginx returns 1234, that’s your target.

Step 3: Attach Strace to a Running Process

Use the -p flag to attach strace to an existing process without restarting it.

sudo strace -p 1234

Replace 1234 with your actual PID. Strace will immediately start printing system calls. You’ll see lines like read(5, "", 4096) or epoll_wait(7, [], 1, 999). Press Ctrl+C to stop tracing at any time.

Step 4: Save Strace Output to a File

Live output scrolls fast. Save it to a file so you can analyze it properly.

sudo strace -p 1234 -o /tmp/strace_output.txt

The -o flag redirects all output to the specified file. Let it run for 30–60 seconds, then press Ctrl+C. Open the file with less /tmp/strace_output.txt to review it.

Step 5: Filter Output by System Call Type

Raw strace output is noisy. Use the -e flag to filter for specific system calls. This is where strace becomes truly powerful for diagnosing failures.

To watch only file-related calls:

sudo strace -p 1234 -e trace=file

To watch only network calls:

sudo strace -p 1234 -e trace=network

To watch only failed calls (return value of -1):

sudo strace -p 1234 -e trace=all -e fault=all 2>&1 | grep "= -1"

Filtering failed calls is especially useful. Every = -1 result means something went wrong. Look for ENOENT (file not found), EACCES (permission denied), or ECONNREFUSED (connection refused).

Step 6: Trace a New Process from the Start

Sometimes you want to trace a process from launch rather than attaching mid-run. Run strace directly before your command.

sudo strace -o /tmp/wp_trace.txt php /var/www/html/wp-cron.php

This is useful for debugging WordPress CLI tasks, cron jobs, or scripts that fail immediately on startup. The full system call log from process start gives you the complete picture.

Step 7: Use Timestamps and Call Counts for Deeper Analysis

Add timestamps to understand timing issues. Use -t for wall-clock time or -T to show time spent in each call.

sudo strace -p 1234 -T -o /tmp/strace_timed.txt

To get a summary of which calls were made most often, use -c.

sudo strace -p 1234 -c

When you press Ctrl+C, strace prints a table showing call counts, errors, and total time per system call. A high error count on open or connect calls points directly to your problem.

Troubleshooting Common Strace Issues When Diagnosing Failures

Even experienced admins hit snags with strace. Here are the most common problems and how to fix them.

Problem: “attach: ptrace(PTRACE_SEIZE): Operation not permitted”

This happens when you run strace without sudo on a process owned by root or another user. Always use sudo strace when attaching to system processes.

Problem: Output is too fast to read

Don’t try to read it live. Always use -o filename to save output, then search through it with grep.

grep "ENOENT" /tmp/strace_output.txt

Problem: Strace slows down the target process significantly

Strace does add overhead. Don’t leave it attached to production processes for long periods. Attach it briefly, capture what you need, then detach with Ctrl+C.

Problem: Multi-threaded process shows incomplete data

Add the -f flag to follow child processes and threads.

sudo strace -f -p 1234 -o /tmp/strace_threads.txt

This is critical for tracing Apache, PHP-FPM, or MySQL, which all use multiple threads or worker processes. The Linux kernel tracing documentation explains the underlying ptrace mechanism if you want a deeper understanding.

Warning: Never run strace on a high-traffic production server without a maintenance window. The performance impact can cause real slowdowns under load.

Conclusion

You now know how to debug running processes on Linux using strace to trace system calls, filter output, and diagnose failures at the system level. You’ve learned how to attach strace to live processes, save and filter output, spot failed calls, and handle threads. These skills apply directly to debugging WordPress servers, Nginx misconfigurations, database connection errors, and any other application running on Linux. From here, consider exploring related tools like ltrace for library calls, perf for performance profiling, or lsof for open file inspection. Each tool complements strace and gives you a more complete debugging toolkit. The more comfortable you get reading system call output, the faster you’ll solve problems that would otherwise take hours of guesswork.

Similar Posts