Skip to content

Latest commit

 

History

History
154 lines (136 loc) · 7.85 KB

performance-troubleshooting.md

File metadata and controls

154 lines (136 loc) · 7.85 KB

Linux Performance Troubleshooting

A short path from "the server is slow" to the resource that is the cause. Start with the USE method (utilisation, saturation, errors for each resource), then use the one-liners below to dig into CPU, memory, disk and network.

Documentation: Brendan Gregg: Linux Performance · USE Method · man sar, man vmstat, man ss

Verification: commands follow the man pages and common practice. No command was executed. Most need sysstat (sudo apt install sysstat), plus htop, iotop, iftop or nethogs where noted. perf and strace can slow the target; use them briefly on production.

See also: Linux · systemd · Storage and Filesystems · Docker Troubleshooting · IPTables · tmux


USE Method Checklist

For each resource ask: how busy is it (U), is work queueing (S), are there errors (E)?

RESOURCE UTILISATION SATURATION ERRORS
CPU mpstat -P ALL 1 (%usr+%sys) vmstat 1 column r above core count, load average dmesg -T | grep -i mce
Memory free -h, vmstat 1 Swapping: si/so columns, sar -B, PSI /proc/pressure/memory dmesg -T | grep -i 'out of memory'
Disk iostat -xz 1 (%util) aqu-sz, await smartctl -a, dmesg | grep -i 'I/O error'
Network sar -n DEV 1 vs link speed ss -s, nstat, drops in ip -s link ip -s link errors, ethtool -S <IF>

First 60 Seconds

uptime                       # load averages: 1, 5, 15 minutes
dmesg -T | tail -20          # kernel errors, OOM kills
vmstat 1 5                   # r, free, si/so, us/sy/id/wa
mpstat -P ALL 1 3            # per-CPU balance
pidstat 1 3                  # per-process CPU
iostat -xz 1 3               # disk latency and utilisation
free -h                      # memory and swap
sar -n DEV 1 3               # network throughput
sar -n TCP,ETCP 1 3          # TCP connections and retransmits
top                          # overview

Load average counts runnable and uninterruptible (usually I/O waiting) tasks. Compare to nproc. A high load with low CPU usually means I/O wait.


CPU

COMMAND DESCRIPTION
top -o %CPU Sort by CPU (press 1 for per-core, H for threads)
ps -eo pid,ppid,cmd,%cpu --sort=-%cpu | head Top CPU processes
mpstat -P ALL 1 Per-core usage. One core pegged means single-threaded work
pidstat -u 1 Per-process CPU, each second
pidstat -t -p <PID> 1 Threads of one process
vmstat 1 us user, sy system, wa I/O wait, st stolen by hypervisor
cat /proc/pressure/cpu Pressure stall information
sudo perf top Live hottest functions
sudo perf record -g -p <PID> -- sleep 30 && sudo perf report Profile a process
lscpu | grep -i mhz; cpupower frequency-info Frequency scaling

High st on a VM means a noisy neighbour or an over-committed host. High sy often means many syscalls, context switches or network interrupts.


Memory

COMMAND DESCRIPTION
free -h Look at available, not free. Cached memory is reclaimable
ps aux --sort=-%mem | head Top memory processes
smem -tk -s pss Proportional memory per process (needs smem)
vmstat 1 si / so above zero for a while = swapping
swapon --show Swap devices and use
cat /proc/meminfo Everything: MemAvailable, Slab, Dirty
cat /proc/pressure/memory Memory pressure (some, full)
sudo slabtop -o | head Kernel cache usage
sudo dmesg -T | grep -i -e 'out of memory' -e 'killed process' OOM killer events
journalctl -k | grep -i oom Same from the journal
cat /proc/<PID>/status | grep -e VmRSS -e VmSwap One process's memory
cat /proc/sys/vm/swappiness Swap tendency (default 60, lower on database hosts)

OOM in containers: the container's cgroup limit is hit, see Docker Troubleshooting.

Drop caches only for testing, never as a fix: echo 3 | sudo tee /proc/sys/vm/drop_caches.


Disk I/O

More detail in Storage and Filesystems.

COMMAND DESCRIPTION
iostat -xz 1 r_await/w_await (ms), aqu-sz, %util
sudo iotop -oPa Which process is writing or reading
pidstat -d 1 Per-process kB read and written
vmstat 1 bi/bo blocks in and out, wa
cat /proc/pressure/io I/O pressure
df -h; df -i Space and inodes
sudo lsof +L1 Deleted files still using space
sudo lsof -p <PID> Files a process holds
sudo iostat -x 1 -p sda One disk with partitions
findmnt -o TARGET,SOURCE,FSTYPE,OPTIONS Mount options (noatime, discard)
sudo smartctl -H /dev/sda Disk health

Network

COMMAND DESCRIPTION
ss -s Socket summary
ss -tulpn Listening sockets and owners
ss -tn state established '( dport = :443 )' | wc -l Established connections to a port
ss -ti dst <IP> TCP internals to one host (rtt, retransmits, cwnd)
ip -s link show <IF> Packets, errors, drops, per interface
sar -n DEV 1 Throughput per interface
sar -n TCP,ETCP 1 TCP active/passive opens, retransmits
nstat -az | grep -i -e retrans -e drop Kernel counters
sudo ethtool <IF> / sudo ethtool -S <IF> Link speed, duplex, NIC counters
sudo iftop -i <IF> Live bandwidth per connection
sudo nethogs <IF> Bandwidth per process
sudo tcpdump -ni <IF> host <IP> and port <PORT> Capture
mtr -rwc 50 <HOST> Path loss and latency
ping -c 20 <HOST> Loss and jitter
iperf3 -s / iperf3 -c <HOST> Throughput test between two hosts
dig +stats <NAME> DNS response time
cat /proc/sys/net/netfilter/nf_conntrack_count /proc/sys/net/netfilter/nf_conntrack_max Conntrack table fill (full table drops packets)

Many TIME_WAIT sockets are normal. Many CLOSE_WAIT sockets mean the application is not closing connections. See also IPTables and nftables for firewall counters.


Processes and Tracing

COMMAND DESCRIPTION
sudo strace -f -tt -p <PID> Syscalls of a running process
sudo strace -c -p <PID> Syscall summary (stop with Ctrl-C)
sudo strace -e trace=network,file -f <CMD> Filtered trace
sudo ltrace -p <PID> Library calls
sudo lsof -p <PID> Open files and sockets
ls /proc/<PID>/fd | wc -l Open file descriptors
cat /proc/<PID>/limits Resource limits
sudo pstack <PID> / gdb -p <PID> -batch -ex "thread apply all bt" Thread stacks
ps -eo stat,pid,cmd | awk '$1 ~ /^D/' Processes stuck in uninterruptible sleep (usually I/O)
ps -eo stat,ppid,pid,cmd | awk '$1 ~ /^Z/' Zombies
systemd-cgtop Per-service CPU, memory, I/O (systemd)
journalctl -p warning -b Warnings and errors since boot

Quick Diagnosis

SYMPTOM LOOK AT
High load, low CPU I/O wait: iostat -xz 1, D state processes, NFS hangs (NFS)
High sy CPU Interrupts, syscalls: perf top, pidstat -w 1 for context switches
Free memory low Normal if available is fine. Swapping or OOM kills are the real signs
Swap growing smem, vmstat 1, a leaking process, swappiness
Slow disk await high: failing disk (SMART), RAID rebuild (RAID with mdadm), a noisy process
Random timeouts Packet loss and retransmits (nstat, ip -s link), conntrack full, DNS slow
"No space left on device" df -h and df -i, deleted-but-open files
Service slow after boot only systemd-analyze blame, critical-chain (systemd)

Monitoring Tools

TOOL USE
htop, btop Interactive process view
glances One-screen overview
atop Records history, replay after an incident (atop -r)
sar (sysstat) Historical CPU, memory, disk, network. Enable sysstat.service
netdata, Prometheus node_exporter Dashboards and alerts