Choose commands from the failure symptom
A user reports a slow application. Running a fixed list of commands or immediately restarting it does not establish the cause. First identify the service and when latency began, then use Linux command-line tools to examine CPU, memory, storage and dependencies. This guide chooses commands from symptoms; deleting logs or restarting before collecting evidence can erase the clues.
Example: slow Nginx with low CPU
Low CPU does not rule out resource pressure. The application might be waiting for storage or its backend. Compare endpoint latency with IO wait, available memory and upstream health.
Command environment
Examples use Ubuntu 24.04 LTS, Bash, systemd and iproute2. Check package and service names on other distributions.
Run these commands on the existing server. No separate VM allocation is required. Disk walks such as du add load to busy storage.
Core tools are coreutils, procps, iproute2 and curl. Protected logs and all-listener PID inspection need sudo. Standard privileges usually suffice for other checks.
Containers and older tools
Inside a container, /proc and network tools usually report that container’s namespace. Inspect the host for host-level state. Ifconfig and netstat are not installed everywhere, so examples use ip and ss.
Install missing diagnostic tools, not a full system upgrade
On the stated Ubuntu 24.04 baseline, the commands below install the packages used here. Installation changes the host and requires sudo, approved repositories and a change window; skip packages already present. apt update refreshes indexes, not installed package versions. lsof and dnsutils may be absent from minimal images.
sudo apt-get update
sudo apt-get install -y procps iproute2 curl lsof dnsutils
command -v vmstat ss curl lsof dig
dpkg-query -W procps iproute2 curl lsof dnsutilsExpected: command -v prints executable paths and dpkg-query reports installed versions. A missing path identifies an unavailable tool; repository/signature errors require repository and clock checks, not disabling signature validation.
Order of investigation
Make the investigation traceable: record the incident time and impact, examine resources, and compare those observations with service logs, DNS and network paths. Once evidence supports a likely cause, make one limited change and repeat the original test so its effect is clear.
Protect evidence and limit sudo
Process, environment and configuration output might contain secrets. Keep it out of public tickets. Correct permissions only for verified paths and owners. Broad recursive chmod or chown is not a general Permission denied fix.
CPU, memory, disks and service state
The following read-only commands capture the server's initial state. Most tools are already installed; add only a missing package from the system's approved repository. Collect evidence first so package installation or configuration changes do not obscure the incident's original state.
date -u --iso-8601=seconds
uptime
free -h
vmstat 1 5
df -hT
df -i
ps -eo pid,ppid,comm,%cpu,%mem --sort=-%cpu | head -n 15
systemctl --failed --no-pager
ip -br address
ip route
ss -lntInterpret load with CPU count and IO wait. In free, inspect available memory because cache is reclaimable. Df -i separates inode from byte exhaustion. Vmstat’s first sample averages since boot, while subsequent samples cover current intervals.
systemctl status nginx --no-pager
sudo journalctl -u nginx --since '15 minutes ago' --no-pager
curl --fail --max-time 5 -I https://example.com
sudo du -xhd1 /var/logReplace the public example.com sample with the failing endpoint. Du -x avoids other filesystems but still incurs IO. Compare results with that application’s latency.
Practice filesystem commands in a disposable directory
pwd shows location; ls -lah exposes hidden entries, mode, owner and human-readable sizes; stat reports exact bytes, inode and timestamps. cp -a preserves attributes, mv changes a path, grep -n prints matching line numbers, head/tail limit evidence and find limits traversal. The following block writes only inside a newly created TEMP directory. It does not delete files or alter service configuration.
LabDir=$(mktemp -d)
cd -- "$LabDir" || exit 1
pwd
printf 'health=ok\nport=8080\n' > app.conf
ls -lah
stat -- app.conf
cp -a -- app.conf app.conf.before
mv -- app.conf.before app.conf.snapshot
grep -n -- 'port=' app.conf
head -n 2 -- app.conf
tail -n 1 -- app.conf
find . -maxdepth 1 -type f -name '*.conf' -print
tar -czf config-demo.tar.gz -- app.conf
tar -tzf config-demo.tar.gzIllustrative excerpt: grep reports 2:port=8080 and tar lists app.conf. grep exit 1 means no match, while exit 2 means an error. ls mode bits and stat owner identify access controls; a successful read does not prove write permission. cp/mv can overwrite existing destinations, so use fresh paths and examine them first. Leave the demo directory for inspection; do not teach rm -rf as a routine repair.
Read CPU, memory and storage fields correctly
For the earlier vmstat 1 5 command, r is runnable tasks, b blocked tasks, si/so swap transfer, and us/sy/id/wa CPU time categories. Sustained wa plus application latency warrants storage investigation; it does not identify a failed disk. Compare uptime load with processor count using nproc. free available estimates memory obtainable without swapping; low free alone is not exhaustion. df -hT reports mounted filesystem space; df -i reports inode exhaustion. ps %CPU is a process lifetime measure, not the same sampling interval as vmstat.
nproc
findmnt -T /var/log
namei -l /var/log/nginx/error.logIllustrative interpretation: Use% 98 with IUse% 20 indicates byte pressure rather than inode pressure. findmnt identifies the filesystem containing the path. namei shows ownership/mode of every directory component: a denied parent search permission can prevent access even when the final file is readable. These checks are read-only; protected path traversal may need sudo.
Compare snapshots with trends
A command result is a snapshot. Capacity decisions need available memory, disk space and inode use over time alongside application latency. Zabbix or Prometheus node_exporter supply that history. Set disk-space warnings with the growth rate in mind so there is time to act before capacity runs out.
Record results and exit codes
Keep timestamps, commands and results with the incident hypothesis. Interpret nonzero exit codes using the tool’s semantics. Choose reboot after assessing cause and impact, rather than to clear visible errors.
df/du disagreement and network failures
When df reports full storage but du totals are small, inspect open deleted files or data hidden beneath mounts. Installed lsof identifies processes holding deleted files.
sudo lsof +L1
sudo ss -lntp
ip route get 1.1.1.1
resolvectl statusAfter identifying the process holding a deleted file open, use its documented log reopen or reload procedure. If the service is active but its endpoint fails, investigate application dependencies. For Permission denied, inspect the complete path with namei -l and the AppArmor policy rather than checking only the final file's mode.
Resolve the failing layer before touching a service
dig +time=2 +tries=1 app.example.com A
ip route get 203.0.113.20
curl --connect-timeout 3 --max-time 10 --fail --show-error --head https://app.example.com/healthReplace the documentation host/address with an authorized endpoint. dig status NXDOMAIN concerns name existence; timeout concerns resolver response. ip route get prints the chosen via/dev/src and can reveal the wrong VPN/interface. curl exit 6 is resolution, 7 connection, 28 timeout, 22 HTTP failure with --fail and 60 certificate verification. HEAD must be supported by that health endpoint; use its documented request method otherwise. Do not use -k to turn a certificate error into acceptance.
Before and after a correction
Return to the original symptom and repeat the slow or failed request. Diagnostic commands do not change service configuration, though some add load. Preserve configuration versions and ownership before editing. For Disk Full, establish retention and backup status before deleting files, then repeat both df and the application test.
Best Practices
- Version-control configuration and review changes.
- Use least privilege and explicit allow-lists.
- Automate repeatable checks and monitor the expected state.
- Test upgrades and restores before production rollout.
Conclusion
A reliable implementation of Linux Commands for Server Diagnostics is more than a successful first run. Keep the configuration documented, observable, recoverable, and aligned with the team's change-management process.
Frequently Asked Questions
Why can df and du disagree?
Open deleted files, mount layout and traversal scope can account for the difference.
Does high load always indicate a CPU bottleneck?
No. Tasks waiting on IO can also increase load.
