Skip to content

Must-Know Facts

Every topic's Must-Know Facts table in one place, for a fast revision pass. Tables are pulled in from the topic files, so this page never drifts from them.

Foundations

What Is Linux

Fact Value Verify with
Linux is A kernel; a full system adds a userland and becomes a distribution uname -s
Author and year Linus Torvalds, first release 1991 cat /proc/version
Kernel license GPL version 2 only, plus the Linux syscall exception, which keeps user programs outside the GPL dnf repoquery --qf '%{license}' kernel-core
Userland license (bash, coreutils) GPL version 3 or later rpm -q --qf '%{LICENSE}\n' bash
Ancestor design Unix (Bell Labs, 1969); Linux reimplements it, sharing no code man 7 standards
Standards POSIX and the Single UNIX Specification (Linux is not certified) getconf _POSIX_VERSION
Release cadence A new mainline kernel roughly every 9 to 10 weeks uname -r

Kernel vs OS vs Distro

Fact Value Verify with
Kernel Process scheduling, memory, filesystems, networking, drivers uname -r
Userland glibc, bash, coreutils, systemd, package manager ldd --version
Shell A user program that starts other programs; not part of the kernel echo $0
Distribution Kernel + userland + package manager + defaults + support lifecycle cat /etc/os-release
GNU/Linux Name that credits the GNU userland; Alpine and Android use Linux without GNU ls --version
Kernel and distro versions Independent numbers uname -r; grep VERSION_ID /etc/os-release
Containers Bring their own userland and share the host kernel uname -r inside a container
Kernel package kernel / kernel-core (RHEL), linux-image-* (Ubuntu) rpm -q kernel

Distributions

Fact Value Verify with
Red Hat family RHEL, Rocky, AlmaLinux, CentOS Stream, Fedora, Amazon Linux grep ID_LIKE /etc/os-release
Debian family Debian, Ubuntu, Linux Mint grep ID_LIKE /etc/os-release
Package formats .rpm with dnf; .deb with apt rpm --version, dpkg --version
CentOS Stream Upstream of RHEL since 2021, not a rebuild cat /etc/redhat-release
RHEL rebuilds Rocky Linux and AlmaLinux grep ^ID= /etc/os-release
Ubuntu LTS Every two years in April (YY.04), 5 years standard support lsb_release -a
RHEL support 10 years per major release grep SUPPORT_END /etc/os-release
Container minimal images Alpine (musl, BusyBox), distroless, UBI cat /etc/os-release in the image
Architecture names x86_64 = amd64; aarch64 = arm64 uname -m, dpkg --print-architecture

Linux vs Windows

Fact Value Verify with
Privilege model Root (UID 0) versus unprivileged users; daily work runs unprivileged id
Password hashes /etc/shadow, readable by root only ls -l /etc/shadow
Package trust Repositories and packages are GPG-signed and checked on install rpm -q gpg-pubkey
Mandatory access control SELinux (RHEL) or AppArmor (Ubuntu) on top of file permissions cat /sys/kernel/security/lsm
Memory randomization ASLR on by default (2 = full) sysctl kernel.randomize_va_space
Configuration Plain-text files under /etc, no central registry ls /etc
Remote administration SSH and a shell; no GUI required systemctl status sshd
License cost No per-core or per-instance fee for the OS itself Vendor pricing

Architecture

Fact Value Verify with
Layers Hardware, kernel, system calls, C library, shell and applications ldd /usr/bin/ls
Kernel space Runs in CPU privileged mode (ring 0 on x86) with full hardware access grep -c . /proc/kallsyms
User space Runs in ring 3; reaches hardware only through system calls strace -c ls
System call The only entry point from a program into the kernel strace -e trace=openat cat /etc/hostname
C library glibc on RHEL and Ubuntu, musl on Alpine; wraps system calls ldd /usr/bin/ls
Dynamic loader /lib64/ld-linux-x86-64.so.2 loads shared libraries at exec file /usr/bin/ls
vDSO Kernel page mapped into every process for fast calls such as clock_gettime grep vdso /proc/self/maps
Kernel design Monolithic, with loadable modules lsmod
PID 1 systemd, the first user-space process ps -o pid,comm -p 1
PID 2 kthreadd, parent of all kernel threads ps --ppid 2
Error reporting System calls return -1 and set errno (ENOENT, EACCES) strace ls /nonexistent

System Information

Fact Value Verify with
Distribution and version /etc/os-release (standard on every systemd distribution) cat /etc/os-release
Kernel release Separate from the distribution version uname -r
CPU architecture x86_64 or aarch64 uname -m
Logical CPUs Count of schedulable CPUs nproc
Memory that can be used now available column, not free free -h
Block devices and mounts Tree of disks, partitions and mount points lsblk
Uptime and load Time since boot, ⅕/15-minute load averages uptime
Virtualization Hypervisor type, or none on bare metal systemd-detect-virt, hostnamectl
Firmware tables dmidecode reads SMBIOS; absent on some microVMs sudo dmidecode -t system
Page size 4096 bytes on x86_64 getconf PAGE_SIZE

Shell and CLI

Shell Basics

Fact Value Verify with
Terminal Device that carries input and output (/dev/pts/N for SSH, /dev/ttyN for consoles) tty
Shell Program that parses and runs commands (bash, dash, zsh) echo $0
Login shell of a user Field 7 of /etc/passwd getent passwd $USER
Allowed login shells /etc/shells cat /etc/shells
/bin/sh bash on RHEL, dash on Ubuntu readlink -f /bin/sh
Virtual consoles getty@ttyN services; Ctrl+Alt+F2 switches on physical machines systemctl list-units 'getty@*'
Interactive shell $- contains i echo $-
Login shell Started by login, sshd or bash -l; reads profile files shopt login_shell
History file ~/.bash_history, written when the shell exits echo $HISTFILE
Trace a command line set -x prints each command after expansion bash -x script.sh

Getting Help

Fact Value Verify with
Man sections 1 commands, 5 file formats, 8 admin commands, 2 system calls, 3 library calls, 7 overviews man -f intro
Page in a section man 5 passwd for the file, man passwd for the command man -f passwd
Search descriptions man -k or apropos; needs an index built by mandb apropos -s 8 'user account'
Builtin help help <builtin>; builtins have no man page of their own help cd
Quick usage <command> --help ls --help
Package docs /usr/share/doc/<package> rpm -qd sudo
Container images Often ship without docs (tsflags=nodocs, dpkg excludes) grep tsflags /etc/dnf/dnf.conf

Command Resolution

Fact Value Verify with
Lookup order Alias, keyword, function, builtin, hash table, PATH type -a <name>
type vs which type is the shell's own answer; which searches PATH only type echo; which echo
Portable lookup in scripts command -v <name> command -v echo
Bypass an alias \name or command name \ls
Skip aliases and functions command name (builtins still win); a full path always runs the file command ls
Hash table Remembers the path of each external command already run hash -t <name>
Clear the hash hash -r; assigning PATH also clears it hash
Not found Exit code 127 nosuchcmd; echo $?
Found but not executable Exit code 126 echo $?
Current directory Not in PATH; run local files as ./name echo $PATH

Variables and Environment

Fact Value Verify with
Shell variable Visible only in the current shell set
Environment variable Exported; copied to child processes at exec env, printenv
Export export VAR=value or declare -x VAR declare -p VAR
One-command variable VAR=value command sets it for that command only LOG_LEVEL=debug env
Children cannot change parents A child's export never reaches the parent shell bash -c 'export X=1'; echo $X
Process environment Fixed at exec; later export does not change a running process tr '\0' '\n' < /proc/<pid>/environ
PATH Colon-separated directories searched in order; . is not included echo $PATH
Login shell files /etc/profile, /etc/profile.d/*.sh, then the first of ~/.bash_profile, ~/.bash_login, ~/.profile bash -l
Non-login interactive files ~/.bashrc (which sources /etc/bashrc on RHEL) bash -i
Scripts, cron, systemd Read no startup files bash -c env
System-wide static variables /etc/environment, read by pam_env at login cat /etc/environment
sudo Resets the environment and sets PATH from secure_path sudo printenv PATH
Read-only variable readonly VAR=value; cannot be changed or unset readonly -p

Locale and Encoding

Fact Value Verify with
Precedence LC_ALL overrides every LC_*, which override LANG locale
C / POSIX locale Byte order, ASCII, English messages; same result on every machine LC_ALL=C sort
C.UTF-8 Byte-order sorting with UTF-8 characters locale -a
System default /etc/locale.conf (RHEL), /etc/default/locale (Ubuntu) localectl status
Installed locales glibc-langpack-* (RHEL), locale-gen (Ubuntu) locale -a
Encoding of a file Detected, not stored file -i <file>
Convert encoding iconv -f <from> -t <to> iconv -l

Quoting and Expansion

Fact Value Verify with
Single quotes Everything literal; no expansion at all echo '$HOME'
Double quotes Expand $var, $(cmd), $((expr)); prevent word splitting and globbing echo "$HOME"
Backslash Escapes one character echo \$HOME
Expansion order Brace, tilde, parameter/command/arithmetic, word splitting, pathname (glob), quote removal set -x
Globs Expanded by the shell, not by the command echo *
Unmatched glob Left as literal text unless nullglob is set echo *.none
Word splitting Unquoted expansions split on IFS (space, tab, newline) set -x
Always quote "$var", "$(cmd)", "$@" bash -x script.sh
End of options -- stops option parsing, for names starting with - rm -- -rf
Command substitution $(cmd) nests cleanly; backticks are the legacy form echo "$(date)"

Streams and Redirection

Fact Value Verify with
Standard streams 0 stdin, 1 stdout, 2 stderr ls -l /proc/self/fd
> / >> Truncate / append stdout to a file echo x >> f
2> Redirect stderr ls /nope 2> err.txt
> file 2>&1 Both streams to the file; order matters cat file
2>&1 > file Only stdout to the file; stderr stays on the terminal cat file
&> file Bash shorthand for > file 2>&1 cat file
Pipe (vertical bar) Connects stdout of one command to stdin of the next; stderr is not piped ls -l /proc/self/fd
Piping both streams 2>&1 before the bar, or the bash bar-ampersand form set -x
Pipeline status Status of the last command, unless pipefail is set echo ${PIPESTATUS[*]}
/dev/null Discards writes, returns end of file on read ls -l /dev/null
Writing root-owned files Pipe into sudo tee file; sudo cmd > file redirects as the caller sudo cat file
Here-document <<EOF expands variables; <<'EOF' does not cat <<'EOF'
Process substitution <(cmd) presents output as a file path diff <(a) <(b)

Exit Codes and Chaining

Fact Value Verify with
Last status $?, overwritten by the next command echo $?
Success 0 true; echo $?
General failure 1 false; echo $?
Misuse or serious error 2 (ls, grep on a missing file) ls /nope; echo $?
Found but cannot execute 126 /usr/lib/os-release; echo $?
Command not found 127 nosuchcmd; echo $?
Killed by signal N 128 + N: 130 SIGINT, 137 SIGKILL, 141 SIGPIPE, 143 SIGTERM kill -l 9
timeout expired 124 timeout 1 sleep 5; echo $?
Values above 255 Reduced modulo 256 (exit 300 gives 44) bash -c 'exit 300'; echo $?
a && b b runs only if a succeeded mkdir d && cd d
OR list (double bar) The second command runs only if the first failed echo $?
a ; b b always runs echo $?
grep statuses 0 match, 1 no match, 2 error grep -q x f; echo $?

Text Editors

Fact Value Verify with
vi on RHEL vim-minimal; full vim is vim-enhanced rpm -qf /usr/bin/vi
vi on Ubuntu Alternatives link to vim.basic or vim.tiny readlink -f /usr/bin/vi
Default editor on Ubuntu nano, through the editor alternative update-alternatives --display editor
Editor used by tools $VISUAL, then $EDITOR, then a built-in default echo $EDITOR
Edit a root file safely sudoedit <file> (sudo -e); runs the editor as the user sudoedit /etc/hosts
Learn vim vimtutor (30 minutes) vimtutor

Scripting Essentials

Fact Value Verify with
Shebang First line, #!/bin/bash; chooses the interpreter head -1 script.sh
Run a script chmod +x script.sh; ./script.sh, or bash script.sh ls -l script.sh
Arguments $0 name, $1..$9, ${10}, $# count, "$@" all, kept separate ./args.sh a "b c"
Test commands [ ] (POSIX), [[ ]] (bash: patterns, no word splitting) help [[
Numeric comparison -eq -ne -lt -le -gt -ge, or (( a < b )) [ 5 -gt 3 ]
String comparison =, !=, -z (empty), -n (not empty) [ -z "$x" ]
File tests -e exists, -f file, -d directory, -r -w -x, -s not empty [ -d /etc ]
Read a file line by line while IFS= read -r line; do ...; done < file help read
Function variables local keeps them inside the function help local
Syntax check and lint bash -n script.sh; shellcheck script.sh (EPEL, Ubuntu universe) echo $?

Files and Filesystem

Filesystem Hierarchy

Fact Value Verify with
One tree Disks, partitions and network shares are mounted into /; no drive letters findmnt
/etc Host-specific configuration, plain text ls /etc
/var Data that changes: logs, caches, spools, databases ls /var
/var/log Log files ls /var/log
/var/lib Persistent application state (package databases, Docker, databases) ls /var/lib
/usr Installed software, read-only in normal operation du -sh /usr
usrmerge /bin, /sbin, /lib, /lib64 are symlinks into /usr ls -l /bin
/usr/local, /opt Software installed outside the package manager ls /usr/local
/tmp vs /var/tmp Both world-writable with the sticky bit; /var/tmp survives reboots and is cleaned less often, or never ls -ld /tmp /var/tmp
/run tmpfs for runtime data (PID files, sockets); emptied at boot findmnt /run
/proc, /sys Virtual filesystems exposing kernel and device state findmnt /proc
/dev Device files, created by the kernel and udev ls -l /dev/null
/boot Kernel, initramfs, bootloader files ls /boot
/root vs /home Root's home (mode 700) vs regular users' homes ls -ld /root /home
References man 7 hier, man 7 file-hierarchy man -w 7 hier

File Types

Fact Value Verify with
- Regular file ls -l
d Directory ls -ld /etc
l Symbolic link ls -l /bin
c Character device (byte stream: terminals, /dev/null) ls -l /dev/null
b Block device (disks, partitions) ls -l /dev/vda
p Named pipe (FIFO) mkfifo p; ls -l p
s Unix domain socket ls -l /run/systemd/journal/stdout
Device numbers Major (driver) and minor (instance) replace the size column ls -l /dev/vda
Type from content file reads magic bytes; extensions mean nothing to the kernel file <path>
Type as text stat -c %F stat -c %F /dev/null
Search by type find -type f,d,l,c,b,p,s find /dev -type b
Fact Value Verify with
Absolute path Starts at / cd /etc/ssh
Relative path Starts at the current directory; . current, .. parent cd ../log
Home cd, cd ~; another user's home is ~user echo ~root
Previous directory cd - (prints it); $OLDPWD holds it echo $OLDPWD
Logical vs physical path pwd keeps symlink names; pwd -P resolves them pwd -P
Hidden files Names starting with .; shown by ls -a ls -a ~
Newest last ls -ltr ls -ltr /var/log
Largest first ls -lS ls -lhS
Directory itself, not contents ls -ld <dir> ls -ld /tmp

File Operations

Fact Value Verify with
Create parents mkdir -p (no error if the directory exists) mkdir -pv a/b
touch Creates an empty file or updates its timestamps stat -c %y f
cp Copies content; new file gets the copier's owner and the current time ls -l
cp -a Archive: recursive, keeps mode, owner, timestamps, links, xattrs ls -l
cp -r Recursive only; does not keep ownership or times ls -l
mv on one filesystem Rename: same inode, instant ls -i
mv across filesystems Copy, then delete the source ls -i
rm Removes a name (unlinks); no recycle bin ls -l
rmdir Removes empty directories only rmdir d
Sparse file Size is larger than the blocks it uses du -h --apparent-size
install Copy plus mode and ownership in one step install -m 755 src dst
Download that fails on HTTP errors curl -fsSL -o file URL echo $?
Fact Value Verify with
Inode holds Type, mode, UID, GID, size, timestamps, link count, block pointers stat <file>
Inode does not hold The file name ls -i
Directory A table of name to inode number ls -li
Hard link Another name for the same inode; same filesystem only; not for directories ln a b
Symbolic link A small file containing a path; can cross filesystems and point to directories ln -s a b
Link count Number of names for an inode; a new directory starts at 2 stat -c %h <file>
Deletion rm removes a name; data is freed at link count 0 and no open descriptors lsof +L1
Inode exhaustion "No space left on device" with free blocks df -i
mtime Content changed stat -c %y
ctime Inode changed (content, mode, owner, links); cannot be set by touch stat -c %z
atime Last read; relatime updates it at most once a day findmnt -o OPTIONS /
Root inode 2 on ext4 (XFS uses a different number) stat -c %i /

File Descriptors

Fact Value Verify with
Descriptor Index into the process's descriptor table ls -l /proc/<pid>/fd
Allocation Lowest free number; 0, 1, 2 are normally taken strace -e trace=openat
Three layers Descriptor table (per process), open file description (offset, flags), inode (the file) cat /proc/<pid>/fdinfo/<n>
After fork() Child gets copies of the descriptors pointing to the same open file descriptions, so offsets are shared read in a subshell
Separate open() New open file description with its own offset /proc/<pid>/fdinfo
dup2(old, new) Makes new refer to the same description; this is 2>&1 strace -e trace=dup2
O_APPEND Every write goes to the end, atomically flags in fdinfo
O_CLOEXEC Descriptor closes on execve; flag 02000000 flags in fdinfo
Per-process limit ulimit -n / RLIMIT_NOFILE, often soft 1024 grep 'open files' /proc/<pid>/limits
System-wide counters /proc/sys/fs/file-nr, file-max, nr_open cat /proc/sys/fs/file-nr
Limit reached EMFILE: "Too many open files" ls /proc/<pid>/fd
Sockets and pipes Also descriptors; count against the same limit lsof -p <pid>

Finding Files

Fact Value Verify with
Syntax find <paths> <tests> <actions>; default action -print find /etc -name '*.conf'
Quote patterns Otherwise the shell expands them first find . -name '*.log'
Size units -size +100M, -size -10k; + more than, - less than find / -xdev -size +1G
Time tests -mtime +30 older than 30 days; -mmin -60 newer than an hour find /var/log -mtime +30
Permission tests -perm 644 exact, -perm -4000 all bits set, -perm /022 any bit set find / -perm -4000
Owner tests -user, -group, -nouser, -nogroup find / -nouser
Run a command -exec cmd {} \; once per file; -exec cmd {} + batched find . -exec ls {} +
Safe with xargs -print0 and xargs -0 handle spaces and newlines find . -print0
Delete -delete; put it last find /tmp -name '*.tmp' -delete
Stay on one filesystem -xdev find / -xdev
Depth -maxdepth N, -mindepth N find . -maxdepth 1
Name index locate searches the database built by updatedb locate sshd_config
Command location which searches PATH; whereis adds man pages whereis sshd

Archiving and Compression

Fact Value Verify with
Create, list, extract tar -c, -t, -x tar -tf <archive>
-f The next argument is the archive file; without it, tar uses stdin or stdout tar -tzf a.tgz
-C <dir> Change to <dir> before extracting or adding files tar -xf a.tgz -C /opt
Compression flags -z gzip, -j bzip2, -J xz, --zstd zstd file <archive>
Extracting GNU tar detects the compression; the flag is optional tar -xf a.tar.xz
Absolute paths Leading / is stripped on create tar -tf
Permissions Stored always; restored for root by default, -p for others tar -tvf
Compressors Ratio on typical text: xz > zstd > bzip2 > gzip; speed: zstd and gzip fastest, xz slowest ls -l, time
Read compressed files zcat, zless, zgrep; xzcat, bzcat, zstdcat zcat f.gz
Useful extras --exclude='*.log', --strip-components=1, one member by path tar -tf <archive>
zip Archive and compression in one; common with Windows users unzip -l f.zip

Text Processing

Viewing and Comparing

Fact Value Verify with
Page through a file less (/ search, G end, F follow, q quit) less /var/log/messages
First or last lines head -n N, tail -n N tail -n 50 file
Follow a log tail -f follows the open file; tail -F follows the name across rotation tail -F /var/log/app.log
Count wc -l lines, -w words, -c bytes wc -l < file
Hidden characters cat -A shows ^M (CR), ^I (tab), $ (line end) cat -A file
Byte view od -c, xxd, hexdump -C od -c file
Compare files diff (exit 0 same, 1 different, 2 error); diff -u for patches diff -u a b
Binary compare cmp reports the first differing byte cmp a b
Compressed logs zcat, zless, zgrep read .gz without extracting zgrep ERROR app.log.1.gz
Checksums sha256sum; -c verifies a list sha256sum -c SHA256SUMS
Base64 Encoding, not encryption base64 -d
Line ranges sed -n '5,7p' file sed -n '5,7p'

grep and Regex

Fact Value Verify with
Common flags -i ignore case, -v invert, -n line numbers, -c count, -w whole word, -x whole line grep -in error app.log
Output only the match -o grep -o '[0-9]*' f
Context -A N after, -B N before, -C N both grep -C2 ERROR log
Recursive -r (-R follows symlinks); --include, --exclude, --exclude-dir grep -r --include='*.conf' x /etc
File names only -l matching, -L not matching grep -rl TODO .
Quiet test -q; status 0 match, 1 no match, 2 error grep -q x f; echo $?
Regex flavors Basic (default), -E extended, -P Perl, -F fixed string grep -E 'a+'
BRE vs ERE In BRE, +, ?, the bar, parentheses and braces need a backslash to be special grep 'a\+'
Anchors ^ line start, $ line end, \b word boundary grep '^#'
Classes [0-9], [^a-z], [[:space:]], [[:digit:]]; \s, \w in GNU grep grep '[[:digit:]]'
Repetition * 0+, + 1+, ? 0 or 1, {n,m} (ERE) grep -E 'a{2,}'
Dot Any character; \. is a literal dot grep -c '\.'
Binary files Reports "binary file matches"; -a treats them as text grep -a

sed

Fact Value Verify with
Substitute s/old/new/ first match per line; g all; N the Nth; I ignore case (GNU) sed 's/a/b/g'
Default output Every line, edited or not; -n plus p prints only selected lines sed -n '2p'
In-place edit -i rewrites the file; -i.bak keeps a backup sed -i.bak ...
BSD and macOS sed -i '' needs an explicit empty suffix man sed
Addresses Line number, $ last line, /regex/, ranges a,b sed -n '/start/,/end/p'
Delete d sed '/^#/d'
Insert and append i text before, a text after, c text replace the line sed '/x/a y'
Delimiter Any character after s: s#a#b# avoids escaping slashes sed 's#/usr#/opt#'
Whole match & in the replacement sed 's/[0-9]\+/<&>/'
Groups \(...\) in BRE, (...) with -E; \1 in the replacement sed -E 's/(a)(b)/\2\1/'
Several commands -e cmd -e cmd or cmd; cmd sed -e ... -e ...
Symlinks -i replaces a symlink with a regular file unless --follow-symlinks ls -l

awk

Fact Value Verify with
Program shape pattern { action }; a missing pattern matches every line, a missing action prints the line awk '/x/'
Fields $1...$NF; $0 is the whole line awk '{print $1}'
Built-in variables NR line number, NF field count, FS/OFS input and output separators, FNR line number per file awk '{print NR, NF}'
Field separator Whitespace by default; -F: or -F' = ' to change it awk -F: '{print $1}' /etc/passwd
Special blocks BEGIN runs before input, END after awk 'END {print NR}'
Comparisons Numeric when both sides look numeric: $9 >= 500 awk '$9 >= 500'
Regex match $7 ~ /re/, $7 !~ /re/ awk '$9 !~ /^2/'
Accumulate sum += $10 in the body, print in END awk '{s+=$10} END {print s}'
Associative arrays count[$1]++, then for (k in count) awk '{c[$1]++} END {for (k in c) print c[k], k}'
Pass shell values -v name=value awk -v t=1.5 '$NF > t'
Implementations gawk on RHEL; mawk by default on Ubuntu until gawk is installed readlink -f "$(command -v awk)"
CSV with quoted commas gawk --csv (gawk 5.3 and later) awk --csv

Cut, Sort, Uniq and Tr

Fact Value Verify with
cut -d X -f N Field N split on exactly one character; repeated delimiters give empty fields cut -d: -f1 /etc/passwd
cut -c Character positions cut -c1-10
sort default Text order by locale; 10 sorts before 9 sort nums.txt
Numeric sorts -n numbers, -h human sizes (2G), -V versions (v1.10) sort -h
Sort by field -t separator, -k2,2 one field; -k2 means field 2 to end of line sort -t: -k3,3n
Key options An option on a key (-k2n) replaces all global ordering options for that key sort -k2,2nr
uniq Removes adjacent duplicates only, so sort first uniq -c after sort
Counting uniq -c; -d only repeated lines; -u only unique lines uniq -c
sort -u Sort and deduplicate in one step sort -u f
tr Translate, squeeze (-s) or delete (-d) characters; reads stdin only tr 'a-z' 'A-Z'
paste, join, comm Merge lines side by side, join on a key, compare sorted lists comm -12 a b
Decimal math bc or awk; $(( )) is integer only awk 'BEGIN {print 10/3}'

xargs and tee

Fact Value Verify with
xargs default Splits input on whitespace and runs the command with as many arguments as fit xargs echo
One item per command -n 1 xargs -n1
Placeholder -I{} runs once per line and replaces {} xargs -I{} echo {}
Safe names -0 with NUL-separated input (find -print0, grep -Z) xargs -0
Parallel -P N runs up to N commands at once xargs -P4
Empty input GNU xargs runs the command once anyway unless -r xargs -r
Show commands -t prints each command before running it xargs -t
Stop on failure A command exiting 255 aborts xargs, which then exits 124 echo $?
Argument limit ARG_MAX bytes; xargs splits long lists automatically getconf ARG_MAX
tee file Writes to the file and to stdout cat f
tee -a Appends tee -a log
Root-owned files sudo tee file, because the shell performs > as the caller ls -l file

JSON and YAML on the CLI

Fact Value Verify with
jq . Pretty-prints and validates JSON jq . file.json
Raw strings -r drops the quotes, for use in shell variables jq -r .name
Iterate arrays .items[] emits each element jq '.items[]'
Filter select(.state == "running") jq 'select(...)'
Build output {id: .Id}, [...], @tsv, @csv jq -c '{id: .Id}'
Shell values --arg name value (string), --argjson (JSON) jq --arg s x
Missing keys Return null, not an error jq '.nope'
Fail on null or false -e sets exit status 1 jq -e .key
yq (mikefarah) jq-like syntax for YAML, JSON, XML; -i edits in place yq --version
Two different yq tools Go mikefarah/yq (EPEL, GitHub releases) and Python kislyuk/yq (Ubuntu apt install yq) apt-cache show yq
Convert formats yq -o=json, yq -p json -o yaml yq -o=json file.yaml

Users and Access

Users

Fact Value Verify with
Superuser UID 0 (any account with UID 0 is root) awk -F: '$3 == 0' /etc/passwd
Account database /etc/passwd, mode 644 (world-readable) ls -l /etc/passwd
Password hashes /etc/shadow, mode 000 (root only) ls -l /etc/shadow
/etc/passwd fields 7: name, x, UID, GID, GECOS, home, shell getent passwd root
Regular UID range 1000 to 60000 grep '^UID_M' /etc/login.defs
System UID range RHEL 10: 201 to 999; Ubuntu: 100 to 999 grep SYS_UID /etc/login.defs
Unprivileged overflow user nobody, UID 65534 id nobody
Account defaults /etc/login.defs and useradd -D useradd -D
Home directory template /etc/skel ls -la /etc/skel
Shell that refuses logins /usr/sbin/nologin su - <service-user>
Lookup through NSS getent passwd <name or UID> getent passwd 0
Consistency check pwck -r (read-only) pwck -r

Groups

Fact Value Verify with
Primary group GID in field 4 of /etc/passwd id -gn <user>
Supplementary groups Member list in field 4 of /etc/group id -Gn <user>
/etc/group fields 4: name, x, GID, member list getent group wheel
Group passwords and admins /etc/gshadow, root only sudo grep <group> /etc/gshadow
User private groups Each new user gets a group of the same name (USERGROUPS_ENAB yes) grep USERGROUPS_ENAB /etc/login.defs
RHEL admin group wheel, GID 10 getent group wheel
Ubuntu admin group sudo, GID 27 getent group sudo
New files get The creator's primary (effective) group touch f; ls -l f
Membership takes effect At the next login or newgrp, not in running sessions grep Groups /proc/$$/status
Append a group usermod -aG <group> <user> (-G alone replaces the list) id <user>

Passwords and Aging

Fact Value Verify with
/etc/shadow fields 9: name, hash, last change, min, max, warn, inactive, expire, reserved sudo getent shadow <user>
Date fields Days since 1970-01-01 sudo chage -l <user>
Hash prefix $y$ yescrypt (default on RHEL 10 and Ubuntu 24.04) sudo grep <user> /etc/shadow
Hash prefix $6$ SHA-512 crypt man 5 crypt
Leading ! on the hash Password locked; the hash is kept sudo passwd -S <user>
passwd -S status P usable, L locked, NP no password sudo passwd -S <user>
Force change at next login chage -d 0 <user> sudo chage -l <user>
Account expiry chage -E YYYY-MM-DD (whole account) sudo chage -l <user>
Password expiry chage -M <days> (password only) sudo chage -l <user>
Defaults for new accounts PASS_MAX_DAYS, PASS_MIN_DAYS, PASS_WARN_AGE in /etc/login.defs grep ^PASS_ /etc/login.defs
Portable scripted password chpasswd (passwd --stdin is RHEL only) sudo chpasswd < <file>

Sudo and Su

Fact Value Verify with
Policy file /etc/sudoers, mode 0440, edited only with visudo ls -l /etc/sudoers
Drop-in directory /etc/sudoers.d/ (root-owned, mode 0440; names containing . or ending in ~ are skipped) ls -l /etc/sudoers.d
Rule syntax who where=(as_whom) what sudo grep -E '^%' /etc/sudoers
Admin group %wheel on RHEL, %sudo on Ubuntu id -Gn
Syntax check visudo -c (all files) or visudo -cf <file> sudo visudo -c
What a user may run sudo -l (self) or sudo -l -U <user> sudo -l -U <user>
Password asked for The caller's own password, not root's sudo -k whoami
Credential cache 5 minutes by default, per terminal ls /run/sudo/ts/
Environment Reset (env_reset) and PATH replaced by secure_path sudo env
Mechanism sudo, su and passwd are SUID-root binaries ls -l /usr/bin/sudo
Audit trail journald (tag sudo); /var/log/secure (RHEL) or /var/log/auth.log (Ubuntu) when rsyslog runs journalctl -t sudo

PAM

Fact Value Verify with
Per-service stacks /etc/pam.d/<service> ls /etc/pam.d
Module types auth, account, password, session cat /etc/pam.d/su
Control flags required, requisite, sufficient, optional, include, substack man 5 pam.conf
RHEL stack manager authselect (do not edit system-auth by hand) authselect current
Ubuntu stack manager pam-auth-update (profiles in /usr/share/pam-configs) ls /usr/share/pam-configs
Failed-login lockout pam_faillock, default deny=3 sudo faillock --user <user>
Password quality pam_pwquality, /etc/security/pwquality.conf grep pwquality /etc/pam.d/*

Login Sessions

Fact Value Verify with
Current sessions /run/utmp (read by who, w) who
Login history and reboots /var/log/wtmp (read by last) last -n 5
Failed logins /var/log/btmp, mode 660, root and group utmp only (read by lastb) sudo lastb -n 5
Last login per user /var/log/lastlog, a sparse file indexed by UID lastlog -u <user>
Load and activity w prints uptime, load and each user's current command w
systemd view loginctl list-sessions loginctl

Centralized Identity

Fact Value Verify with
Lookup order /etc/nsswitch.conf (passwd:, group:, shadow: lines) grep ^passwd: /etc/nsswitch.conf
Query through NSS getent passwd <user>; getent -s files limits the source getent -s files passwd root
Directory client SSSD (sssd service, /etc/sssd/sssd.conf, mode 600) systemctl status sssd
Join a domain realm join <domain> (realmd) realm list
Kerberos ticket kinit <user>, klist klist
Create homes on first login pam_mkhomedir (oddjob-mkhomedir on RHEL) grep mkhomedir /etc/pam.d/*

Permissions

Basic Permissions

Fact Value Verify with
Permission string Type, then rwx for user, group, other ls -l
Octal values r 4, w 2, x 1; 640 = rw-r----- stat -c '%a %A' f
File r, w, x Read content, change content, run as a program cat, echo >>, ./f
Directory r List names ls dir
Directory w Create, delete and rename entries (with x) touch dir/f
Directory x Enter and reach entries by name cd dir
Deleting a file Needs w and x on the directory, not on the file rm dir/f
Check order Owner match, else group match, else other; only the first match applies ls -l
Root Bypasses read and write checks; needs at least one x bit to execute sudo cat f
chown Only root changes the owner sudo chown user:group f
chgrp Owner may change to a group they belong to chgrp devs f
Capital X Execute only for directories and already-executable files chmod -R u=rwX,go=rX dir
Every parent matters Each directory on the path needs x namei -l /path/to/file

umask

Fact Value Verify with
Starting modes Programs usually request 666 for files and 777 for directories strace -e trace=openat touch f
Rule Final mode = requested mode with the umask bits removed (bitwise, not subtraction) umask 0123; touch f
Execute on files Never added by the umask; files from touch get no x ls -l
Scope Per process, inherited by children grep Umask /proc/$$/status
Show umask (octal), umask -S (symbolic) umask -S
Rocky 10.2 default 0022 for root and regular users su - user -c umask
Ubuntu 24.04 default 0022 for root, 0002 for regular users (pam_umask with user private groups) su - user -c umask
System setting UMASK in /etc/login.defs, applied by pam_umask at login grep UMASK /etc/login.defs
Per user umask line in ~/.bashrc or ~/.profile umask
Services UMask= in the unit; PID 1 runs with 0000 systemctl show <unit> -p UMask

Special Permissions

Fact Value Verify with
SUID (4000) Executable runs with the file owner's effective UID ls -l /usr/bin/passwd
SGID on a file (2000) Executable runs with the file group's effective GID ls -l /usr/bin/write
SGID on a directory New files and subdirectories inherit the directory's group; subdirectories also get SGID ls -ld /srv/team
Sticky bit (1000) In a writable directory, only the file owner, directory owner or root may delete or rename a file ls -ld /tmp
Display s in the user or group execute position, t in the other position ls -l
Capital S or T The special bit is set but the execute bit under it is not chmod 4644 f
Octal Fourth digit in front: 4755, 2770, 1777 stat -c %a f
Real vs effective UID Real is who started the process; effective is used for permission checks /proc/<pid>/status
Scripts Linux ignores SUID and SGID on interpreted scripts ls -l script
chown Clears SUID and SGID on the file ls -l
nosuid mount option Disables SUID and SGID for the whole filesystem findmnt -no OPTIONS <mount>
Audit find / -perm -4000 and -perm -2000 sudo find / -xdev -perm /6000

ACL

Fact Value Verify with
Show getfacl <path> getfacl f
Add or change setfacl -m u:<user>:<perms> or g:<group>:<perms> setfacl -m u:amor:r f
Remove setfacl -x u:<user> (one entry), setfacl -b (all) setfacl -x u:amor f
Marker in ls -l A + after the mode ls -l
Mask Upper limit for named users, named groups and the owning group getfacl f
chmod on group bits Changes the mask, not the owning group entry, once an ACL exists chmod 600 f; getfacl f
Default ACL setfacl -d -m ... on a directory; inherited by new files and subdirectories getfacl dir
Recursive setfacl -R; capital X adds execute to directories only setfacl -R -m g:ops:rX dir
Backup and restore getfacl -R dir > file; setfacl --restore=file getfacl -R
Copies cp drops ACLs; cp -a, rsync -A, tar --acls keep them ls -l

File Attributes

Fact Value Verify with
Show lsattr <file>; lsattr -d <dir> for the directory itself lsattr f
Set and clear chattr +i, chattr -i (root or CAP_LINUX_IMMUTABLE only) sudo chattr +i f
i immutable No write, delete, rename, link or chmod, even for root lsattr f
a append-only Opens for append only; no truncate, overwrite or delete lsattr log
i on a directory No entries created, removed or renamed; existing files stay writable lsattr -d dir
e Extents in use (ext4); informational lsattr
Error text Operation not permitted (EPERM), not Permission denied strace
Package e2fsprogs; works on ext4, XFS and btrfs, not on /proc rpm -qf /usr/bin/chattr
Not shown by ls -l Mode bits look normal ls -l

Package Management

Packaging Concepts

Fact Value Verify with
RPM file name name-version-release.arch.rpm (NEVRA with an optional epoch) rpm -qp --qf '%{NEVRA}' f.rpm
DEB file name name_version-revision_arch.deb dpkg -I f.deb
Epoch Integer that overrides version comparison (2:1.26.3) rpm -q --qf '%{EPOCH}'
Release or revision Packaging build number; el10_2 and ubuntu13.18 mark the distribution rpm -q bash
Architectures x86_64/amd64, aarch64/arm64, noarch/all uname -m
Low-level vs high-level rpm, dpkg handle files; dnf, apt add repositories and dependency resolution dnf history
Dependencies Declared as package names, files or library sonames (libc.so.6()(64bit)) rpm -q --requires
Scriptlets Pre and post install and remove scripts run as root rpm -q --scripts
Signatures Packages or repository metadata are signed with GPG keys; managers refuse unsigned or unknown-key content rpm -K f.rpm
Trusted keys RPM: gpg-pubkey pseudo-packages; APT: keyring files referenced by Signed-By rpm -q gpg-pubkey
Configuration files Tracked specially; upgrades keep local changes (.rpmnew, dpkg prompt) rpm -qc, dpkg -s
Source packages .src.rpm and .dsc build binary packages rpm -qi (Source RPM)

rpm and dnf

Fact Value Verify with
Is it installed rpm -q <pkg> rpm -q bash
Package details rpm -qi rpm -qi openssh-server
Files, configs, docs rpm -ql, -qc, -qd rpm -qc openssh-server
Which package owns a file rpm -qf <path> rpm -qf /usr/bin/ls
Which package would provide a file dnf provides '*/<name>' dnf provides /usr/bin/dig
Verify installed files rpm -V (size, mode, digest, owner, mtime flags) sudo rpm -V openssh-server
Check a downloaded package rpm -K (signatures), rpm -qpl (files) rpm -K pkg.rpm
Install, remove, update dnf install, dnf remove, dnf upgrade dnf history
Undo a transaction dnf history undo <id> or last dnf history list
Updates available dnf check-update exits 100 echo $?
Security errata dnf updateinfo, dnf upgrade --security dnf updateinfo summary
Pin a version dnf versionlock add <pkg> (plugin) or exclude= in dnf.conf dnf versionlock list
Parallel kernels installonly_limit=3 keeps three grep installonly /etc/dnf/dnf.conf
Scriptlets and other queries rpm -q --scripts, --whatrequires; rpm -qp... on a package file rpm -q --scripts <pkg>
Backported fixes Changelog lists CVEs fixed in an old upstream version rpm -q --changelog <pkg>

dpkg and apt

Fact Value Verify with
Installed packages dpkg -l (ii = installed) dpkg -l bash
Status and dependencies dpkg -s <pkg> dpkg -s openssh-server
Files of a package dpkg -L <pkg> dpkg -L openssh-server
Owner of a file dpkg -S <path> dpkg -S /usr/sbin/sshd
Package for a file not installed apt-file search (package apt-file) apt-file search -x '/usr/bin/dig$'
Refresh indexes apt update; install and upgrade never refresh by themselves apt update
Remove vs purge remove keeps configuration files (rc); purge deletes them dpkg -l <pkg>
Unused dependencies apt autoremove apt autoremove --dry-run
Candidate version and origin apt-cache policy <pkg> apt-cache policy nginx
Hold a version apt-mark hold <pkg> apt-mark showhold
Verify files dpkg --verify, debsums -c (-e for config files) sudo debsums -ce openssh-client
History /var/log/apt/history.log, /var/log/dpkg.log tail /var/log/apt/history.log
Scripts Use apt-get; apt warns that its CLI is not stable and waits on locks apt-get -y install

Repositories

Fact Value Verify with
RHEL definitions /etc/yum.repos.d/*.repo (baseurl or mirrorlist, gpgcheck, gpgkey); a local repository is a createrepo_c directory with baseurl=file:///<dir> dnf repolist --all
Ubuntu definitions /etc/apt/sources.list.d/*.sources (deb822: Types, URIs, Suites, Components, Signed-By; default on 24.04) or *.list cat /etc/apt/sources.list.d/ubuntu.sources
Keys RPM imports into its database; APT uses keyring files in /etc/apt/keyrings/ via Signed-By rpm -q gpg-pubkey
Extra RHEL repositories CRB (build dependencies) and EPEL (community packages), enabled with dnf config-manager --set-enabled <id>; Red Hat repositories need subscription-manager register (not on Rocky) dnf repolist
Unreachable repository dnf stops with Failed to download metadata; apt update warns and continues dnf makecache

Flatpak and Snap

Fact Value Verify with
Flatpak remotes Sources such as Flathub; system-wide or per user flatpak remotes
Install, update, remove flatpak remote-add, install, update, uninstall; apps pull shared runtimes flatpak list
Scope --system (default for root, /var/lib/flatpak) or --user (~/.local/share/flatpak) flatpak list --columns=installation
Clean up flatpak uninstall --unused removes orphaned runtimes du -sh /var/lib/flatpak
Snap Squashfs images mounted under /snap, managed by snapd snap list
Snap updates Automatic by default; snap refresh --hold pauses them snap refresh --list

Shared Libraries

Fact Value Verify with
Loader /lib64/ld-linux-x86-64.so.2, named in the ELF header file <binary>
Needed libraries NEEDED entries in the dynamic section readelf -d <binary>
Resolved paths ldd shows where each library is found ldd <binary>
Search order LD_LIBRARY_PATH, the binary's RUNPATH, /etc/ld.so.cache, then the default directories; a legacy RPATH is searched before LD_LIBRARY_PATH LD_DEBUG=libs <binary>
Cache ldconfig builds /etc/ld.so.cache from /etc/ld.so.conf.d/*.conf ldconfig -p
soname Name a library promises to stay compatible under (libgreet.so.1) readelf -d lib.so
Symlinks libx.so.1.0 real file; libx.so.1 for programs (soname); libx.so for the compiler (-dev or -devel packages) ls -l
RUNPATH Directories stored in the binary; $ORIGIN means the binary's own directory patchelf --set-rpath
LD_PRELOAD Loads a library first, overriding symbols; ignored for SUID programs LD_PRELOAD=... cmd
Symbol versions GLIBC_2.34 in a binary sets the minimum glibc objdump -T <binary>
Static binaries Contain their libraries; ldd prints not a dynamic executable ldd <binary>

Other Install Methods

Fact Value Verify with
Release binaries Single files in /usr/local/bin (install -m 0755 after a checksum check); bundles in /opt/<app> command -v kubectl
Source builds ./configure && make && sudo make install, default prefix /usr/local ./configure --help
Untracked files Nothing in /usr/local belongs to a package dpkg -S, rpm -qf
Alternatives One command name, several implementations: update-alternatives (Debian), alternatives (RHEL); --display, --config, --set update-alternatives --query editor
Python on Ubuntu 24.04 System pip install refused (PEP 668, EXTERNALLY-MANAGED); use a venv or pipx ls /usr/lib/python3*/EXTERNALLY-MANAGED
Python on Rocky 10.2 No EXTERNALLY-MANAGED marker; pip install --user works, root pip warns pip3 --version
Unpack an RPM without installing rpm2cpio output piped into cpio -idm; dpkg-deb -x for a .deb find . -type f

Processes

Process Fundamentals

Fact Value Verify with
Program vs process A program is a file on disk; a process is a running instance with its own PID and memory ps -C sleep
PID Unique number for a running process; reused after the process is reaped echo $$
PPID PID of the parent that created the process ps -o pid,ppid -p $$
PID 1 systemd on RHEL and Ubuntu; adopts orphans and cannot be killed by a stray signal ps -p 1 -o comm
PID 2 kthreadd, parent of every kernel thread (shown in [brackets]) ps -o ppid,comm --ppid 2
PID limit kernel.pid_max, 4194304 on 64-bit systems with systemd cat /proc/sys/kernel/pid_max
Thread A task that shares memory and file descriptors with its process; has its own TID ps -T -p <pid>
Thread count NLWP column, Threads: in /proc/<pid>/status ps -o nlwp -p <pid>
$$ vs $BASHPID $$ stays the script's PID in subshells; $BASHPID is the current process ( echo $$ $BASHPID )
/proc/<pid>/ Kernel view of one process: status, cmdline, exe, cwd, fd/, environ, maps ls /proc/$$
Other users' processes cwd, exe, fd/ and environ are readable only by the owner or root ls -l /proc/1/cwd
Process tree pstree -p, ps -ef --forest, systemd-cgls pstree -p 1
Process credentials Real, effective, saved and filesystem UID and GID, stored per process grep Uid /proc/$$/status

Process Lifecycle

Fact Value Verify with
fork() Creates a copy of the calling process; returns the child PID to the parent and 0 to the child strace -f -e trace=clone bash -c 'ls; true'
execve() Replaces the program in the current process; the PID stays the same bash -c 'echo $$; exec bash -c "echo \$\$"'
Copy-on-write Parent and child share memory pages until one writes; fork() copies page tables only /proc/<pid>/smaps_rollup
exit() Frees memory and files; the process becomes a zombie holding its exit status ps -o stat shows Z
wait() Parent reads the exit status and the kernel removes the zombie (reaping) strace -e trace=wait4
SIGCHLD Sent to the parent when a child exits or stops strace shows --- SIGCHLD
Zombie Exited, not yet reaped; uses no memory, only a process table entry ps -o pid,ppid,stat
Orphan Parent exited first; adopted by PID 1 or the nearest subreaper ps -o ppid shows 1
Exit status 0 to 255; 128+N means killed by signal N echo $?
Process group Jobs of one pipeline; signals from the terminal go to the foreground group ps -o pgid
Session Process groups under one login; the leader owns the controlling terminal ps -o sid,tty
Daemon Background process with no controlling terminal (TTY ?), usually managed by systemd ps -o tty -p <pid>
setsid Starts a program in a new session with no terminal setsid sleep 60

Viewing Processes

Fact Value Verify with
ps aux BSD syntax: all processes, user-oriented columns, %CPU and %MEM ps aux
ps -ef UNIX syntax: all processes with PPID and full command ps -ef
Custom columns -o pid,ppid,user,stat,%cpu,rss,etime,cmd; = after a name removes the header ps -o pid= -p 1
Sorting --sort=-%cpu, --sort=-rss ps -eo pid,rss,comm --sort=-rss
Select by name -C nginx matches the command name exactly ps -C nginx -o pid,args
Tree view ps -ef --forest, pstree -p pstree -p <pid>
%CPU in ps CPU time divided by lifetime, not current usage top for current usage
RSS vs VSZ Resident memory in KiB vs virtual address space size ps -o rss,vsz
pgrep / pkill Match the name (15 chars); -f matches the full command line; exit 1 when nothing matches pgrep -a nginx
pidof PIDs of an exact program name, space-separated pidof nginx
top batch mode top -b -n 1 prints one screen for scripts and tickets top -b -n 1
top keys P CPU, M memory, 1 per-CPU, H threads, c full command, k kill inside top
Load average Runnable plus uninterruptible (D) tasks, averaged over 1, 5 and 15 minutes uptime
watch Reruns a command every 2 seconds (-n to change, -d to highlight changes) watch -n 1 'ps -C nginx'

Process States

Fact Value Verify with
R Running or runnable (waiting in the CPU run queue) ps -o stat -p <pid>
S Interruptible sleep: waiting for an event, wakes on a signal ps -o stat,wchan
D Uninterruptible sleep: waiting inside the kernel, usually for I/O; signals wait ps -eo stat,pid,wchan
T Stopped by SIGSTOP or SIGTSTP (Ctrl-Z) kill -STOP <pid>
t Stopped by a debugger or tracer gdb -p <pid>
Z Zombie: exited, waiting for the parent to reap it ps -o stat,ppid
I Idle kernel thread (not counted in load average) ps -eo stat,comm
X Dead; never seen in practice man ps
Flags s session leader, l multithreaded, + foreground group, < high priority, N low priority ps aux
Load average Counts R and D tasks cat /proc/loadavg
wchan Kernel function where a sleeping process waits ps -o wchan:30 -p <pid>
Kernel stack Full blocked call path (root only) sudo cat /proc/<pid>/stack
SIGKILL and D The signal stays pending until the kernel call returns grep ShdPnd /proc/<pid>/status

Signals

Fact Value Verify with
SIGHUP (1) Terminal hung up; daemons use it as "reload configuration" kill -HUP <pid>
SIGINT (2) Ctrl-C kill -INT <pid>
SIGQUIT (3) Ctrl-\; terminate with a core dump kill -QUIT <pid>
SIGKILL (9) Terminate immediately; cannot be caught, blocked or ignored kill -9 <pid>
SIGTERM (15) Polite termination; the default of kill, pkill and systemctl stop kill <pid>
SIGSTOP (19) / SIGCONT (18) Stop (uncatchable) and resume kill -STOP <pid>
SIGTSTP (20) Ctrl-Z; a catchable stop kill -TSTP <pid>
SIGCHLD (17) A child exited or stopped strace -e signal=SIGCHLD
SIGSEGV (11) / SIGPIPE (13) Invalid memory access / write to a pipe with no reader dmesg, exit code 141
SIGUSR1 (10), SIGUSR2 (12) Application-defined (log reopen, debug toggle) man 7 signal
Exit code 128 + signal number: 130 INT, 137 KILL, 141 PIPE, 143 TERM kill -l 137
kill -0 Sends nothing; tests existence and permission kill -0 <pid>; echo $?
Permission A user can signal only processes with the same real or effective UID; root can signal any kill 1 as a user
Masks SigBlk, SigIgn, SigCgt, ShdPnd in /proc/<pid>/status (bit N-1 = signal N) grep Sig /proc/<pid>/status
trap Shell handler: trap 'cmd' TERM; trap '' INT ignores trap -p

Job Control

Fact Value Verify with
cmd & Runs in the background; the shell prints [job] PID sleep 60 &
jobs -l Lists the shell's jobs with PIDs; + is current, - is previous jobs -l
Ctrl-Z Sends SIGTSTP; the job stops jobs shows Stopped
bg %n / fg %n Continue a job in the background / bring it to the foreground bg %1
Ctrl-C Sends SIGINT to the foreground process group; exit status 130 echo $?
Ctrl-D End of input (EOF), not a signal; closes the shell at an empty prompt cat then Ctrl-D
Job specs %1, %+, %-, %sleep (name prefix) kill %1
Terminal hangup The session leader gets SIGHUP; bash sends SIGHUP to all its jobs close the terminal
nohup Ignores SIGHUP; writes output to nohup.out if stdout is a terminal nohup cmd &
disown Removes a job from the shell's table, so bash does not send it SIGHUP disown %1
setsid Starts a command in a new session with no terminal setsid cmd
huponexit If set, bash sends SIGHUP to jobs on a normal exit too (off by default) shopt huponexit
tmux / screen Keep a whole terminal session alive on the server across disconnects tmux attach

Priority and Nice

Fact Value Verify with
Nice range -20 (highest priority) to 19 (lowest); default 0 nice
Start with a nice value nice -n 10 cmd; the default increment is 10 nice -n 10 nice
Change a running process renice -n 5 -p <pid>; also -u user, -g pgrp ps -o ni -p <pid>
Unprivileged users May only raise the nice value (lower priority) of their own processes renice -n 2 after 5 fails
Negative nice Needs root, CAP_SYS_NICE, or a nice limit in limits.conf ulimit -e
Weight Each nice step changes CPU share by about 10% (weight ratio 1.25) two CPU hogs on one core
PR in top 20 + nice for normal tasks; rt or negative for real-time top
I/O class ionice -c 1 realtime, -c 2 -n 0..7 best-effort, -c 3 idle ionice -p <pid>
I/O priority effect Honored by the BFQ scheduler; mq-deadline and none mostly ignore it cat /sys/block/<dev>/queue/scheduler
Real-time policies SCHED_FIFO and SCHED_RR, priority 1 to 99, root only chrt -p <pid>
Scheduler CFS up to Linux 6.5; EEVDF from 6.6 (RHEL 10 ships 6.12) uname -r
Autogroup When enabled, the scheduler groups tasks by session, so nice works only within a session cat /proc/sys/kernel/sched_autogroup_enabled
systemd Nice=, CPUWeight=, IOSchedulingClass=, IOWeight= in a unit systemctl show -p Nice <unit>

System Calls and Tracing

Fact Value Verify with
System call Controlled entry from user mode into the kernel (the syscall instruction on x86_64) strace true
Return value -1 plus errno on failure; strace prints the symbolic name strace cat /nonexistent
Common errno ENOENT 2, EACCES 13, EPERM 1, EAGAIN 11, EMFILE 24, ENOSPC 28, ECONNREFUSED 111 python3 -c 'import os; print(os.strerror(13))'
EACCES vs EPERM Permission bits or ACL denied vs operation not allowed for this identity (ownership, capability) strace output
vDSO Kernel code mapped into every process so time calls avoid a real system call grep vdso /proc/self/maps
strace -f Follow children and threads strace -f bash -c 'ls; true'
strace -e trace= Filter by name or class: %file, %network, %process, %memory strace -e trace=%file
strace -p <pid> Attach to a running process; needs the same user or root timeout 5 strace -p <pid>
strace -c Count calls, errors and time per call strace -c ls
Useful flags -o file, -tt timestamps, -T time in call, -y file names for fds, -s 200 longer strings, -P path man strace
Overhead strace stops the target on every traced call; it can slow a busy service many times over strace -c on a hot path
ltrace Traces calls into shared libraries (malloc, getenv) ltrace -c ls
Stack of a hung process /proc/<pid>/wchan, /proc/<pid>/stack (root), gdb -p or gstack for user space sudo cat /proc/<pid>/stack
Ptrace restrictions Yama kernel.yama.ptrace_scope=1 on Ubuntu allows tracing only of children without root sysctl kernel.yama.ptrace_scope

Systemd and Services

Init and Targets

Fact Value Verify with
Init on RHEL and Ubuntu systemd; /sbin/init is a symlink to it ls -l /sbin/init
SysV init Ran /etc/rc.d/rcN.d scripts one after another; replaced in RHEL 7 and Ubuntu 15.04 man systemd-sysv-generator
Target A unit that groups other units into a system state systemctl list-units --type=target
Default target /etc/systemd/system/default.target symlink systemctl get-default
Server default multi-user.target (runlevel 3); desktops use graphical.target (runlevel 5) systemctl get-default
Change the default systemctl set-default multi-user.target (takes effect at next boot) ls -l /etc/systemd/system/default.target
Switch now systemctl isolate multi-user.target; only units with AllowIsolate=yes systemctl show -p AllowIsolate <target>
Rescue / emergency rescue.target (local filesystems, root shell) / emergency.target (root filesystem only, no services) systemctl cat rescue.target
Boot into a target once Kernel parameter systemd.unit=rescue.target cat /proc/cmdline
Current runlevel runlevel, who -r (compatibility) runlevel
Power commands poweroff, reboot, halt, shutdown are symlinks to systemctl ls -l /usr/sbin/reboot
Scheduled shutdown shutdown -r +30 "msg"; shutdown -c cancels; --show lists it shutdown --show
System health systemctl is-system-running: running, degraded (a unit failed), starting systemctl --failed
Boot time systemd-analyze systemd-analyze blame

systemctl

Fact Value Verify with
start / stop / restart Change the running state now systemctl is-active <unit>
reload Ask the service to reread its configuration (ExecReload=); the main PID stays systemctl show -p MainPID
enable / disable Create or remove [Install] symlinks; affects the next boot only systemctl is-enabled <unit>
enable --now Enable and start in one command systemctl status <unit>
mask / unmask Link the unit to /dev/null so nothing can start it systemctl is-enabled shows masked
is-active exit codes 0 active, 3 inactive or failed systemctl is-active <unit>; echo $?
status exit codes 0 active, 3 not running, 4 no such unit systemctl status <unit>; echo $?
Enablement states enabled, disabled, static (no [Install]), masked, alias, indirect systemctl list-unit-files
Preset Vendor default for enable: RHEL disables most new services, Ubuntu enables and starts them on install systemctl list-unit-files <unit>
Failed units systemctl --failed; clear with systemctl reset-failed systemctl is-failed <unit>
Reading config systemctl cat <unit> prints the unit and its drop-ins systemctl cat sshd
Properties systemctl show <unit> -p MainPID -p Restart -p NRestarts systemctl show <unit>
After editing units systemctl daemon-reload Warning "changed on disk"
Without root Read commands work; changes need sudo ("Interactive authentication required") systemctl start <unit> as a user
User units systemctl --user manages the per-user manager systemctl --user status

Unit Files

Fact Value Verify with
Unit types .service, .socket, .timer, .target, .mount, .automount, .path, .slice, .scope, .device, .swap systemctl -t help
Sections [Unit] (description, dependencies), [Service] / [Socket] / ... (type-specific), [Install] (enable) systemctl cat <unit>
Search order /etc/systemd/system overrides /run/systemd/system, which overrides /usr/lib/systemd/system systemd-analyze unit-paths
Vendor units /usr/lib/systemd/system (packages); never edit them systemctl show -p FragmentPath
Drop-ins /etc/systemd/system/<unit>.d/*.conf change single settings systemctl show -p DropInPaths
systemctl edit <unit> Creates override.conf and reloads; --full copies the whole unit systemctl cat <unit>
List-valued settings ExecStart= with an empty value resets the list before a new value systemctl cat <unit>
daemon-reload Required after any file change; otherwise systemd warns "changed on disk" systemctl status <unit>
Type= simple (default), exec, forking, oneshot, notify, dbus, idle systemctl show -p Type <unit>
Restart= no (default), on-failure, always, on-abnormal; RestartSec= delay systemctl show -p NRestarts <unit>
Start limit StartLimitBurst=5 in StartLimitIntervalSec=10s by default, then "Start request repeated too quickly" systemctl reset-failed <unit>
Wants= / Requires= Pull in another unit; Requires= also fails or stops with it systemctl list-dependencies <unit>
After= / Before= Ordering only; without them, units start in parallel systemctl show -p After <unit>
network-online.target Needs Wants= and After= to wait for configured networking systemctl show -p WantedBy network-online.target
Check a file systemd-analyze verify <file> systemd-analyze verify <file>
Overrides report systemd-delta lists overridden and extended units systemd-delta --type=extended

Writing a Service

Fact Value Verify with
Location /etc/systemd/system/<name>.service systemctl cat <name>
Minimum [Service] with ExecStart= and an absolute path; [Install] to enable it systemd-analyze verify <file>
Foreground The program must not daemonize under Type=exec or simple systemctl status
Logging stdout and stderr go to the journal journalctl -u <name>
Identity User=, Group=; DynamicUser=yes creates a transient user ps -o user -p <pid>
Configuration Environment=KEY=value, EnvironmentFile=/etc/<name>/<name>.env (- prefix makes it optional) systemctl show -p Environment
Writable state StateDirectory=, RuntimeDirectory=, LogsDirectory=, created and owned for the service user ls -l /var/lib/<name>
Sandboxing ProtectSystem=strict, ProtectHome=yes, PrivateTmp=yes, NoNewPrivileges=yes systemd-analyze security <name>
Resource limits MemoryMax=, CPUQuota=, TasksMax=, LimitNOFILE= systemctl show -p MemoryMax
Exposure score 0 (sandboxed) to 10 (unrestricted) systemd-analyze security
Exit code 203 EXEC: the binary could not be executed systemctl status
Test a setting systemd-run -P -p <Setting>=<value> <cmd> runs a command as a transient unit systemd-run -P -q true

Systemd Toolbox

Fact Value Verify with
Boot time systemd-analyze, blame, critical-chain <unit> systemd-analyze blame
Calendar syntax check systemd-analyze calendar "<expr>" systemd-analyze calendar daily
Sessions and users loginctl list-sessions, show-user, terminate-session loginctl list-users
Lingering loginctl enable-linger <user> keeps user services running without a login ls /var/lib/systemd/linger
Transient units systemd-run runs a command as a service, scope or timer systemd-run --unit=<name> <cmd>
cgroup tree systemd-cgls; live usage with systemd-cgtop systemd-cgls -u <unit>
Temporary files tmpfiles.d rules create and clean paths; systemd-tmpfiles --create / --clean cat /usr/lib/tmpfiles.d/tmp.conf
Escaping names systemd-escape --path turns a path into a unit name systemd-escape --path /mnt/backup
Logging from scripts systemd-cat -t <tag> sends output to the journal journalctl -t <tag>
Other tools hostnamectl, timedatectl, localectl, resolvectl, networkctl hostnamectl

Logging

Log Locations

Fact Value Verify with
Two log paths journald receives almost everything; rsyslog copies it into text files systemctl is-active systemd-journald rsyslog
General log RHEL /var/log/messages, Ubuntu /var/log/syslog ls /var/log
Authentication log RHEL /var/log/secure, Ubuntu /var/log/auth.log sudo tail /var/log/secure
Cron log RHEL /var/log/cron; Ubuntu writes cron lines to syslog journalctl -u crond or -u cron
Kernel log dmesg, journalctl -k; Ubuntu also /var/log/kern.log journalctl -k -n 5
Package logs RHEL /var/log/dnf.log, dnf.rpm.log; Ubuntu /var/log/dpkg.log, /var/log/apt/history.log ls /var/log/apt
Login records (binary) wtmp (last), btmp (lastb), lastlog last -n 3
Application logs Own directory, for example /var/log/nginx/ ls /var/log/nginx
Journal storage /var/log/journal (persistent) or /run/log/journal (lost at reboot) journalctl --header (the File path line)
Who can read RHEL files are root only (0600); Ubuntu files are group adm (0640) ls -l /var/log/messages
Journal readers Members of adm, systemd-journal and (RHEL) wheel see all entries id
Syslog socket /dev/log is a symlink to journald's socket ls -l /dev/log

journalctl

Fact Value Verify with
By unit -u nginx (repeatable) journalctl -u nginx -n 5
Follow -f, like tail -f journalctl -fu nginx
By boot -b current, -b -1 previous; list with --list-boots journalctl --list-boots
By priority -p err means err and worse; ranges with -p warning..err journalctl -p err -b
Kernel only -k (implies the current boot unless -b is given) journalctl -k -n 5
By time --since "1 hour ago", --until "2026-09-17 05:48", -S/-U journalctl --since today
Text search -g <regex> (--grep) journalctl -g "bad command"
By field _PID=, _COMM=, _UID=, SYSLOG_IDENTIFIER= (-t) journalctl -F _SYSTEMD_UNIT
Output modes short-iso, short-precise, cat, verbose, json, json-pretty journalctl -o json -n 1
Storage Storage=auto: persistent only if /var/log/journal exists journalctl --header (the File path line)
Size limit SystemMaxUse= (default 10% of the filesystem, capped at 4G) journalctl --disk-usage
Cleanup --vacuum-size=, --vacuum-time=, --vacuum-files= journalctl --vacuum-time=2d
Config /etc/systemd/journald.conf.d/*.conf; restart systemd-journald systemd-analyze cat-config systemd/journald.conf
Rate limit Per service, RateLimitBurst= in RateLimitIntervalSec=; drops are logged as "Suppressed N messages" journalctl -u systemd-journald
Write to it logger, systemd-cat, or stdout of any service logger -t test hello

rsyslog

Fact Value Verify with
Config /etc/rsyslog.conf, then /etc/rsyslog.d/*.conf grep include /etc/rsyslog.conf
Selector facility.priority means that priority and higher; .= exactly; .none excludes grep authpriv /etc/rsyslog.conf
Facilities kern, user, mail, daemon, auth, authpriv, cron, local0 to local7 man 3 syslog
Stop processing & stop after a rule, or stop inside if cat /etc/rsyslog.d/*.conf
Validate rsyslogd -N1 (exit 1 on errors) rsyslogd -N1
Forward over UDP / TCP *.* @host:514 / *.* @@host:514 Rule in /etc/rsyslog.d/
Receive module(load="imtcp") and input(type="imtcp" port="514") ss -tlnp "sport = :514"
Property filters :msg, contains, "segfault" /var/log/crashes.log matches text instead of facility logger -t test segfault
Test messages logger -p local3.err -t app "text"; remote: logger -n host -P 514 -T tail /var/log/messages
Leading - on a file Ubuntu syntax for "do not sync after each line" /etc/rsyslog.d/50-default.conf

logrotate

Fact Value Verify with
Config /etc/logrotate.conf (globals), /etc/logrotate.d/<app> (per log) cat /etc/logrotate.conf
Schedule logrotate.timer, daily on both families (cron.daily on older systems) systemctl list-timers logrotate.timer
State file Last rotation per file: RHEL /var/lib/logrotate/logrotate.status, Ubuntu /var/lib/logrotate/status cat the file
Dry run logrotate -d <conf> changes nothing logrotate -d /etc/logrotate.conf
Force logrotate -f <conf> rotates even if not due logrotate -v -f /etc/logrotate.d/app
Frequency daily, weekly, monthly, yearly, or size 100M; maxsize combines both man logrotate.conf
Keep rotate 7 keeps seven old files ls /var/log/app
Naming Numbered (app.log.1) or dated with dateext (RHEL default) ls /var/log
create Rename the file, create a new empty one; the program must reopen postrotate block
copytruncate Copy then truncate in place; the program keeps its descriptor; lines written between copy and truncate are lost ls -l /proc/<pid>/fd
compress / delaycompress gzip old files; skip the newest one ls *.gz
postrotate ... endscript Command after rotation, usually a reload or kill -USR1 cat /etc/logrotate.d/nginx
sharedscripts Run postrotate once for all files that matched cat /etc/logrotate.d/nginx
su user group Rotate as that user; required when the directory is group- or world-writable logrotate -d warning

Log Parsing Recipes

Fact Value Verify with
Count by field awk '{print $N}', then sort, uniq -c, sort -rn, head Top client IPs
nginx combined format fields $1 client, $4 time, $6 method (with a quote), $7 path, $9 status, $10 bytes head -1 access.log
Split on quotes awk -F'"' '{print $6}' gives the user agent Top user agents
Numeric filter awk '$9 >= 500' 5xx lines
Time window awk '$4 >= "[17/Sep/2026:05:49" && $4 < "[17/Sep/2026:05:50"' (same day only) Requests in one minute
Per minute awk '{print substr($4, 2, 17)}', then uniq -c Traffic shape
Sum and average awk '{s += $10} END {print s/NR}' Bytes per request
Only the match grep -o / grep -oP 'user \K\S+' Invalid usernames
Live filtering tail -f into grep --line-buffered Watching 5xx
Compressed rotations zgrep, or zcat file.gz into the same pipeline Older days
Journal as input journalctl -u ssh -o cat, or -o json into jq -r .MESSAGE Same pipelines, no files
Sort order sort -rn numeric descending; sort -k4 -rn by column 4 Error-rate table

Scheduling

cron and at

Fact Value Verify with
Field order minute, hour, day of month, month, day of week, command cat /etc/crontab
Ranges and steps 1-5, 1,15, */10; day of week 0 and 7 are Sunday man 5 crontab
Day fields If both day of month and day of week are set, either match runs the job man 5 crontab
Shortcuts @reboot, @hourly, @daily, @weekly, @monthly, @yearly man 5 crontab
User crontabs crontab -e, -l, -r, -u user; stored in /var/spool/cron/ (RHEL) or /var/spool/cron/crontabs/ (Ubuntu) sudo ls /var/spool/cron
System crontabs /etc/crontab and /etc/cron.d/* have a user field after the schedule cat /etc/cron.d/0hourly
Periodic directories /etc/cron.hourly, daily, weekly, monthly, run by run-parts ls /etc/cron.daily
anacron Runs daily, weekly and monthly jobs missed while the host was off cat /etc/anacrontab
Environment SHELL=/bin/sh, minimal PATH, HOME of the user, no profile files * * * * * env > /tmp/cron-env.txt
% Means newline in a crontab command; write \% date +\%F
Output Mailed to MAILTO (default the owner); without an MTA it is logged or discarded journalctl -u crond
Access cron.allow wins if present, otherwise cron.deny; same model for at.allow and at.deny ls /etc/cron.*
Daemon crond (cronie) on RHEL, cron on Ubuntu; reloads crontabs automatically systemctl status crond
Logs RHEL /var/log/cron; Ubuntu syslog, both in the journal journalctl -u cron
at / batch One-off job at a time / when load is low; atq, atrm, at -c atq

Systemd Timers

Fact Value Verify with
Pairing name.timer starts name.service unless Unit= says otherwise systemctl cat name.timer
Calendar OnCalendar=Mon..Fri *-*-* 09:00:00; shortcuts hourly, daily, weekly systemd-analyze calendar "..."
Monotonic OnBootSec=, OnUnitActiveSec=, OnActiveSec= (relative to boot, last run, timer start) man systemd.timer
Missed runs Persistent=true runs a missed calendar job at the next boot ls /var/lib/systemd/timers
Spread load RandomizedDelaySec= adds a random delay systemctl list-timers
Precision AccuracySec= (default 1 minute) lets systemd batch wake-ups systemctl show -p AccuracyUSec x.timer
Enable Enable and start the timer, not the service systemctl enable --now name.timer
Service type Usually Type=oneshot; no [Install] needed in the service systemctl status name.service
Run now systemctl start name.service journalctl -u name
List systemctl list-timers --all shows next and last run systemctl list-timers
One-off systemd-run --on-active=30s or --on-calendar= creates a transient timer systemd-run --on-active=1m /bin/true
User timers ~/.config/systemd/user/, systemctl --user, lingering for no-login runs loginctl show-user $USER -p Linger

Kernel and Hardware

proc and sys

Fact Value Verify with
/proc procfs: one directory per PID plus system-wide files mount -t proc
/proc/sys Writable kernel parameters, managed by sysctl ls /proc/sys
/sys sysfs: the device and driver model, one value per file ls /sys/class/net
/dev devtmpfs: device nodes, populated by the kernel and udev mount -t devtmpfs
/run, /dev/shm tmpfs: memory-backed, cleared at reboot df -h -t tmpfs
File sizes Most files show size 0 (procfs) or 4096 (sysfs); content is generated on read ls -l /proc/meminfo
Memory /proc/meminfo (free reads it) grep MemAvailable /proc/meminfo
CPU /proc/cpuinfo, /proc/loadavg, /proc/stat nproc
Kernel /proc/version, /proc/cmdline, /proc/modules, /proc/config.gz if built in uname -r
Storage /proc/partitions, /proc/mounts, /sys/block/<dev>/ lsblk
Network /proc/net/tcp, /sys/class/net/<if>/ ss, ip link
Writing Needs root for most files; sudo echo x > file fails because the shell opens the file sudo tee file with the value on standard input
Persistence Writes last until reboot; persistent settings go in sysctl.d or udev rules sysctl --system
Per-process files /proc/<pid>/status, cmdline, environ, fd/, limits, maps, cgroup cat /proc/self/status

sysctl

Fact Value Verify with
Name mapping net.ipv4.ip_forward is /proc/sys/net/ipv4/ip_forward cat /proc/sys/net/ipv4/ip_forward
Read sysctl key, sysctl -a, sysctl -a --pattern <regex> sysctl vm.swappiness
Write now sysctl -w key=value (root; lost at reboot) sysctl key
Persistent A file in /etc/sysctl.d/ named NN-name.conf, then sysctl --system sysctl --system
Load one file sysctl -p <file>; sysctl -p alone reads /etc/sysctl.conf sysctl -p /etc/sysctl.d/90-app.conf
Directories /etc/sysctl.d, /run/sysctl.d, /usr/lib/sysctl.d; a file in /etc overrides one with the same name systemd-analyze cat-config sysctl.d/99-sysctl.conf
Order All files sorted by name; for the same key, the last file read wins sysctl --system
/etc/sysctl.conf Symlinked as /etc/sysctl.d/99-sysctl.conf on both families ls -l /etc/sysctl.d/99-sysctl.conf
Boot systemd-sysctl.service applies the files systemctl status systemd-sysctl
Module keys Keys appear only after their module loads (net.sctp.*, net.bridge.*) sysctl net.sctp.rto_min
Namespaces Most net.* keys are per network namespace; containers can have their own ip netns exec x sysctl net.ipv4.ip_forward
Routing net.ipv4.ip_forward = 1 for routers, NAT, Kubernetes nodes sysctl net.ipv4.ip_forward
Memory vm.swappiness, vm.overcommit_memory, vm.max_map_count, vm.dirty_ratio sysctl -a --pattern ^vm
Files and processes fs.file-max, fs.inotify.max_user_watches, kernel.pid_max sysctl fs.file-nr

Kernel Modules

Fact Value Verify with
Location /lib/modules/$(uname -r)/kernel/**.ko (often .ko.xz or .ko.zst) find /lib/modules/$(uname -r) -name "*.ko*"
List loaded lsmod (reads /proc/modules): name, size, use count, users lsmod
Details modinfo <name>: file, license, dependencies, signature, parameters modinfo -p nbd
Load modprobe <name> resolves dependencies from modules.dep; insmod needs a path and resolves nothing modprobe -v <name>
Unload modprobe -r (with unused dependencies) or rmmod; fails while in use lsmod use count
Built-in Code compiled into the kernel image; cannot be unloaded; listed in modules.builtin modinfo loop shows (builtin)
Parameters modprobe nbd nbds_max=4, persistently options nbd nbds_max=4 in /etc/modprobe.d/*.conf cat /sys/module/nbd/parameters/nbds_max
Blacklist blacklist <name> stops automatic loading only modprobe -c
Block completely install <name> /bin/false makes every load fail modprobe <name>
Load at boot One name per line in /etc/modules-load.d/*.conf, read by systemd-modules-load.service systemctl status systemd-modules-load
Dependency index depmod -a rebuilds modules.dep after adding a module ls /lib/modules/$(uname -r)/modules.dep
Taint Out-of-tree or unsigned modules set /proc/sys/kernel/tainted cat /proc/sys/kernel/tainted
Early boot Storage drivers needed to mount / must be in the initramfs lsinitrd or lsinitramfs

Devices and udev

Fact Value Verify with
Type, major, minor b block or c character; major selects the driver, minor the instance ls -l /dev/vda /dev/null, cat /proc/devices
/dev devtmpfs created by the kernel; udev adds permissions and symlinks mount -t devtmpfs
Stable names /dev/disk/by-uuid, by-id, by-path, by-label ls -l /dev/disk/by-uuid
Device database udevadm info --name=<dev> (properties), --attribute-walk (rule keys) udevadm info --name=/dev/vda
Rules /usr/lib/udev/rules.d/ (vendor), /etc/udev/rules.d/ (local, wins on same name) ls /etc/udev/rules.d
Apply rules udevadm control --reload, then udevadm trigger for existing devices ls -l /dev/<name>
Hardware and events lspci -k, lsusb, udevadm monitor --udev; sensors and ipmitool need physical hardware (No sensors found! in this VM) lspci -k
Create a node by hand mknod <path> c <major> <minor> sudo mknod /tmp/mynull c 1 3

dmesg and Kernel Messages

Fact Value Verify with
Ring buffer Fixed size; old messages are overwritten; the journal keeps a copy dmesg
Readable times dmesg -T (can drift after suspend); journalctl -k has real timestamps dmesg -T
Filter by level dmesg -l err,warn; journalctl -k -p warning dmesg -l err
Show level and facility dmesg -x dmesg -x
Follow dmesg -w, journalctl -kf New messages appear
Previous boot Only the journal: journalctl -k -b -1 journalctl --list-boots
Access kernel.dmesg_restrict = 1 limits dmesg to root (0 on both playgrounds) sysctl kernel.dmesg_restrict
Console level kernel.printk: messages below the first number reach the console sysctl kernel.printk
OOM kill Out of memory: Killed process or Memory cgroup out of memory journalctl -k -g "Killed process"
Segfault <prog>[pid]: segfault at <addr> ip ... error N in <binary> journalctl -k -g segfault
I/O error Buffer I/O error on dev ..., I/O error, dev sdb, sector ... dmesg -l err
Hung task INFO: task <name>:<pid> blocked for more than N seconds sysctl kernel.hung_task_timeout_secs
Core dumps systemd-coredump stores them; coredumpctl list, info, debug sysctl kernel.core_pattern

Storage

Disks and Devices

Fact Value Verify with
List block devices lsblk; lsblk -f adds filesystem, label, UUID and usage lsblk -o NAME,SIZE,TYPE,MOUNTPOINTS
Filesystem signatures blkid (as root for uncached devices) sudo blkid
SATA, SAS, USB disks /dev/sda, /dev/sdb; partitions sda1 lsblk
Virtio disks (KVM) /dev/vda; partitions vda1 lsblk -o NAME,TRAN
NVMe /dev/nvme0n1 (controller 0, namespace 1); partitions nvme0n1p1 nvme list
Xen disks (older EC2) /dev/xvda lsblk
Other block devices loopN (file), dm-N (LVM, LUKS), mdN (software RAID), sr0 (optical) lsblk -o NAME,TYPE
Stable names /dev/disk/by-uuid, by-id, by-path, by-label, by-partuuid ls -l /dev/disk/by-uuid
Rotational flag 1 spinning disk (and most virtual disks), 0 SSD cat /sys/block/<dev>/queue/rotational
Size in sectors /sys/block/<dev>/size, always in 512-byte units cat /sys/block/vda/size
MBR 4 primary partitions (or 3 plus an extended one), 2 TiB limit fdisk -l
GPT 128 partitions by default, backup table at the end of the disk, needed above 2 TiB and for UEFI gdisk -l
Disk health smartctl -H (SATA, SAS), nvme smart-log (NVMe); physical or passthrough disks only sudo smartctl -i /dev/sda
Loop device Presents a file as a disk: losetup -fP --show <file> losetup -l

Partitioning

Fact Value Verify with
fdisk Interactive, MBR and GPT; changes stay in memory until w sudo fdisk -l /dev/sdb
gdisk, sgdisk GPT only; type codes such as 8300, 8E00, EF00, 8200 sudo gdisk -l /dev/sdb
parted MBR and GPT, scriptable with -s; writes each command at once sudo parted /dev/sdb print free
sfdisk Dumps and restores a table as text sudo sfdisk -d /dev/sdb > sdb.dump
MBR type IDs 83 Linux, 82 swap, 8e LVM, fd RAID, ef EFI fdisk command l
MBR layout 4 primary, or 3 primary plus 1 extended holding logical partitions (5 and up) sudo fdisk -l
Alignment Partitions start at 1 MiB (sector 2048) sudo parted /dev/sdb align-check optimal 1
Re-read the table partprobe <disk>, or partx -a (add) and partx -u (update) lsblk <disk>
Remove signatures wipefs -a <device> (without -a it only lists them) sudo wipefs /dev/sdb
ESP GPT type EF00 (ESP flag in parted), FAT32, mounted at /boot/efi lsblk -o NAME,PARTTYPENAME
BIOS boot GPT disk booting in BIOS mode needs a 1 MiB EF02 partition for GRUB sudo gdisk -l

Filesystems

Fact Value Verify with
Defaults RHEL: XFS; Ubuntu and Debian: ext4 findmnt -no FSTYPE /
Create mkfs.ext4, mkfs.xfs, mkfs.vfat -F 32, mkfs.btrfs; -L sets a label lsblk -f
Existing signature mkfs.xfs and mkfs.btrfs refuse without -f; mke2fs asks on a terminal wipefs <dev>
Resize ext4: resize2fs, grows online, shrinks offline; XFS: xfs_growfs <mountpoint> grows only (apart from an experimental trim of the last allocation group) df -h
Check and repair ext4: e2fsck (unmounted); XFS: xfs_repair (unmounted; -n checks only) exit status
Inspect and label tune2fs -l, dumpe2fs -h, xfs_info; tune2fs -L, xfs_admin -L (unmounted), fatlabel blkid
Reserved blocks ext4 keeps 5% for root; tune2fs -m 1 lowers it tune2fs -l
Inodes ext4 fixes the count at creation (mkfs.ext4 -i, -N); XFS allocates them dynamically df -i
Backup superblocks ext4 keeps copies; mke2fs -n lists them, e2fsck -b <block> uses one dumpe2fs <dev>
Btrfs Copy-on-write, subvolumes, snapshots, checksums; Fedora and SUSE default; removed from RHEL 8 btrfs filesystem show

Mounting and fstab

Fact Value Verify with
Mount and unmount mount <dev> <dir>, umount <dir> (not "unmount") findmnt <dir>
Show mounts findmnt (tree), findmnt -t xfs, mount, /proc/self/mountinfo findmnt -no OPTIONS /
fstab fields device, mount point, type, options, dump, fsck pass man 5 fstab
Device field UUID=, LABEL=, PARTUUID= or /dev/...; UUID survives renames blkid
fsck pass 1 root, 2 other local filesystems, 0 skip (always 0 for XFS) cat /etc/fstab
Test fstab findmnt --verify, then mount -a exit status
systemd systemd-fstab-generator turns each line into a .mount unit; daemon-reload after edits systemctl list-units -t mount
nofail Boot continues if the device is missing systemctl show -p WantedBy <unit>
_netdev Wait for the network (NFS, iSCSI); implied for network filesystem types man systemd.mount
x-systemd.device-timeout= How long boot waits for the device (default 90 s) systemctl cat <unit>
Common options ro, noatime, noexec, nosuid, nodev, defaults (rw,suid,dev,exec,auto,nouser,async) findmnt -no OPTIONS <dir>
Busy target fuser -vm <dir>, lsof +f -- <dir>; umount -l detaches lazily umount <dir>
vfat and NTFS ownership Set at mount time with uid=, gid=, umask= ls -l

Swap

Fact Value Verify with
Show swap swapon --show, free -h, /proc/swaps swapon --show
Prepare mkswap [-L label] <dev or file> blkid shows TYPE="swap"
Enable, disable swapon <dev>, swapoff <dev>; -a for every fstab entry swapon --show
fstab line UUID=... none swap defaults 0 0 (pri=N sets the priority) systemctl list-units -t swap
Swap file Created with dd or fallocate (no holes), mode 0600 ls -l /swapfile
Priority Higher is used first; equal priorities are used in round robin swapon --show
vm.swappiness 0 to 200, kernel default 60; lower values keep more anonymous memory in RAM sysctl vm.swappiness
Per process VmSwap in /proc/<pid>/status grep VmSwap /proc/*/status
Per cgroup MemorySwapMax=, memory.swap.max; MemorySwapCurrent systemctl show -p MemorySwapCurrent <unit>
Sizing Up to the RAM size on small hosts, a few GiB on large ones; RAM plus more for hibernation free -h
Kubernetes The kubelet refuses to start with swap on unless failSwapOn: false is set swapon --show
zram Compressed swap in RAM; the default on Fedora zramctl

LVM

Fact Value Verify with
Layers Physical volume (PV) → volume group (VG) → logical volume (LV) → filesystem lsblk
Extents A VG is split into physical extents (PE, 4 MiB by default); an LV is a list of them vgdisplay
Create pvcreate, vgcreate <vg> <pvs>, lvcreate -n <lv> -L <size> <vg> (-l 100%FREE for extents) pvs, vgs, lvs
Device paths /dev/<vg>/<lv> and /dev/mapper/<vg>-<lv>, both links to /dev/dm-N ls -l /dev/mapper
Grow vgextend adds a PV; lvextend -r grows the LV and the filesystem together df -h
Shrink lvreduce -r (ext4 only; unmounts it); XFS cannot shrink lvs
Snapshot lvcreate -s -L <size> -n <snap> <vg>/<lv>; invalid when full; lvconvert --merge rolls back lvs (Data%)
Move data pvmove <pv> empties a PV online; then vgreduce and pvremove pvs
Thin pools --type thin-pool, lvcreate -V <size> -T <vg>/<pool>; allows overprovisioning lvs (Data%, Meta%)
Devices file RHEL 9 and later only use PVs listed in /etc/lvm/devices/system.devices; Ubuntu 24.04 does not use one lvmdevices
Metadata backup /etc/lvm/backup/<vg>, restored with vgcfgrestore vgcfgbackup

Resizing and Cloud Disks

Fact Value Verify with
Layers to grow Disk → partition → PV (if LVM) → LV → filesystem lsblk
See the new disk size NVMe and Virtio: automatic; SCSI: echo 1 > /sys/class/block/sdX/device/rescan lsblk
Grow a partition growpart <disk> <number> (package cloud-utils-growpart on RHEL, cloud-guest-utils on Ubuntu) lsblk
Grow a PV pvresize <partition> pvs
Grow an LV and its filesystem lvextend -r -l +100%FREE <vg>/<lv> df -h
Grow a filesystem ext4: resize2fs <device>; XFS: xfs_growfs <mountpoint> df -h
Online Every step above works on mounted filesystems, including / findmnt /
Shrinking Cloud volumes cannot shrink; create a smaller volume and copy the data provider console
Boot-time growth cloud-init growpart and resize_rootfs grow the root partition and filesystem on first boot cloud-init status --long
Partition position Only the last partition, or one followed by free space, can grow in place sudo parted <disk> print free
MBR limit A partition on an MBR disk cannot use space beyond 2 TiB sudo fdisk -l
AWS aws ec2 modify-volume --size; repeated modifications of one volume are rate-limited aws ec2 describe-volumes-modifications

Disk Usage

Fact Value Verify with
Filesystem usage df -h, df -hT (type), df -x tmpfs (exclude a type) df -h /
Inode usage df -i df -i /
Directory totals du -sh <dir>, du -xh --max-depth=1 / (-x stays on one filesystem) du -sh /var/log
Sort sizes sort -h orders human-readable sizes such as 1.2G and 51M sort --help
Large files find / -xdev -type f -size +100M ls -lh
Interactive ncdu -x / (EPEL on RHEL, apt install ncdu) ncdu --version
Apparent vs allocated ls -l and du --apparent-size show the length; du shows the blocks used du -h <sparse file>
Deleted but open Space is freed when the last process closes the file lsof -a +L1 <mountpoint>
Free it without a restart : > /proc/<pid>/fd/<fd> truncates the open file df -h
Reserved blocks ext4 keeps 5% for root, so users hit No space left at Avail 0 before Size is used tune2fs -l <dev>
Inode exhaustion No space left on device with free blocks; ext4 inode count is fixed df -i
Hidden files Files written to a directory before a filesystem was mounted over it count in df, not in du mount --bind / /mnt/rootfs
Usual growth /var/log, the journal, package caches, container images, /tmp, core dumps, backups du -xh --max-depth=2 /var
Journal size journalctl --disk-usage, --vacuum-size=, SystemMaxUse= journalctl --disk-usage

Quotas

Fact Value Verify with
Limits Soft (may be exceeded for the grace period, 7 days by default) and hard (never exceeded) quota -s <user>
XFS Mount options uquota, gquota, pquota at mount time; managed with xfs_quota -x findmnt -no OPTIONS <dir>
ext4 tune2fs -O quota -Q usrquota,grpquota (unmounted), mount with usrquota,grpquota quotaon -p <dir>
Set limits xfs_quota -x -c 'limit bsoft=50m bhard=60m <user>'; setquota -u <user> <bsoft> <bhard> <isoft> <ihard> <dir>; edquota -u <user> opens an editor quota -s <user>
Reports and grace xfs_quota -x -c 'report -h', repquota -s <dir>; grace with xfs_quota -x -c 'timer ...' or setquota -t as root

Backup and Restore

Fact Value Verify with
Full Copies everything; slowest to take, one step to restore backup size
Incremental Copies changes since the last backup of any kind; restore needs the full plus every incremental in order tar --listed-incremental
Differential Copies changes since the last full; restore needs the full plus the latest differential backup size
3-2-1 rule 3 copies, on 2 kinds of media, 1 off site backup inventory
RPO and RTO How much data may be lost; how long a restore may take runbook
rsync -a Recursive, keeps links, modes, times, owner and group; add -H (hard links), -A (ACLs), -X (xattrs, SELinux labels) rsync -aHAX
Trailing slash src/ copies the contents; src copies the directory itself ls dest
Dry run rsync -n -i shows what would change; --delete removes files gone from the source rsync -ani --delete
Snapshot-style copies rsync --link-dest=<previous> hard-links unchanged files ls -li
Consistent copy LVM, cloud or filesystem snapshot, or the application's own dump (pg_dump, mysqldump) lvs
Block image dd if=<dev> of=<file> bs=4M; ddrescue for failing disks sha256sum
Verify tar -d (compare), rsync -anc (checksums), a real test restore exit status

RAID and Encryption

Fact Value Verify with
Levels 0 striping, no redundancy; 1 mirror; 5 and 6 striping with one or two parity blocks mdadm --detail /dev/md0
RAID 10 Striped mirrors; common for databases mdadm --detail
Status /proc/mdstat: [UU] healthy, [U_] degraded cat /proc/mdstat
Replace a disk --fail, --remove, then --add the new one; rebuild runs online /proc/mdstat
Persist the array mdadm --detail --scan into /etc/mdadm.conf (Ubuntu: /etc/mdadm/mdadm.conf), then dracut -f (Ubuntu: update-initramfs -u) cat /proc/mdstat after a reboot
LUKS2 cryptsetup luksFormat, open, close; up to 32 keyslots (passphrases or key files) cryptsetup luksDump
Open at boot /etc/crypttab: name, device, key file or none, options; fstab then mounts /dev/mapper/<name> systemctl status systemd-cryptsetup@<name>
Stratis and VDO Removed from the RHCSA objectives; VDO now lives in LVM (lvcreate --type vdo) man lvmvdo

Networking

Interfaces and Addresses

Fact Value Verify with
Tool ip (iproute2) replaced ifconfig, route and arp (net-tools); net-tools may be absent ip -br addr
Link vs address ip link is layer 2 (state, MAC, MTU); ip addr is layer 3 ip -br link
State flags UP = administratively up; LOWER_UP = carrier present ip link show eth0
Predictable names enp0s3 (PCI path), ens3 (slot), eno1 (onboard), enx<MAC>; net.ifnames=0 keeps eth0 udevadm test-builtin net_id /sys/class/net/eth0
Loopback lo, 127.0.0.1/8 and ::1; traffic never leaves the host ip addr show lo
Link-local IPv6 fe80::/64, created on every IPv6-enabled link; derived from the MAC unless privacy addresses are on ip -6 addr
Runtime only ip addr add and ip link set are lost at reboot; persistence lives in NetworkManager or netplan ip -br addr after a reboot
Connected route Adding 10.0.0.1/24 to an up link adds the route 10.0.0.0/24 ip route show dev <if>
ARP / neighbor IPv4 to MAC cache; states REACHABLE, STALE, DELAY, INCOMPLETE, FAILED ip neigh
Counters Errors and drops per interface ip -s link show eth0
Driver and speed ethtool eth0 (speed, duplex, link), ethtool -i (driver), ethtool -S (NIC counters) sudo ethtool eth0
MTU Default 1500; jumbo frames 9000; tunnels and cloud overlays often lower ip link show eth0
sysfs /sys/class/net/<if>/ holds address, mtu, carrier, statistics/ cat /sys/class/net/eth0/mtu

Network Configuration

Fact Value Verify with
RHEL stack NetworkManager; profiles are keyfiles in /etc/NetworkManager/system-connections/ nmcli connection show
RHEL 10 No ifcfg-rh plugin, so /etc/sysconfig/network-scripts files are ignored; RHEL 9 still reads them but writes keyfiles ls /usr/lib64/NetworkManager/*/
Connection vs device A connection (profile) is applied to a device (interface) nmcli device status
Change a profile nmcli con mod edits the file; nmcli con up or nmcli dev reapply applies it ip -br addr
Add to a list +ipv4.addresses, -ipv4.addresses; without the sign the list is replaced nmcli -g ipv4.addresses con show <name>
Auto profiles NetworkManager creates Wired connection 1 (DHCP) for an unconfigured NIC; no-auto-default=* stops it nmcli connection show
Ubuntu stack netplan YAML in /etc/netplan/*.yaml generates systemd-networkd (server) or NetworkManager (desktop) files netplan get
netplan files Mode 0600; lists use YAML sequences; indentation is significant sudo netplan generate
Apply safely netplan try reverts after the timeout unless confirmed; netplan apply does not sudo netplan try
cloud-init Writes /etc/netplan/50-cloud-init.yaml on cloud images ls /etc/netplan
Hostname hostnamectl set-hostname writes /etc/hostname; hostname -s is the short name hostnamectl
Name for the host Add the hostname to /etc/hosts when no DNS record exists getent hosts $(hostname)

Routing

Fact Value Verify with
Lookup rule Longest prefix wins; among equal prefixes, the lowest metric wins ip route get <ip>
Default route default = 0.0.0.0/0, used when nothing more specific matches ip route show default
Connected route Created with each address on an up link (proto kernel scope link) ip route
via Next-hop router; it must be reachable on a connected subnet ip route get <ip>
Forwarding A Linux host routes between interfaces only with net.ipv4.ip_forward = 1 sysctl net.ipv4.ip_forward
Return path Replies use the destination host's own routing table; a missing return route drops the reply tcpdump -e on the destination
TTL Each router decrements TTL by one; ttl=63 from a Linux host means one hop ping
Reverse path filter rp_filter=1 (strict) drops packets arriving on an interface that would not be used to reply sysctl net.ipv4.conf.all.rp_filter
Policy routing ip rule chooses a table (by source, mark, interface) before the route lookup ip rule show
Tables local (own addresses), main (normal), default; custom tables by number or /etc/iproute2/rt_tables ip route show table all
Special routes blackhole, unreachable, prohibit drop traffic on purpose ip route
Persistence NetworkManager ipv4.routes, netplan routes:, sysctl.d for forwarding nmcli -g ipv4.routes con show <name>
Legacy view route -n (net-tools); flags U up, G gateway, H host route -n

DNS Resolution

Fact Value Verify with
Lookup order hosts: line in /etc/nsswitch.conf; files = /etc/hosts, dns = resolvers in /etc/resolv.conf grep ^hosts /etc/nsswitch.conf
RHEL hosts: files dns myhostname; /etc/resolv.conf written by NetworkManager head -1 /etc/resolv.conf
Ubuntu systemd-resolved stub at 127.0.0.53; /etc/resolv.conf is a symlink to stub-resolv.conf ls -l /etc/resolv.conf, resolvectl status
Application view getent hosts <name> follows nsswitch like applications getent ahostsv4 <name>
DNS view dig, host, nslookup query DNS only and ignore /etc/hosts (the resolved stub serves hosts entries itself) dig @<server> <name>
resolv.conf limits Up to 3 nameserver lines; search domains; options timeout:N attempts:N cat /etc/resolv.conf
Search list Short names get each search domain appended getent hosts db
Status codes NOERROR, NXDOMAIN (name does not exist), SERVFAIL (server failed), REFUSED (policy) status: line of dig
aa flag Authoritative answer from the zone's own server dig @<server> <name>
TTL Seconds a resolver may cache the record; counts down in cached answers dig +noall +answer twice
Record types A, AAAA, CNAME, MX, NS, TXT, SOA, PTR (reverse, dig -x), SRV dig <name> <type>
Transport UDP 53, TCP 53 for large answers and zone transfers ss -ulpn 'sport = :53'
Split DNS resolved routing domain Domains=~zone sends only that zone to a server resolvectl domain
Cache control resolvectl flush-caches, resolvectl statistics sudo resolvectl statistics

Ports and Sockets

Fact Value Verify with
Tool ss (iproute2) replaced netstat (net-tools) ss -tulpn
Flags -t TCP, -u UDP, -l listening, -p process (needs root for other users), -n numeric, -a all sudo ss -tulpn
Socket identity Protocol + local IP:port + remote IP:port; many connections share one listening port ss -tn
0.0.0.0 / [::] / * Listening on all addresses (IPv4, IPv6, both) ss -tln
127.0.0.1 Reachable only from the same host (or the same network namespace) ss -tln
Privileged ports Below ip_unprivileged_port_start (1024) need root or CAP_NET_BIND_SERVICE sysctl net.ipv4.ip_unprivileged_port_start
Ephemeral ports Client source ports from ip_local_port_range (32768 to 60999) sysctl net.ipv4.ip_local_port_range
Service names /etc/services maps names to ports getent services 443
Well-known ports 22 SSH, 25 SMTP, 53 DNS, 80 HTTP, 123 NTP (UDP), 443 HTTPS, 3306 MySQL, 5432 PostgreSQL, 6379 Redis, 6443 Kubernetes API, 2379 etcd, 10250 kubelet ss -tlnp
Who owns a port ss -tlpn 'sport = :8080', lsof -i :8080, fuser -v 8080/tcp sudo lsof -i :8080 -P -n
Refused vs timeout Refused: host answered with RST, nothing listens; timeout: packets dropped on the way nc -vz host port
Raw source /proc/net/tcp and /proc/net/tcp6 in hex, host byte order cat /proc/net/tcp
Socket activation systemd can hold the listening socket (sshd.socket here) systemctl list-sockets

Sockets and TCP States

Fact Value Verify with
Server calls socket(), bind(), listen(backlog), accept() returns a new fd per client strace -e trace=network
Client calls socket(), connect(); the kernel picks an ephemeral source port ss -tn
Handshake SYN, SYN-ACK, ACK; the kernel completes it without the application tcpdump 'tcp[tcpflags] & tcp-syn != 0'
SYN queue Half-open connections (SYN-RECV); limit tcp_max_syn_backlog; tcp_syncookies=1 survives floods sysctl net.ipv4.tcp_syncookies
Accept queue Completed connections waiting for accept(); limit min(backlog, somaxconn), and somaxconn is 4096 since kernel 5.4 ss -ltn (Recv-Q/Send-Q)
Queue overflow New SYNs are dropped, clients sit in SYN-SENT and retry nstat -az TcpExtListenOverflows
Active closer Sends the first FIN, passes FIN-WAIT-1, FIN-WAIT-2, then TIME-WAIT (60 s on Linux) ss -tan state time-wait
Passive closer Receives the FIN and stays in CLOSE-WAIT until the application calls close() ss -tanp state close-wait
CLOSE_WAIT pile-up Always an application bug (sockets not closed) ss -tanp state close-wait
TIME_WAIT pile-up Normal on busy clients; costs little memory; tcp_tw_reuse lets outgoing connections reuse them ss -s
Orphan FIN-WAIT-2 Closed by the kernel after tcp_fin_timeout (60 s) sysctl net.ipv4.tcp_fin_timeout
Connection tracking Netfilter tracks flows for stateful rules and NAT; full table drops new flows conntrack -C, sysctl net.netfilter.nf_conntrack_max
Per-socket detail RTT, congestion window, retransmits ss -tin

Connectivity Testing

Fact Value Verify with
ping ICMP echo request and reply; shows loss, RTT and the reply's TTL ping -c3 <host>
ICMP blocked A host that drops ping can still serve TCP; test the port instead nc -vz <host> <port>
traceroute Sends probes with TTL 1, 2, 3...; each router that drops one returns ICMP Time Exceeded traceroute -n <host>
Probe types traceroute UDP by default, -I ICMP, -T TCP (root); tracepath UDP without root man traceroute
Markers * * * no reply from that hop; !H host unreachable, !N network unreachable, !X prohibited traceroute -n
mtr Continuous traceroute with per-hop loss; loss only at a middle hop is ICMP rate limiting mtr -n -r -c 10 <host>
MTU probe ping -M do -s <size>: payload + 28 bytes of headers must fit the MTU (1472 for 1500) ping -M do -s 1472 <host>
PMTU cache The kernel remembers a lower path MTU per destination for 10 minutes ip route get <host>
Port test nc -vz host port; timeout 2 bash -c '< /dev/tcp/host/port' without nc exit status
Refused vs timeout Refused: RST (nothing listens or reject); timeout: dropped nc -vz -w3
HTTP curl -v (headers), -I (HEAD), -w (timings), --resolve (skip DNS), -k (skip TLS verify) curl -sv <url>
Throughput iperf3 -s on one side, iperf3 -c on the other; port 5201 iperf3 -c <host>
telnet Often not installed; nc or curl telnet://host:port replace it command -v telnet

Packet Capture

Fact Value Verify with
Privilege Capturing needs root or CAP_NET_RAW and CAP_NET_ADMIN sudo tcpdump -D
Interfaces -i eth0, -i any (all, cooked headers with direction); -D lists them tcpdump -D
Basic flags -n no name lookups, -c N stop after N packets, -v more detail, -e MAC addresses sudo tcpdump -ni eth0 -c 5
Payload -A ASCII, -X hex and ASCII, -s 0 full packets (default snap length 262144) sudo tcpdump -A
Files -w file.pcap writes, -r reads, -U flushes per packet; -C/-G/-W rotate tcpdump -nr file.pcap
Filter primitives host, net, port, portrange, src/dst, tcp/udp/icmp, arp man pcap-filter
Combining and, or, not, parentheses in quotes 'host 10.0.0.5 and not port 22'
TCP flags filter tcp[tcpflags] with tcp-syn, tcp-ack, tcp-fin, tcp-rst, tcp-push 'tcp[tcpflags] & tcp-rst != 0'
Flag letters S SYN, . ACK, P push, F FIN, R reset; S. is SYN-ACK handshake capture
Where it sees traffic Incoming before the firewall's input rules, outgoing after the output rules capture plus firewall counters
Other tools tshark (Wireshark CLI), capinfos, termshark tshark -r file.pcap

Bridges, Bonds and VLANs

Fact Value Verify with
Bridge Learns MAC addresses per port (forwarding database); docker0 and cni0 are bridges bridge fdb show
veth pair Two linked interfaces; one end in a container namespace, the other on a bridge ip -br link type veth
Bond modes 0 balance-rr, 1 active-backup, 2 balance-xor, 4 802.3ad (LACP, needs switch support), ⅚ adaptive; miimon polls link state cat /proc/net/bonding/bond0
Teaming teamd deprecated in RHEL 9 and absent from RHEL 10 repositories; use bonding dnf list teamd
VLAN 802.1Q tag with a 12-bit ID (1 to 4094); interface name eth0.10 by convention ip -d link show eth0.10
Persistence NetworkManager type bridge, bond, vlan profiles; netplan bridges:, bonds:, vlans: nmcli con show

Time and Timezones

Fact Value Verify with
Owner timedatectl shows and sets time, timezone and NTP state timedatectl
Timezone /etc/localtime is a symlink into /usr/share/zoneinfo/; TZ= overrides it per process ls -l /etc/localtime
Set timezone timedatectl set-timezone Asia/Karachi timedatectl list-timezones
NTP client chrony (chronyd) on RHEL and Ubuntu with the chrony package; systemd-timesyncd is Ubuntu's minimal default systemctl is-active chronyd
chrony config RHEL /etc/chrony.conf; Ubuntu /etc/chrony/chrony.conf plus sources.d/*.sources grep ^pool /etc/chrony.conf
Server vs pool server is one host; pool resolves a name to several; iburst speeds up the first sync chronyc sources
Step vs slew makestep 1.0 3 steps the clock if it is off by more than 1 s during the first 3 updates; otherwise chrony slews it gradually journalctl -u chronyd
Serving time allow <network> makes chronyd an NTP server on UDP 123 ss -ulpn 'sport = :123'
Stratum Distance from a reference clock; each NTP hop adds one chronyc tracking
Manual time timedatectl set-time requires NTP off, and interprets the value in the local timezone timedatectl set-ntp false
RTC The hardware clock (hwclock), normally in UTC; many cloud and microVM instances have none timedatectl (RTC time)
Epoch Seconds since 1970-01-01 UTC date +%s, date -d @0 -u

Reverse Proxy and Load Balancing

Fact Value Verify with
Reverse vs forward proxy A reverse proxy fronts servers for clients; a forward proxy fronts clients for the internet architecture
nginx pieces upstream block lists backends; proxy_pass http://<upstream> sends requests there sudo nginx -T
Algorithms Round robin (default), least_conn, ip_hash/hash (stickiness), weights upstream config
Client address Backends see the proxy's IP; the original goes in X-Forwarded-For backend log
Passive checks nginx open source marks a backend failed after max_fails errors for fail_timeout nginx error log
Active checks HAProxy option httpchk probes each server; fall/rise set the thresholds show stat
502 Bad Gateway The proxy could not get a valid response: backend down, refused or crashed error log connect() failed
504 Gateway Timeout The backend accepted but did not answer within proxy_read_timeout error log upstream timed out
Retries nginx tries the next upstream on connection errors (proxy_next_upstream) $upstream_addr in the log
Validate first nginx -t; haproxy -c -f <file> (silent on success, -V prints Configuration file is valid) exit status
SELinux On RHEL, nginx needs httpd_can_network_connect to reach backends on non-HTTP ports getsebool httpd_can_network_connect
L4 vs L7 Layer 4 forwards TCP (HAProxy mode tcp, nginx stream); layer 7 reads HTTP (paths, headers) mode in config

VPN (WireGuard)

Fact Value Verify with
Transport UDP, default port 51820; no TCP mode ss -ulpn 'sport = :51820'
Identity Each peer has a key pair; wg genkey, wg pubkey; private keys stay mode 0600 ls -l /etc/wireguard
AllowedIPs Routing table and access list in one: outgoing packets pick the peer by destination, incoming packets are accepted only from these sources wg show wg0 allowed-ips
Handshake Every two minutes while traffic flows; no handshake means no tunnel wg show wg0 latest-handshakes
Keepalive PersistentKeepalive = 25 keeps NAT mappings open for a peer behind NAT wg show
MTU wg-quick sets 1420 (1500 minus 80 bytes of overhead) ip link show wg0
Tooling wg-quick up wg0 reads /etc/wireguard/wg0.conf; wg-quick@wg0.service makes it permanent systemctl status wg-quick@wg0

Troubleshooting Ladder

Fact Value Verify with
Order Link, IP, route, gateway ARP, ping, DNS, port, firewall, application this page
Link UP and LOWER_UP; NO-CARRIER means cable or virtual NIC ip -br link
Address Right IP and prefix on the right interface ip -br addr
Route The path the kernel will use for this destination ip route get <ip>
Gateway Neighbor entry REACHABLE or STALE, never FAILED ip neigh show <gw>
Reachability ICMP may be blocked; a failure here is a hint, not proof ping -c3 <ip>
Name Resolve the way the application does getent hosts <name>
Port refused = nothing listens or reject; timed out = dropped nc -vz -w3 <ip> <port>
Listener Service bound to the right address on the server sudo ss -tlnp 'sport = :<port>'
Firewall Host rules, then security groups and network ACLs sudo nft list ruleset, firewall-cmd --list-all
Proof on the wire SYNs arriving without SYN-ACK = dropped on the server sudo tcpdump -nn -i any port <port>
Application HTTP status, TLS errors, application logs curl -sv <url>, journalctl -u <unit>
Compare A working client or path narrows the fault faster than any single tool same tests from two hosts

SSH and Remote Access

SSH Client

Fact Value Verify with
Key type ed25519 is the default choice; RSA keys need 3072 bits or more ssh-keygen -l -f ~/.ssh/id_ed25519.pub
Key files Private key ~/.ssh/id_ed25519 (mode 600), public key .pub (mode 644), directory ~/.ssh (mode 700) ls -la ~/.ssh
Server side Public keys go into ~/.ssh/authorized_keys of the target user ssh-copy-id user@host
Host identity First connection asks to trust the host key; the answer is stored in ~/.ssh/known_hosts ssh-keygen -F host
Hashed hosts Ubuntu sets HashKnownHosts yes, so entries hold a hash instead of the host name; RHEL stores names in clear text ssh -G host
Agent ssh-agent holds decrypted keys; ssh-add loads them; SSH_AUTH_SOCK points to it ssh-add -l
Config order Command line, then ~/.ssh/config, then /etc/ssh/ssh_config; the first value found wins ssh -G host
Bastion ProxyJump (-J) connects through a jump host without copying keys to it ssh -J gw web
Scripts BatchMode=yes fails instead of prompting; ConnectTimeout bounds the wait ssh -o BatchMode=yes host true
Exit status The remote command's status, or 255 for an SSH error ssh host false; echo $?
Escape keys ~. kills a hung session, ~? lists escapes; only after a newline type Enter ~ .

SSH Tunnels

Fact Value Verify with
-L lport:host:hport Local forward: the client listens on lport, the server connects to host:hport ss -tlnp 'sport = :lport' on the client
-R rport:host:hport Remote forward: the server listens on rport, the client connects to host:hport ss -tln on the server
-D port Dynamic forward: a SOCKS proxy on the client; the server connects wherever the application asks curl --socks5-hostname 127.0.0.1:port URL
Destination view In -L, host is resolved and reached from the server, so 127.0.0.1 means the server itself ssh -L 9000:127.0.0.1:8008 web
Bind address Forwards listen on loopback by default; GatewayPorts (server) and -g or a bind address (client) change that ss -tln
Background -f goes to the background after authentication, -N runs no remote command pgrep -af 'ssh -f'
Failure handling A failed forward is only a warning unless ExitOnForwardFailure=yes echo $?
Server switch AllowTcpForwarding (default yes) and PermitOpen limit forwarding in sshd sudo sshd -T
Config form LocalForward, RemoteForward and DynamicForward in ~/.ssh/config ssh -G host

sshd Server

Fact Value Verify with
Files /etc/ssh/sshd_config, drop-ins in /etc/ssh/sshd_config.d/*.conf (included near the top), host keys /etc/ssh/ssh_host_*_key grep -n Include /etc/ssh/sshd_config
Precedence The first value read for an option wins, so an early drop-in beats the main file sudo sshd -T
Test and dump sshd -t checks syntax; sshd -T prints the effective settings; -C user=,host=,addr= evaluates Match blocks sudo sshd -t && echo ok
Apply systemctl reload sshd (RHEL) or ssh (Ubuntu 24.04, started by ssh.socket); existing sessions survive a reload systemctl list-sockets
Key settings PermitRootLogin (default prohibit-password, shown as without-password), PasswordAuthentication (default yes), AllowUsers, AllowGroups, MaxAuthTries (default 6) sudo sshd -T
Penalties OpenSSH 9.8 and later delay sources that fail authentication (PerSourcePenalties) sudo sshd -T
SFTP jail Match Group with ChrootDirectory and ForceCommand internal-sftp; the chroot path must be owned by root and not group- or world-writable ls -ld /srv/sftp/*
Brute force Key-only login, then fail2ban bans repeated failures (EPEL on RHEL) sudo fail2ban-client status sshd

File Transfer

Fact Value Verify with
scp syntax scp SRC [user@]host:DEST, scp host:SRC DEST; -r recursive, -p keep times and modes, -P port scp -v file host:/tmp/
scp protocol OpenSSH 9.0 and later use SFTP underneath; -O selects the old SCP protocol scp -v shows Sending subsystem: sftp
rsync basics -a archive (recursive, links, modes, times, owner for root), -v verbose, -z compress, -P = --partial --progress rsync -av src/ host:dst/
Trailing slash rsync src host:dst creates dst/src; rsync src/ host:dst copies the contents find dst
Deletion --delete removes files missing from the source; always preview with -n rsync -avn --delete src/ dst/
Change codes -i prints one line per change, such as *deleting or >f.st...... rsync -ai src/ dst/
Both ends rsync must be installed on the remote host too ssh host rsync --version
sftp Interactive or batch (-b file, -b -) file sessions; no remote shell needed sftp -b - host
Streams tar czf - dir piped into ssh host 'tar xzf - -C /dst' copies a tree over one connection ssh host 'ls /dst'
Through a bastion All three honor ProxyJump; rsync -e 'ssh -J gw' for one-offs rsync -e 'ssh -J gw' ...

SSH Troubleshooting

Fact Value Verify with
Client debug -v, -vv, -vvv add detail; the last debug1 line before the error names the stage ssh -v host true
Server log RHEL: unit sshd, file /var/log/secure; Ubuntu: unit ssh, file /var/log/auth.log sudo journalctl -u sshd -n 20
Connection refused The host answered, nothing listens on the port (or a firewall rejects) nc -vz host 22
Connection timed out No answer: a firewall drops the packets, or the route is broken ssh -o ConnectTimeout=5 host
Host key verification failed The stored key does not match; check why before ssh-keygen -R host ssh-keygen -F host
Permission denied (publickey) No offered key was accepted; the list in brackets is what the server allows ssh -v
StrictModes Home, ~/.ssh and authorized_keys not writable by group or others: 755 or stricter, 700, 600 namei -l ~/.ssh/authorized_keys
SELinux authorized_keys must be labeled ssh_home_t; mv keeps the old label ls -Z ~/.ssh
Too many keys Each offered key counts against MaxAuthTries ssh-add -l
Slow login Reverse DNS (UseDNS yes) or GSSAPI waiting on an unreachable server sudo sshd -T
Penalties OpenSSH 9.8 and later refuse connections from a source that keeps failing sudo journalctl -u sshd -g penalty

Security

firewalld and ufw

Fact Value Verify with
Defaults RHEL: firewalld on, zone public allows ssh, dhcpv6-client, cockpit; Ubuntu: ufw installed but inactive sudo firewall-cmd --list-all, sudo ufw status
Backend Both write nftables rules (firewalld table inet firewalld; ufw through iptables-nft) sudo nft list tables
Runtime vs permanent firewall-cmd changes runtime only; --permanent changes the saved config; --reload drops runtime-only rules; --runtime-to-permanent saves them sudo firewall-cmd --permanent --list-all
Zones A packet uses the zone of its source address first, then the zone of its interface, then the default zone sudo firewall-cmd --get-active-zones
Services Named port sets in /usr/lib/firewalld/services/*.xml (263 on Rocky 10.2); zone config in /etc/firewalld/zones/public.xml sudo firewall-cmd --get-services
Rich rules Per-source or logged rules; deny rules run before allow rules unless priority is set sudo firewall-cmd --list-rich-rules
Reject vs drop firewalld rejects (clients see refused or unreachable); ufw deny drops (clients time out) nc -vz host port
ufw rules ufw allow 22/tcp, ufw allow from 10.0.0.0/8 to any port 5432 proto tcp, ufw delete allow 22/tcp; blocks are logged as [UFW BLOCK] sudo ufw status numbered

nftables and iptables

Fact Value Verify with
Hooks prerouting, input, forward, output, postrouting; local traffic uses input, routed traffic uses forward sudo nft list ruleset
Objects Table (family ip, ip6, inet, ...) holds chains; a base chain has a hook, priority and policy; rules have handles sudo nft -a list tables
iptables today RHEL 9/10 and Ubuntu 24.04 ship iptables-nft, which writes nftables tables ip filter, ip nat; iptables-translate prints the nft form iptables -V shows (nf_tables)
Stateful ct state established,related accept lets replies through; conntrack tracks each flow sudo conntrack -L
Evaluation Every base chain on a hook runs; a packet must be accepted by all of them (firewalld, Docker and custom tables together) sudo nft list chains
NAT masquerade or snat in postrouting (source), dnat in prerouting (destination, port forwarding); routing also needs ip_forward=1 sudo nft list table ip nat
Sets Named address or port sets, optionally with timeout, replace long rule lists and ipset sudo nft list sets
Persistence RHEL: /etc/sysconfig/nftables.conf + nftables.service; Ubuntu: /etc/nftables.conf; iptables: iptables-save with iptables-persistent systemctl cat nftables

SELinux

Fact Value Verify with
Modes enforcing (deny and log), permissive (log only, a diagnostic step), disabled; setenforce 0/1 switches until reboot getenforce
Config /etc/selinux/config: SELINUX= and SELINUXTYPE=targeted; full disable needs the kernel option selinux=0 sestatus
Context user:role:type:level; the type decides almost everything in the targeted policy ls -Z, ps -Z, id -Z
Type enforcement A process type (domain, nginx: httpd_t) may access a file type (web content: httpd_sys_content_t, writable: httpd_sys_rw_content_t) only if an allow rule exists sesearch --allow -s httpd_t
Labels New files inherit the parent directory's type; mv keeps the old label, cp takes the new one; touch /.autorelabel and a reboot relabel everything ls -Z
Fix a label semanage fcontext -a -t TYPE 'PATH(/.*)?' then restorecon -Rv PATH; chcon is temporary matchpathcon PATH
Ports Daemons may bind only ports with their type; semanage port -a -t http_port_t -p tcp 8090 semanage port -l
Booleans Policy switches such as httpd_can_network_connect; -P makes them persistent getsebool -a
Logs AVC denials in /var/log/audit/audit.log; ausearch -m AVC -ts recent, audit2why, sealert sudo ausearch -m AVC

AppArmor

Fact Value Verify with
Model Profiles attach to executables by path; rules name files (r, w, ix, ...), capabilities and network types ls /etc/apparmor.d
Modes enforce (deny and log apparmor="DENIED") or complain (log only), per profile; the kernel must run AppArmor as its active LSM sudo aa-status
Tools aa-enforce, aa-complain, aa-disable, apparmor_parser -r (reload), aa-genprof and aa-logprof (build from logs) sudo journalctl -k -g apparmor
Containers Docker and containerd apply docker-default; --security-opt apparmor=unconfined removes it docker inspect -f '{{.AppArmorProfile}}' ID

Capabilities

Fact Value Verify with
Process sets Effective (checked now), Permitted (may enable), Inheritable, Bounding (upper limit), Ambient (kept across execve for non-root) grep Cap /proc/self/status
Root A root process has all capabilities in its effective set; a normal user has none sudo grep CapEff /proc/self/status
Decode capsh --decode=HEX turns a mask into names capsh --decode=0000000000000400
File capabilities Stored in the security.capability extended attribute; setcap cap_x=+ep FILE getcap FILE
Copies cp without --preserve=xattr drops file capabilities; so does rewriting the file getcap on the copy
Low ports Ports below net.ipv4.ip_unprivileged_port_start (1024) need CAP_NET_BIND_SERVICE sysctl net.ipv4.ip_unprivileged_port_start
ping Ubuntu: file capability cap_net_raw; RHEL: unprivileged ICMP sockets via net.ipv4.ping_group_range getcap /usr/bin/ping
systemd AmbientCapabilities= grants, CapabilityBoundingSet= limits; User= alone drops everything systemctl show -p CapabilityBoundingSet UNIT
Dropping capsh --drop=cap_x -- -c CMD runs a command without that capability, even as root sudo capsh --print
Containers Docker keeps 14 capabilities by default; --cap-drop ALL --cap-add NET_BIND_SERVICE is the least-privilege pattern docker inspect
Dangerous ones CAP_SYS_ADMIN (mount, many admin calls), CAP_SYS_PTRACE, CAP_DAC_OVERRIDE, CAP_SETUID are almost root man 7 capabilities

auditd

Fact Value Verify with
Pieces Kernel audit subsystem, auditd daemon, rules in /etc/audit/rules.d/*.rules compiled by augenrules; package audit (RHEL, enabled) or auditd (Ubuntu) sudo auditctl -s
Rules Watch: -w PATH -p rwxa -k KEY; syscall: -a always,exit -F arch=b64 -S execve -F euid=0 -F auid>=1000 -k KEY sudo auditctl -l
auid The login UID, set at login and kept through sudo and su; unset for services cat /proc/self/loginuid
Search ausearch -k KEY -i, -m AVC, -ts recent (10 minutes) or today, --format text; reports with aureport -au, -k, -x sudo aureport -k --summary

GPG

Fact Value Verify with
Sign, encrypt gpg --armor --detach-sign FILE, gpg --verify FILE.asc FILE; gpg -e -r RECIPIENT FILE or gpg -c FILE, gpg -d to decrypt echo $? after verify
Keys ~/.gnupg/ holds the keyring and, from creation, a revocation certificate; gpg --armor --export ID shares the public key gpg --show-keys key.asc
RPM Keys imported with rpm --import, listed as gpg-pubkey packages; gpgcheck=1 and gpgkey= per repo rpm -q gpg-pubkey
APT Repository keys in /usr/share/keyrings/ or /etc/apt/keyrings/, bound per repo with Signed-By, so a vendor key cannot sign other repositories; apt-key is deprecated grep Signed-By /etc/apt/sources.list.d/*

OpenSSL and Trust Store

Fact Value Verify with
Chain Server certificate, then intermediates, signed up to a root the client trusts; the server sends leaf and intermediates openssl s_client -showcerts
Names Clients match the host name against subjectAltName; the CN alone is ignored openssl x509 -noout -ext subjectAltName
Inspect openssl x509 -in FILE -noout -subject -issuer -dates openssl x509 -text
Expiry check -checkend SECONDS exits 1 if the certificate expires within that time openssl x509 -checkend 2592000
Key match Public key hash of certificate and private key must be equal openssl pkey -pubout
Test a server openssl s_client -connect HOST:PORT -servername NAME; -servername sends SNI curl -v https://HOST
Formats PEM (Base64, -----BEGIN), DER (binary), PKCS#12 (.p12/.pfx, key and chain in one file) openssl x509 -inform der
RHEL trust Drop CA files in /etc/pki/ca-trust/source/anchors/, run update-ca-trust; bundle /etc/pki/tls/certs/ca-bundle.crt trust list --filter=ca-anchors
Ubuntu trust Drop *.crt files in /usr/local/share/ca-certificates/, run update-ca-certificates; bundle /etc/ssl/certs/ca-certificates.crt ls -l /etc/ssl/certs
Crypto policy RHEL sets allowed protocols and key sizes system-wide update-crypto-policies --show
Private keys Mode 600, owned by root or the service user; never in a repository ls -l *.key

Compliance and Integrity

Fact Value Verify with
Package verify rpm -Va and dpkg --verify: S size, 5 digest, M mode, U user, T mtime, P capabilities; c marks config files; debsums -s on Debian sudo rpm -Va
AIDE aide --init builds a baseline, aide --check compares (nonzero exit on differences), aide --update accepts changes; keep the database off the host; rkhunter and chkrootkit are signature-based and prove little when clean sudo aide --check
OpenSCAP oscap xccdf eval --profile ID --report FILE DATASTREAM with content from scap-security-guide oscap info --profiles FILE
Backporting RHEL and Ubuntu fix CVEs in the shipped version, so the upstream version says nothing; dnf updateinfo list --security, Ubuntu unattended-upgrades rpm -q --changelog PKG

Hardening Checklist

Fact Value Verify with
Updates Security updates applied; automatic on servers that allow it (dnf-automatic, unattended-upgrades) sudo dnf updateinfo list --security
Accounts Only root has UID 0; no empty password fields; service accounts use nologin awk -F: '$3 == 0' /etc/passwd
sudo Narrow command rules; NOPASSWD: ALL only where justified sudo -l -U USER
setuid Inventory setuid files and remove unneeded ones; world-writable directories carry the sticky bit find / -xdev -perm -4000 -type f
Services Every listening socket is known and needed sudo ss -tulpn
SSH Keys only, no root login, MaxAuthTries low, a legal banner sudo sshd -T
Firewall, MAC Default deny for incoming traffic; SELinux enforcing or AppArmor profiles loaded getenforce
Kernel kptr_restrict, dmesg_restrict, rp_filter on; ICMP redirects off; ASLR 2 sysctl -a
Filesystems Unused modules (cramfs, udf, usb-storage on servers) blocked with install ... /bin/false modprobe -n -v cramfs
Logging, audits auditd and time sync active, logs shipped off the host; Lynis or OpenSCAP scans systemctl is-active auditd chronyd

Boot and Recovery

Boot Process

Fact Value Verify with
Firmware UEFI (modern) or BIOS (legacy) starts the bootloader [ -d /sys/firmware/efi ] && echo UEFI
Bootloader GRUB2 on most distributions; loads kernel and initramfs /boot/grub2/grub.cfg
Kernel args Passed by the bootloader, seen at runtime cat /proc/cmdline
initramfs Temporary root with drivers to reach the real root lsinitrd / lsinitramfs
switch_root Pivot from initramfs to the real root filesystem in /proc/cmdline root=
init PID 1 is systemd; brings the system to a target ps -p 1 -o comm=
Default target The target booted by default systemctl get-default
Boot timing Kernel plus userspace startup time systemd-analyze
Slowest units Per-unit startup time systemd-analyze blame
Critical path The dependency chain that gated boot systemd-analyze critical-chain
Secure Boot Firmware verifies the bootloader and kernel signatures mokutil --sb-state

GRUB2

Fact Value Verify with
Config file grub.cfg is generated; do not edit it directly head /boot/grub2/grub.cfg
Source /etc/default/grub plus scripts in /etc/grub.d cat /etc/default/grub
Regenerate (RHEL) grub2-mkconfig -o the active grub.cfg after editing defaults
Regenerate (Ubuntu) update-grub (wraps grub2-mkconfig) after editing defaults
Persistent kernel args GRUB_CMDLINE_LINUX in /etc/default/grub grep CMDLINE /etc/default/grub
One-boot edit Press e at the menu, edit the linux line, Ctrl-x at the console
Active args What the running kernel actually got cat /proc/cmdline
Default entry GRUB_DEFAULT, or grub2-editenv list for saved grub2-editenv list
Change default grubby --set-default (RHEL) grubby --default-kernel
Menu password grub2-setpassword protects editing entries /boot/grub2/user.cfg
Timeout GRUB_TIMEOUT seconds before the default boots /etc/default/grub

Recovery

Fact Value Verify with
Rescue target Minimal system, root shell, local mounts systemctl rescue
Emergency target Bare minimum, read-only root, no mounts systemctl emergency
Boot into a target Append systemd.unit=rescue.target at GRUB at the menu
Reset root password Append rd.break, remount, passwd, relabel RHEL method
SELinux relabel .autorelabel or load_policy -i after a chroot edit touch /.autorelabel
Bad fstab A missing device without nofail drops boot to emergency journalctl -b
Test fstab Validate before reboot sudo mount -a
Failed units List what did not start systemctl --failed
System state running, degraded (a unit failed), maintenance systemctl is-system-running
Read boot logs The last boot's journal journalctl -b
kdump Captures a kernel crash dump for analysis systemctl status kdump

Kernel Panic

Fact Value Verify with
Oops Kernel error in one context; logs and may continue dmesg, journalctl -k
Panic Kernel halts; nothing runs after on-console message
Common panic Unable to mount root fs, init died, hardware fault /proc/cmdline root=
Oops to panic An oops in an atomic context escalates to a panic kernel.panic_on_oops
kernel.panic Seconds to auto-reboot after panic; 0 means halt sysctl kernel.panic
kernel.panic_on_oops 1 makes any oops a panic (fail fast) sysctl kernel.panic_on_oops
Magic SysRq Kernel-level key combos for a hung system cat /proc/sys/kernel/sysrq
SysRq bitmask A number enables a subset of functions sysctl kernel.sysrq
kdump Boots a capture kernel to save a crash dump systemctl status kdump
netconsole Streams kernel messages over UDP to another host modinfo netconsole
Tainted Flag showing non-standard modules loaded cat /proc/sys/kernel/tainted

Kernel Updates

Fact Value Verify with
Running kernel The one booted now uname -r
Installed kernels Packages present, possibly newer than running rpm -q kernel / dpkg -l 'linux-image*'
initramfs Rebuilt per kernel on install dracut / update-initramfs
needs-restarting Reports whether a reboot is required (RHEL, dnf-utils) needs-restarting -r
/boot full Too many kernels fill a small /boot, breaking updates df -h /boot
Livepatch Applies some fixes without reboot kpatch / Ubuntu Livepatch
Rollback Boot the previous kernel from the GRUB menu at the menu

Performance and Troubleshooting

Methodology

Fact Value Verify with
USE method For each resource check Utilization, Saturation, Errors vmstat 1, iostat -xz 1, dmesg
The four resources CPU, memory, disk I/O, network top, free, iostat, ss -s
60-second checklist uptime, dmesg, vmstat 1, mpstat -P ALL 1, pidstat 1, iostat -xz 1, free -m, sar -n DEV 1, top run in order
Saturation signal Work queued behind a busy resource: run queue, aqu-sz, swap-in vmstat r, iostat aqu-sz
Utilization trap 100% busy is not always the limit; a queue behind it is %util with aqu-sz
Live vs history vmstat/iostat show now; sar reads the archive from sysstat sar -f /var/log/sa/saNN
sysstat collector sar needs the collector enabled and running systemctl status sysstat
Load average Runnable plus uninterruptible (D) tasks over 1, 5, 15 min uptime, cat /proc/loadavg
First move Read dmesg for OOM, resets and errors before tuning anything dmesg -T
Baseline A number means nothing without the normal value to compare it to keep sar history
Package sar, iostat, mpstat, pidstat come from sysstat rpm -q sysstat / dpkg -l sysstat

CPU and Load

Fact Value Verify with
Load average Runnable plus uninterruptible (D) tasks, averaged over 1, 5, 15 min uptime, cat /proc/loadavg
Load vs cores Load equal to core count is full; above it, tasks wait nproc
Run queue vmstat r above CPU count means CPU saturation vmstat 1
us User-space CPU time (application code) mpstat -P ALL 1
sy Kernel CPU time (syscalls, drivers) mpstat
wa Idle CPU waiting on I/O; high wa is a disk problem vmstat, iostat
st Steal: time the hypervisor gave to another guest vmstat, top
id Idle: CPU with nothing to run mpstat
Context switch cswch/s voluntary (blocked), nvcswch/s involuntary (preempted) pidstat -w 1
High sy Excessive syscalls, interrupts or lock contention pidstat, strace -c
cgroup throttling cpu.max caps a group; throttling appears in cpu.stat cat /sys/fs/cgroup/.../cpu.stat
/proc/loadavg Load trio, running/total tasks, last PID cat /proc/loadavg

Memory

Fact Value Verify with
available vs free available estimates what a new process can get, including reclaimable cache; free is unused free -h
buff/cache Page cache and buffers; reclaimable under pressure, not waste free -h, /proc/meminfo
Low free is normal Linux uses spare RAM for cache; judge by available free -h
Swap Overflow to disk; si/so in vmstat show active paging vmstat 1, free -h
swappiness 0 to 200, higher favours swapping anon pages; default 60 cat /proc/sys/vm/swappiness
RSS Resident set: physical RAM a process uses now /proc/PID/status VmRSS
VSZ Virtual size: address space reserved, not all backed by RAM /proc/PID/status VmSize
PSS Proportional set: RSS with shared pages divided among sharers /proc/PID/smaps_rollup
OOM killer Frees memory by killing a task; picks by oom_score dmesg, /proc/PID/oom_score
oom_score_adj -1000 (never kill) to 1000 (kill first) cat /proc/PID/oom_score_adj
Exit 137 128 + 9 (SIGKILL); a container OOM shows this echo $? after a kill
Committed_AS Total memory promised to all processes grep Committed /proc/meminfo

Virtual Memory

Fact Value Verify with
Virtual address space Per-process, mapped to physical frames lazily /proc/PID/maps
Page Fixed unit of memory, 4 KiB on x86_64 getconf PAGE_SIZE
Page table + TLB Maps virtual to physical; TLB caches recent translations /proc/PID/status VmPTE
Minor fault Page mapped without disk I/O (cache hit, zero page) /proc/PID/stat, time -v
Major fault Page required a disk read (file or swap-in) time -v, vmstat si
Demand paging RAM is assigned when a page is first touched, not at allocation smaps after touching
RSS vs VSZ RSS is mapped-and-resident; VSZ is the whole address space /proc/PID/status
Overcommit Kernel may promise more than RAM plus swap sysctl vm.overcommit_memory
Committed_AS Sum of memory promised; compared to CommitLimit grep Commit /proc/meminfo
Page cache File pages kept in RAM; Dirty awaits writeback grep Dirty /proc/meminfo
fsync Forces dirty pages to stable storage strace -e fsync
THP Transparent huge pages, 2 MiB, reduce TLB misses /sys/kernel/mm/transparent_hugepage/enabled
Slab Kernel object cache (inodes, dentries) slabtop -o

Disk I/O

Fact Value Verify with
Tool iostat -xz (extended, skip idle devices), from sysstat iostat -xz 1
r/s w/s Reads and writes completed per second (IOPS) iostat -xz 1
rkB/s wkB/s Throughput in KB per second iostat -xz 1
r_await w_await Average ms per read/write, including queue time iostat -xz 1
aqu-sz Average queue depth; high means requests are waiting iostat -xz 1
%util Fraction of time the device had at least one request iostat -xz 1
%util caveat 100% is not saturation on SSDs and arrays that serve in parallel pair with aqu-sz
Per-process I/O iotop needs task_delayacct enabled iotop -o
iowait CPU idle time waiting on I/O; a symptom, not a cause iostat, vmstat wa
First check iostat -xz 1, then find the writer with iotop or pidstat -d run in order
Deleted open file Space held by a process still writing to an unlinked file lsof +L1

Limits and File Descriptors

Fact Value Verify with
Soft vs hard Soft is the enforced limit; hard is the ceiling a user may raise the soft to ulimit -Sn, ulimit -Hn
Open files (-n) Per-process file descriptor limit; default soft often 1024 ulimit -n
EMFILE Too many open files: this process hit its -n limit /proc/PID/limits
ENFILE System-wide file table full (rare); governed by fs.file-max sysctl fs.file-max
file-nr Allocated, unused, and max open file handles system-wide cat /proc/sys/fs/file-nr
Processes (-u) RLIMIT_NPROC: max processes per real user id ulimit -u
EAGAIN on fork Resource temporarily unavailable: hit -u or pid_max ulimit -u
pid_max System-wide ceiling on process ids cat /proc/sys/kernel/pid_max
Shell limits ulimit sets the calling shell and its children ulimit -a
PAM limits /etc/security/limits.conf and limits.d for login sessions man limits.conf
systemd limits LimitNOFILE=, LimitNPROC= in the unit; ulimit does not reach services systemctl show -p LimitNOFILE UNIT
Live limits A running process shows its effective limits cat /proc/PID/limits

Profiling and Tracing

Fact Value Verify with
strace Traces syscalls of one process; high overhead strace -f -p PID
strace -c Summarises syscall counts, time and errors strace -c cmd
strace cost Stops the target on every syscall; not for hot production paths measure before use
perf stat Hardware and software counters for a command perf stat -- cmd
perf record Samples the CPU; perf report shows hot functions perf record -g -- cmd
perf top Live system-wide CPU profile perf top
Sampling vs tracing perf samples periodically (low overhead); strace traces every event choose by overhead
eBPF In-kernel programs for low-overhead tracing bpftrace, bcc tools
bcc tools execsnoop, opensnoop, biolatency, tcplife execsnoop-bpfcc
Flame graph Folded perf stacks rendered as nested bars perf script + FlameGraph
Which tool syscalls → strace; CPU → perf; system-wide, low cost → eBPF match to the question
Symbols Profiling needs debug symbols to name functions -debuginfo/-dbgsym

Monitoring and Capacity

Fact Value Verify with
SLI A measured indicator: latency, error rate, availability dashboards, logs
SLO The target for an SLI, for example 99.9% success agreed internally
SLA The contract with consequences if the SLO is missed contract
Error budget The allowed failure under the SLO (0.1% for 99.9%) derived from SLO
Health check A cheap probe of liveness and readiness curl, systemctl is-active
Liveness vs readiness Alive means the process runs; ready means it can serve probe endpoints
Agent A collector runs on each host (node_exporter, the Datadog agent) on-host service
Agentless A central poller scrapes or SSHes in central config
Baseline The normal value a metric is judged against sar history
Headroom Spare capacity kept for spikes and failures capacity plan

Tuning

Fact Value Verify with
tuned Daemon that applies workload profiles (RHEL family) tuned-adm active
List profiles Show available and current profile tuned-adm list
Recommend Suggest a profile for this host tuned-adm recommend
Apply Switch profile, persists across reboot tuned-adm profile NAME
Common profiles throughput-performance, latency-performance, virtual-guest, powersave tuned-adm list
sysctl Set individual kernel parameters sysctl -w, /etc/sysctl.d
Persist sysctl A .conf in /etc/sysctl.d, applied with sysctl --system sysctl --system
Measure first Tune the proven bottleneck, one change at a time vmstat, iostat
Ubuntu No tuned by default; use sysctl and scheduler settings sysctl -a

Network Storage

NFS

Fact Value Verify with
Server package nfs-utils (RHEL), nfs-kernel-server (Ubuntu) rpm -q nfs-utils
Client package nfs-utils (RHEL), nfs-common (Ubuntu) mount.nfs -V
Exports file /etc/exports defines shared directories and who may mount cat /etc/exports
Apply exports Re-read and show the export table exportfs -rav
Current exports What the server is exporting now exportfs -s
List from client Show a server's exports showmount -e SERVER
Default version NFSv4.2 over TCP on modern distros findmnt -o FSTYPE,OPTIONS
Port NFSv4 uses a single port, 2049 ss -tlnp | grep 2049
hard mount Retries forever if the server stops; process blocks in D mount option hard
soft mount Fails I/O after retries; risks data corruption mount option soft
Hang signature D state, WCHAN rpc_wait_bit_killable, load rising ps -o stat,wchan
Boot-safe mount _netdev and nofail in /etc/fstab man nfs

Autofs

Fact Value Verify with
Package autofs on both families rpm -q autofs
Master map /etc/auto.master and /etc/auto.master.d/* cat /etc/auto.master
Mount point Master map ties a directory to a map file ls /etc/auto.master.d
Direct map Absolute paths, key /- man 5 autofs
Indirect map A base directory with keys as subdirectories the map file
On-demand Mounts on access, not at boot findmnt after ls
Idle timeout Unmounts after inactivity (default 300s) --timeout
Service Reload maps by restarting the service systemctl restart autofs

Samba and CIFS

Fact Value Verify with
Protocol SMB (formerly CIFS); default SMB3 mount -o vers=3.0
Client package cifs-utils on both families rpm -q cifs-utils
Server config /etc/samba/smb.conf, one section per share testparm -s
Samba user A separate password store, not /etc/passwd smbpasswd -a
Mount type -t cifs, with credentials= for the password findmnt -t cifs
Credentials file username=/password=, mode 600 chmod 600

iSCSI and NBD

Fact Value Verify with
Exports A block device, not a filesystem lsblk -S on the client
Target / initiator Server side / client side targetcli / iscsiadm
IQN Name identifying a target or initiator /etc/iscsi/initiatorname.iscsi
ACL Restricts a target to specific initiator IQNs (TCP 3260) targetcli .../acls

Containers

Namespaces

Fact Value Verify with
Count Eight types: mnt, pid, net, uts, ipc, user, cgroup, time ls /proc/self/ns/
mnt Isolates the mount table (each container's own filesystem view) unshare --mount
pid Isolates process ids; the first process becomes PID 1 unshare --pid --fork
net Isolates interfaces, routes, ports, firewall unshare --net
uts Isolates hostname and domain name unshare --uts
ipc Isolates System V IPC and POSIX message queues unshare --ipc
user Isolates uid/gid; maps container root to an unprivileged host uid unshare --user
cgroup Isolates the cgroup root the process sees unshare --cgroup
Identity A namespace is an inode number under /proc/PID/ns/ ls -l /proc/self/ns/net
List them Show namespaces and the processes in each lsns
Create Start a process in new namespaces unshare
Join Enter the namespaces of a running process nsenter -t PID
Syscalls clone, unshare, setns create and join namespaces man 7 namespaces

Cgroups

Fact Value Verify with
Version v2 is the unified hierarchy; v1 had a tree per controller stat -fc %T /sys/fs/cgroup
Mount point A single tree at /sys/fs/cgroup mount | grep cgroup2
Controllers cpu, memory, io, pids, cpuset, hugetlb cat /sys/fs/cgroup/cgroup.controllers
Enable in children Write to cgroup.subtree_control of the parent echo +memory > .../cgroup.subtree_control
Add a process Write its PID to cgroup.procs echo $$ > .../cgroup.procs
CPU limit cpu.max = quota period, in microseconds echo "20000 100000" > cpu.max
CPU throttling cpu.stat counts throttled periods grep throttled .../cpu.stat
Memory limit memory.max is a hard cap; exceeding it triggers cgroup OOM cat .../memory.max
OOM record memory.events counts oom and oom_kill cat .../memory.events
Process limit pids.max caps the number of tasks cat .../pids.max
systemd slices systemd is the cgroup manager; units are scopes and slices systemd-cgls
Exit 137 128 + SIGKILL(9): the OOM killer killed the process dmesg, container exit code

Overlayfs and Chroot

Fact Value Verify with
chroot Changes / for a process and its children chroot <dir> <cmd>
chroot limit Not a security boundary on its own; root can escape man 2 chroot
pivot_root The real container root switch; replaces the mount root man 8 pivot_root
OverlayFS Union filesystem: lower (read-only) under upper (writable) mount -t overlay
lowerdir One or more read-only layers (image layers) mount option
upperdir The single writable layer (container changes) mount option
workdir Empty scratch dir OverlayFS needs on the upper's filesystem mount option
merged The combined view processes actually use the mountpoint
Copy-up Writing a lower file copies it into the upper first edit a merged file
Image layers Each image layer is a lowerdir; the container adds an upperdir podman inspect
Whiteout A deleted lower file is masked by a whiteout in the upper ls -l upperdir

Containers vs VMs

Fact Value Verify with
Container A host process with its own namespaces and cgroups ps on the host finds it
VM A guest kernel on virtual hardware, via a hypervisor virsh list
Kernel Containers share the host kernel; VMs run their own uname -r inside each
Isolation Namespaces plus cgroups vs hardware virtualization /proc/PID/ns/
Startup Container in milliseconds; VM boots a kernel time podman run
Density Many containers per host; fewer VMs memory per instance
Boundary strength A VM's boundary is stronger (own kernel) threat model
Image Container image is layered files; VM image is a full disk podman inspect
Container PID 1 The entrypoint process; must reap and handle signals ps inside
No init by default A bare entrypoint does not reap zombies --init adds one
Guest OS A container has no kernel of its own to boot no init boot logs

Podman and Quadlet

Fact Value Verify with
Daemonless No central daemon; each command is a process podman version
Rootless Runs as a normal user by default podman info | grep rootless
subuid/subgid Ranges a user may map into a container grep $USER /etc/subuid
uid mapping Container root maps to the user; other ids to the subuid range cat /proc/PID/uid_map
Docker CLI podman is command-compatible with docker alias docker=podman
Image build Containerfile (or Dockerfile) with podman build podman build -t x .
Volume Persist data outside the container layer podman volume create
SELinux label :Z relabels a bind mount for the container -v /data:/data:Z
skopeo Inspect and copy images between registries skopeo inspect
Quadlet .container unit files run containers as systemd services systemctl start x
Quadlet path /etc/containers/systemd/ (system), ~/.config/containers/systemd/ (user) drop the unit there

Virtualization and Provisioning

KVM and libvirt

Fact Value Verify with
KVM Kernel module using CPU virtualization (VT-x, AMD-V) lsmod | grep kvm
CPU support vmx (Intel) or svm (AMD) flag, and /dev/kvm grep -E 'vmx|svm' /proc/cpuinfo
QEMU Emulates the virtual hardware for the guest qemu-system-x86_64 --version
libvirt Management API and daemon over KVM/QEMU systemctl status libvirtd
virsh CLI to libvirt: list, start, define domains virsh list --all
Host check Validates KVM readiness virt-host-validate
Default network NAT network default, 192.168.122.0/24 virsh net-list
Snapshot Point-in-time state of a domain virsh snapshot-create-as

VM Images and Cloning

Fact Value Verify with
qcow2 QEMU copy-on-write: thin, snapshots, backing files qemu-img info disk.qcow2
raw Full flat file; fastest, no thin provisioning qemu-img info disk.raw
Convert Change format between qcow2 and raw qemu-img convert
virtio Paravirtual disk and network drivers, faster than emulated lspci in guest
Template A cleaned base image cloned for new VMs virt-clone, virt-sysprep
machine-id Must be unique per host; reset on clone cat /etc/machine-id
SSH host keys Must be unique per host; regenerate on clone ssh-keygen -A
virt-sysprep Cleans an image (logs, keys, machine-id) for templating virt-sysprep -a img

Cloud-init and Kickstart

Fact Value Verify with
cloud-init Configures a cloud image on first boot cloud-init status
user-data The #cloud-config YAML: users, packages, commands cloud-init schema
First-boot only Runs once per instance, keyed by instance-id /var/lib/cloud/instance
Validate Check a cloud-config before use cloud-init schema --config-file
Re-run Reset state so next boot is a first boot cloud-init clean
Kickstart Automates a full RHEL install (Anaconda) inst.ks=URL boot arg
bootc Image-mode RHEL: boot from an OCI image, transactional bootc status