Round 1: Screening¶

Say first: strictly, Linux is the kernel; an operating system also needs a C library, shell and tools, which distributions supply.

Proof: uname -s names the kernel; /etc/os-release names the distribution.

Follow-up: What does a distribution add on top of the kernel?

Say first: if the company distributes the modified kernel, it must provide the source under GPL version 2; internal use carries no obligation.

Proof: the kernel's COPYING file; rpm -q --qf '%{LICENSE}' rpm.

Follow-up: How is that different from software under the MIT license?

Say first: Linux is a Unix-like clone written from scratch; it follows POSIX but contains no Unix source code.

Proof: getconf _POSIX_VERSION reports the POSIX level the C library supports.

Follow-up: Name a certified Unix still in use. (macOS, AIX.)

Say first: the kernel manages hardware and processes, the shell is a user program that starts other programs, and a distribution bundles the kernel, userland, package manager and support into an installable system.

Proof: uname -r, echo $0, cat /etc/os-release.

Follow-up: Which of the three does a container image contain?

Say first: most of the userland on a typical distribution (C library, core tools, shell, compiler) comes from the GNU project; Linux is the kernel.

Proof: ls --version prints "GNU coreutils"; ldd --version prints "GNU libc".

Follow-up: Name a Linux system that is not GNU/Linux. (Alpine with musl and BusyBox, Android.)

Say first: CentOS Stream is the development stream that the next RHEL minor release is cut from; RHEL is the supported product; Rocky and AlmaLinux rebuild RHEL for free.

Proof: cat /etc/redhat-release on each.

Follow-up: What changed for CentOS users in 2021?

Say first: fixed package versions with backported security fixes keep behavior stable for years, which matters more on servers than new features.

Proof: grep SUPPORT_END /etc/os-release shows the end of support.

Follow-up: What is the cost of that stability? (Older language runtimes and tools.)

Say first: Alpine is a few megabytes because it uses musl and BusyBox, which shrinks pull time and attack surface.

Proof: docker images alpine versus ubuntu.

Follow-up: What breaks when moving a glibc-built binary into Alpine?

Say first: name mechanisms: users work unprivileged and gain root per command through sudo, software comes from signed repositories, SELinux or AppArmor confine services, and the source is open to review.

Proof: ls -l /etc/shadow, rpm -q gpg-pubkey, cat /sys/kernel/security/lsm.

Follow-up: What are the most common ways Linux servers still get compromised?

Say first: no license cost per instance, full automation through text configuration and SSH, and containers are a Linux kernel feature.

Proof: ls /proc/self/ns shows the namespaces containers use.

Follow-up: How do Windows containers differ?

Say first: plain-text configuration files, mostly under /etc, plus per-user dotfiles in the home directory.

Proof: ls /etc/ssh/sshd_config /etc/fstab

Follow-up: What does that make easier for configuration management and version control?

Say first: kernel space runs with full hardware privilege in one shared address space; user space runs unprivileged in isolated address spaces and asks the kernel for everything through system calls.

Proof: strace cat /etc/hostname shows every request crossing the boundary.

Follow-up: What happens to the system when a user program dereferences a bad pointer, compared with a driver doing the same?

Say first: a system call is the controlled entry into the kernel; the CPU blocks direct device access from unprivileged mode, so the kernel can enforce permissions and share hardware.

Proof: strace -e trace=openat,read cat /etc/hostname

Follow-up: How does the kernel report a failed system call?

Say first: monolithic with loadable modules: drivers run in kernel space but can be loaded and unloaded at runtime.

Proof: lsmod; CONFIG_MODULES=y in /boot/config-$(uname -r).

Follow-up: What is the risk of that design, and what does the kernel do when a module faults?

Say first: /etc/os-release for the distribution and uname -r for the kernel; they are separate components with separate versions.

Proof: cat /etc/os-release; uname -r

Follow-up: Why can two different distributions report the same kernel release?

Say first: the kernel uses spare RAM as page cache and gives it back when programs need it; available is the number that matters.

Proof: free -h shows a large buff/cache and an available close to total.

Follow-up: When does low available become a problem?

Say first: the terminal carries text in and out, the shell interprets the commands, and the console is the machine's own terminal (screen or serial line).

Proof: tty shows /dev/pts/0 over SSH; ps -p $$ shows the shell.

Follow-up: What changes for a program started by cron, which has no terminal?

Say first: a login shell is the first shell of a session and reads profile files; an interactive shell reads commands from a user; SSH gives both, a script gets neither.

Proof: echo $- contains i; shopt login_shell reports on after bash -l.

Follow-up: Which startup files does each type read?

Say first: section 1 documents the passwd command; section 5 documents the /etc/passwd file format.

Proof: man -f passwd lists both sections.

Follow-up: Which sections hold system calls and admin commands?

Say first: alias, keyword, function, builtin, then the hash table and PATH.

Proof: type -a echo lists the builtin before the files.

Follow-up: Why can which give the wrong answer?

Say first: a child process cannot change its parent's working directory, so cd must run inside the shell.

Proof: type cd; /usr/bin/cd /tmp (where it exists) leaves the shell's directory unchanged.

Follow-up: Which other commands must be builtins for the same reason?

Say first: a shell variable stays in the current shell; an exported variable is copied into the environment of every program the shell starts.

Proof: X=1; bash -c 'echo $X' prints nothing; after export X, it prints 1.

Follow-up: Can a child process change a variable in its parent?

Say first: login shells read ~/.bash_profile (or ~/.profile on Ubuntu); interactive non-login shells read ~/.bashrc; the profile usually sources .bashrc so both get the same aliases.

Proof: strace -e trace=openat bash -l -i -c true lists the files opened.

Follow-up: Which of them does a cron job read? (Neither.)

Say first: a file named like a system command in a writable directory would run instead of the real one.

Proof: echo $PATH has no .; local files run as ./name.

Follow-up: What does an empty entry (::) in PATH mean?

Say first: it gives byte-order sorting, ASCII character classes and English messages, so output is the same on every machine and parsing does not break.

Proof: LC_ALL=C sort and LC_ALL=en_US.UTF-8 sort order the same file differently.

Follow-up: What does LC_ALL=C break? (Multibyte characters are counted as bytes.)

Say first: single quotes keep everything literal; double quotes still expand variables, command substitution and arithmetic, but stop word splitting and globbing.

Proof: echo '$HOME' versus echo "$HOME".

Follow-up: How do you put a single quote inside a single-quoted string? ('it'\''s'.)

Say first: the shell; ls receives a list of file names, or the literal *.log when nothing matches.

Proof: echo *.log; set -x; ls *.log.

Follow-up: Why does find . -name *.log sometimes fail? (The shell expands the unquoted pattern first.)

Say first: descriptors 0, 1 and 2; errors go to stderr so they reach the operator even when stdout is piped or saved.

Proof: ls /etc/hostname /nope | wc -l prints the error and counts only one line.

Follow-up: How do you send both streams through a pipe?

Say first: redirections apply left to right; the first sends both streams to the file, the second leaves stderr on the terminal.

Proof: the both.txt and order.txt demonstration above.

Follow-up: What is the bash shorthand for the first form?

Say first: the calling shell opens /etc/file before sudo runs, and that shell is not root.

Proof: echo text | sudo tee /etc/file

Follow-up: How do you append instead of overwrite? (sudo tee -a.)

Say first: success, general failure, found but not executable, not found, and killed by SIGKILL (128 + 9).

Proof: nosuchcmd; echo $? prints 127; sleep 30 & kill -9 $!; wait $!; echo $? prints 137.

Follow-up: Where do you see 137 most often in DevOps work?

Say first: ; always runs the next command, && only after success, || only after failure.

Proof: mkdir /tmp/x && cd /tmp/x; mkdir /tmp/x || echo exists.

Follow-up: Why is a && b || c not a safe if-then-else?

Say first: press Esc, then :wq to save and quit, or :q! to quit and discard changes.

Proof: :x also saves and quits, writing only if the file changed.

Follow-up: What does E37: No write since last change mean?

Say first: quoted, "$@" expands to each argument as a separate word, while "$*" joins them into one string.

Proof: ./args.sh web "db server" prints [db server] as one argument with "$@".

Follow-up: What happens to "db server" if $@ is not quoted?

Say first: [ is a command that follows normal word splitting; [[ is bash syntax that does not split variables and supports pattern and regex matching.

Proof: with y unset, [ $y = a ] errors while [[ $y == a ]] returns 1.

Follow-up: Which one works in /bin/sh on Ubuntu?

Say first: /etc holds configuration, /var holds data that changes at runtime, and /usr holds installed programs and libraries that change only on updates.

Proof: ls /etc/ssh, ls /var/log, rpm -qf /usr/sbin/sshd.

Follow-up: Which of them do you need to back up?

Say first: both are world-writable with the sticky bit; /tmp is for short-lived files and may be a tmpfs, while /var/tmp persists across reboots and is cleaned less often.

Proof: grep -v '^#' /usr/lib/tmpfiles.d/tmp.conf

Follow-up: What does the sticky bit prevent?

Say first: virtual filesystems generated by the kernel: /proc exposes processes and tunables, /sys exposes devices, drivers and cgroups.

Proof: findmnt -T /proc, cat /proc/loadavg, ls /sys/class/net.

Follow-up: Why does du -sh /proc report nothing useful?

Say first: devices, pipes, sockets and kernel state appear as paths and are used with the same open, read and write calls as regular files.

Proof: ls -l /dev/null /run/systemd/journal/stdout; cat /proc/loadavg.

Follow-up: What is the exception? (Network interfaces have no device file.)

Say first: a character device is an unbuffered byte stream (terminals, /dev/null); a block device is addressed in fixed-size blocks with kernel caching (disks).

Proof: ls -l /dev/tty1 /dev/vda shows c and b.

Follow-up: What do the two numbers in place of the size mean?

Say first: an absolute path starts at / and means the same thing everywhere; a relative path starts from the current directory.

Proof: cd /var/log works from anywhere; cd log works only from /var.

Follow-up: Why should scripts and cron jobs use absolute paths?

Say first: -r copies directories recursively; -a also preserves ownership, permissions, timestamps, symlinks and extended attributes.

Proof: sudo cp -a keeps the original owner and date; sudo cp produces root-owned files with the current date.

Follow-up: Which one should a backup script use, and what would you use for a remote copy?

Say first: on one filesystem mv renames the directory entry; across filesystems it must copy every block and then delete the original.

Proof: ls -i shows the same inode after a local rename and a new inode after moving to /dev/shm.

Follow-up: What happens if a cross-filesystem move is interrupted?

Say first: the on-disk metadata record of a file (type, mode, owner, size, timestamps, block pointers); it does not contain the name, which lives in the directory.

Proof: stat file; ls -i shows the inode number next to the name.

Follow-up: Given that, what does mv change on the same filesystem?

Say first: a hard link is another name for the same inode; a symlink is a separate file that stores a path.

Proof: ls -li shows the same inode and a link count of 2 for hard links, and a different inode with -> for symlinks.

Follow-up: What happens to each when the original name is deleted?

Say first: mtime changes with content, ctime with any inode change including permissions, atime with reads.

Proof: chmod changes only ctime in stat output.

Follow-up: Why can touch not set ctime?

Say first: an integer index into a process's table of open files, pipes, sockets and devices; 0, 1 and 2 are standard input, output and error.

Proof: ls -l /proc/$$/fd

Follow-up: Which number does the next open() return?

Say first: the process hit its per-process descriptor limit (EMFILE), usually the soft nofile limit of 1024.

Proof: grep 'open files' /proc/<pid>/limits; ls /proc/<pid>/fd | wc -l.

Follow-up: How do you raise it for a systemd service?

Say first: \; runs the command once per file; + appends as many files as fit and runs the command a few times, which is much faster.

Proof: find /etc -name '*.conf' -exec ls {} +

Follow-up: When must you use \;? (When the command accepts only one file, or {} is not last.)

Say first: find walks the filesystem live and can test any attribute; locate searches a name index built by updatedb, which is fast but can be stale.

Proof: touch /tmp/new-file; locate new-file finds nothing until sudo updatedb.

Follow-up: Why can locate hide files that exist?

Say first: archiving combines many files and their metadata into one stream (tar); compression shrinks a single stream (gzip, xz, zstd).

Proof: tar -cf a.tar dir produces a file about as large as the directory; gzip a.tar shrinks it.

Follow-up: Why does zip not need tar?

Say first: without -f, tar reads its archive from stdin, which is the terminal here, so it refuses; -f archive.tgz points it at the file instead.

Proof: tar -xz archive.tgz reports "Refusing to read archive contents from terminal"; tar -xzf archive.tgz extracts.

Follow-up: When is leaving out -f the right choice? (Piping an archive from curl or over ssh, where stdin is the source.)

Say first: -f follows the file it opened; -F follows the name, so it reopens the log after rotation.

Proof: after mv app.log app.log.1, tail -F prints "has appeared; following new file".

Follow-up: Which rotation method keeps tail -f working? (copytruncate.)

Say first: basic regex needs backslashes for + ? | ( ) { }; -E makes them special without escapes; -F treats the pattern as a fixed string.

Proof: grep -E 'a+b' does not match a+b; grep 'a+b' does.

Follow-up: When is -P needed?

Say first: exit status 1 for no match and 2 for an error, so scripts can distinguish them.

Proof: grep nomatch file; echo $? prints 1.

Follow-up: What does that do to a set -e script?

Say first: it replaces the first a on each line with b and prints every line; g replaces every match on the line, and -i writes the result back to the file.

Proof: echo a-a | sed 's/a/b/' prints b-a; with g, b-b.

Follow-up: How do you keep a backup when editing in place?

Say first: the default cycle prints the pattern space after each line; -n turns that off so only explicit p commands print.

Proof: sed '2p' file prints line 2 twice; sed -n '2p' file prints it once.

Follow-up: How do you print lines 10 to 20?

Say first: use $NF, because NF holds the field count for the current line, so $NF is always the last field even when lines have different widths; $0 is the whole line and NR the line number.

Proof: awk '{print $NF}' file prints the last column regardless of how many columns each line has.

Follow-up: What is FNR, and when does it differ from NR? (Per-file line number, useful across multiple input files.)

Say first: when the decision or output depends on field values: numeric comparisons, sums, counts per key, or reformatting.

Proof: awk '$9 >= 500' compares numbers; grep can only match text.

Follow-up: Why does cut -d' ' fail on ps output where awk works? (Repeated spaces.)

Say first: uniq compares each line only with the previous one, so duplicates must be adjacent.

Proof: printf 'b\na\nb\n' | uniq prints all three lines; adding sort first leaves two.

Follow-up: What does sort -u do differently? (Sorts and deduplicates in one process.)

Say first: the default is text order, compared character by character; -n compares numbers, -h sizes, -V versions.

Proof: printf '10\n9\n' | sort -n

Follow-up: Which option sorts du -h output correctly?

Say first: a pipe feeds stdin, but many commands (rm, chmod, kill) take arguments instead; xargs converts input lines into arguments.

Proof: find . -name '*.tmp' | xargs rm versus find . -name '*.tmp' | rm, which does nothing.

Follow-up: What breaks with file names that contain spaces?

Say first: tee writes its input to files and to stdout; with sudo, the file is opened by a root process instead of by the calling shell.

Proof: echo x | sudo tee /etc/f works where sudo echo x > /etc/f fails.

Follow-up: How do you append instead of overwrite?

Say first: JSON can change whitespace, key order and line breaks without changing meaning; jq parses the structure, so the query keeps working.

Proof: kubectl get pod web -o json | jq -r .status.phase

Follow-up: What does -r change?

Say first: /etc/passwd must be world-readable so any process can map UIDs to names, so the hashes moved to /etc/shadow, which only root can read.

Proof: ls -l /etc/passwd /etc/shadow shows modes 644 and 000.

Follow-up: How does a normal user change their own password if they cannot read /etc/shadow? (See sudo and su: SUID passwd.)

Say first: only the UID range convention in /etc/login.defs; the kernel treats every UID except 0 the same way.

Proof: useradd -r picks a UID below 1000 and skips password aging; grep SYS_UID /etc/login.defs shows the range.

Follow-up: What makes an account unable to log in, if not its UID?

Say first: the primary group is the single GID in /etc/passwd and becomes the group of new files; supplementary groups come from the member lists in /etc/group and only add access.

Proof: id amor shows gid= (primary) and groups= (all of them).

Follow-up: How do you create a file owned by a supplementary group without chgrp? (newgrp, or an SGID directory.)

Say first: user private groups (USERGROUPS_ENAB yes) let the default umask allow group write without exposing files to other users.

Proof: useradd amor creates group amor with the same ID; grep USERGROUPS_ENAB /etc/login.defs.

Follow-up: What changes about group collaboration because of this default?

Say first: password expiry (-M) forces a new password but keeps the account usable; account expiry (-E) disables the account on that date regardless of the password.

Proof: chage -l lists them as separate lines: "Password expires" and "Account expires".

Follow-up: Which one stops a contractor who logs in with an SSH key?

Say first: the password is locked; the original hash follows the !, so passwd -u restores it.

Proof: passwd -l amor then grep amor /etc/shadow shows !$y$...; passwd -S reports L.

Follow-up: Does the lock stop every way of logging in?

Say first: su switches user by knowing the target's password; sudo runs a command as another user after checking the caller's own password against a policy, and logs it.

Proof: sudo -l lists the policy; journalctl -t sudo shows each command with the invoking user.

Follow-up: Why do teams disable root's password and rely on sudo?

Say first: su - starts a login shell with the target's environment and home directory; plain su keeps the caller's environment and directory.

Proof: su amor -c 'echo $PWD' prints the current directory; su - amor -c 'echo $PWD' prints /home/amor.

Follow-up: Which sudo options map to the same two behaviours? (-s and -i.)

Say first: the account type, because it runs after auth has proved identity and decides whether access is allowed now (an expired account, a time restriction, a locked shell); auth, password and session handle proving identity, changing credentials and setting up the session.

Proof: the first column of any file in /etc/pam.d/ names the type; an expired-account denial comes from an account module such as pam_unix or pam_faillock.

Follow-up: Where would a "too many failed attempts" lockout be enforced instead? (auth with pam_faillock.)

Say first: read /var/log/btmp with sudo lastb, because it records failed attempts; who reads /run/utmp, which shows only sessions that succeeded, and last reads /var/log/wtmp for history and reboots.

Proof: who, last and sudo lastb read utmp, wtmp and btmp respectively.

Follow-up: Why is btmp not world-readable? (Users sometimes type a password into the username prompt, which lands in the log.)

Say first: the order of sources glibc queries for users, groups, hosts and other databases.

Proof: grep ^passwd: /etc/nsswitch.conf shows files first on both families.

Follow-up: Why does getent return a user that grep /etc/passwd does not?

Say first: r lists names, w creates and deletes entries (together with x), and x lets a process enter the directory and reach entries by name.

Proof: with --x, ls dir fails but cat dir/f works.

Follow-up: Which permission is needed to delete a file?

Say first: owner rwx, group r-x, others nothing: 7 = 4+2+1, 5 = 4+1, 0.

Proof: chmod 750 dir; stat -c '%a %A' dir

Follow-up: Why is 750 a common mode for application directories?

Say first: only the owner bits; the kernel stops at the first matching class.

Proof: mode 070 with owner amor in group devs: amor gets Permission denied.

Follow-up: How can the owner regain access? (chmod u+r.)

Say first: a mask of bits removed from newly created files; with 022, files requested as 666 become 644 and directories requested as 777 become 755.

Proof: umask 022; touch f; mkdir d; ls -ld f d

Follow-up: Why do new files never get the execute bit from touch?

Say first: no, its bits are cleared with a bitwise AND of the complement; subtraction gives wrong answers when the mask removes bits that were not set.

Proof: umask 0123; touch f gives 644, not 543.

Follow-up: What mode does a directory get with the same mask?

Say first: SUID runs a program as its owner, SGID runs it as its group or makes a directory pass on its group, and the sticky bit limits deletion in shared directories to each file's owner.

Proof: ls -l /usr/bin/passwd /usr/bin/write; ls -ld /tmp

Follow-up: Why does passwd need SUID?

Say first: lowercase means the special bit and the execute bit are both set; uppercase means the special bit is set without execute, so it has no effect.

Proof: chmod 4644 f shows -rwSr--r--.

Follow-up: What does T mean on a directory?

Say first: the real UID is the user who started the process; the effective UID is the one the kernel uses for permission checks, and SUID changes only the effective one.

Proof: the SUID showid program prints real uid=1002 effective uid=0.

Follow-up: Where can you see both for a running process? (Uid: line in /proc/<pid>/status.)

Say first: when more than one specific user or group needs different access to the same file, which one owner and one group cannot express.

Proof: setfacl -m u:amor:r,g:ops:rw secret.conf

Follow-up: How do you see that a file has an ACL? (The + in ls -l.)

Say first: the maximum permission for named users, named groups and the owning group; entries above it are cut down to it.

Proof: after chmod 600, getfacl shows #effective:--- on every named entry.

Follow-up: Which command changes the mask indirectly?

Say first: an extended attribute such as immutable (i) or append-only (a); also a read-only mount, and on RHEL an SELinux denial.

Proof: lsattr file shows ----i---; the error is Operation not permitted, not Permission denied.

Follow-up: How do you tell a read-only filesystem apart? (Read-only file system error, findmnt -no OPTIONS.)

Say first: i forbids every change; a allows only appending, which suits logs.

Proof: echo x >> f works with +a and fails with +i.

Follow-up: Why does a break logrotate?

Say first: rpm and dpkg install and query package files; dnf and apt add repositories, dependency resolution and transaction handling on top.

Proof: rpm -ivh on a package with missing dependencies fails; dnf install ./pkg.rpm fetches them.

Follow-up: Which database does each pair use?

Say first: the signature proves the files came from the publisher and were not altered in a mirror or in transit.

Proof: rpm -K pkg.rpm reports digests signatures OK; dnf refuses an unsigned package with GPG check FAILED.

Follow-up: Where does the package manager get the publisher's public key?

Say first: rpm installs and queries individual package files without resolving dependencies; dnf resolves dependencies from repositories and records transactions.

Proof: rpm -ivh pkg.rpm fails on missing dependencies; dnf install ./pkg.rpm fetches them.

Follow-up: When is rpm still the right tool? (Queries and verification.)

Say first: remove deletes the program but keeps its configuration files; purge deletes both.

Proof: after remove, dpkg -l shows rc or leaves the -common package with its config.

Follow-up: How do you list every package in the rc state? (dpkg -l | grep '^rc'.)

Say first: the package index is empty or stale until apt update runs.

Proof: sudo apt update && sudo apt install nginx succeeds.

Follow-up: Why do Dockerfiles combine apt-get update and install in one RUN?

Say first: the metadata or packages are signed, and the manager checks them against keys the administrator imported.

Proof: rpm -q gpg-pubkey; Signed-By: in ubuntu.sources.

Follow-up: Why is signed-by safer than a global trusted keyring?

Say first: they bundle an application with its runtime, run it sandboxed, and update independently of the distribution; RPM and DEB share system libraries.

Proof: flatpak info <app> names its own runtime; snap list shows each snap's revision.

Follow-up: What does that cost in disk space and in patching responsibility?

Say first: static linking copies library code into the binary; dynamic linking records library names that the loader finds and maps at start-up, so libraries are shared and updated separately.

Proof: ldd /usr/bin/ssh lists libraries; a static Go binary reports not a dynamic executable.

Follow-up: What is the security consequence of each for patching?

Say first: /usr/local/bin (or /opt/<app> for a bundle), because the package manager never writes there.

Proof: echo $PATH lists /usr/local/bin before /usr/bin; rpm -qf /usr/local/bin/kubectl finds no owner.

Follow-up: How do you keep track of what was installed that way?

Say first: a program is an executable file; a process is a running instance with its own address space and PID; a thread is an execution path inside a process that shares its memory and open files.

Proof: two sleep 300 commands give two PIDs for one file; ps -T -p <pid> lists a process's threads.

Follow-up: Why does one crashing thread take down the whole process?

Say first: PID 1 is the first user-space process the kernel starts (systemd on modern distributions); it starts services and adopts orphaned processes.

Proof: ps -p 1 -o comm,args; an orphan shows PPID 1 in ps -o pid,ppid.

Follow-up: What happens to the system if PID 1 exits?

Say first: kernel threads, children of kthreadd (PID 2), which run only in kernel space and have no user memory.

Proof: ps -o pid,ppid,vsz,comm --ppid 2 | head

Follow-up: What does high CPU in kswapd0 tell you?

Say first: a child that has exited but whose parent has not called wait(); it is removed when the parent reaps it or when the parent dies and PID 1 reaps it.

Proof: ps -o pid,ppid,stat,comm shows Z; kill -9 on the zombie changes nothing; killing the parent clears it.

Follow-up: Why can't kill -9 remove it?

Say first: an orphan is still running but lost its parent and was adopted by PID 1 or a subreaper; a zombie has finished but was not reaped.

Proof: an orphan shows PPID 1 and state S or R; a zombie shows state Z.

Follow-up: Why do containers need an init process such as tini?

Say first: the child's PID in the parent, 0 in the child, and -1 on failure.

Proof: strace -f shows clone(...) = 1883 in the parent while the child continues as PID 1883.

Follow-up: How does a program run a different binary after fork()?

Say first: both list every process; aux is BSD style with %CPU, %MEM, VSZ and RSS, and -ef is UNIX style with PPID and start time.

Proof: ps aux | head -2; ps -ef | head -2

Follow-up: Which command gives exactly the columns you choose? (ps -eo.)

Say first: the average number of tasks that are running, waiting for a CPU, or in uninterruptible sleep, over 1, 5 and 15 minutes.

Proof: uptime; compare with nproc.

Follow-up: How can load be high while the CPUs are idle?

Say first: VSZ is the size of the virtual address space, and RSS is the part currently in physical memory, including shared libraries.

Proof: ps -o pid,vsz,rss,comm -p <pid>; a Java process often shows a VSZ many times its RSS.

Follow-up: Why does summing RSS over all processes overcount memory? (Shared pages; PSS in smaps_rollup divides them.)

Say first: running or runnable, interruptible sleep, uninterruptible sleep in the kernel, stopped, and exited but not reaped.

Proof: ps -eo stat= | cut -c1 | sort | uniq -c

Follow-up: Which two cannot be removed with kill -9, and why?

Say first: an S process can be woken by a signal; a D process is inside a kernel operation that must finish first, so signals wait.

Proof: a process writing to a frozen filesystem shows D in ps and ignores kill -9 until fsfreeze -u.

Follow-up: What kind of problem usually causes many D processes?

Say first: it is stopped by SIGSTOP or Ctrl-Z; its memory, files and sockets stay allocated until SIGCONT or termination.

Proof: kill -STOP <pid>; ps -o stat -p <pid>

Follow-up: What happens to a SIGTERM sent to a stopped process?

Say first: SIGTERM asks a process to exit and can be caught for cleanup; SIGKILL ends it in the kernel with no chance to react.

Proof: a script with trap ... TERM prints its cleanup message on kill, while kill -9 gives exit 137 with no output.

Follow-up: Why should SIGKILL be the last resort?

Say first: 128 plus the signal number: SIGINT (Ctrl-C), SIGKILL and SIGTERM.

Proof: kill -l 137 prints KILL.

Follow-up: A container exits with 137 and nobody ran kill. What sent it? (The OOM killer, or a stop timeout.)

Say first: by convention a daemon reloads its configuration and reopens log files; a program with the default action terminates.

Proof: kill -HUP <nginx master PID> reloads nginx with the same master PID.

Follow-up: Why do terminal programs die when the SSH session drops?

Say first: it sends SIGTSTP and stops the foreground job; bg continues it in the background and fg in the foreground.

Proof: jobs shows Stopped, then Running after bg %1.

Follow-up: Which signal do bg and fg send? (SIGCONT.)

Say first: the hangup sends SIGHUP to the shell, which forwards it to its jobs, and the default action of SIGHUP is to terminate.

Proof: after closing the terminal, only the nohup, setsid and disown jobs remain in ps.

Follow-up: Which of those still dies if it writes to the closed terminal?

Say first: Ctrl-C sends SIGINT to the foreground process group; Ctrl-D sends end-of-file to a program reading the terminal.

Proof: sleep 100 then Ctrl-C gives $? 130; cat then Ctrl-D exits 0.

Follow-up: Why does Ctrl-D log out an empty shell?

Say first: it weights a process's share of CPU time when CPUs are contended, from -20 (most favored) to 19 (least), default 0.

Proof: two pinned CPU hogs at nice 0 and 10 get about 90% and 10%.

Follow-up: Why does nice change nothing on an idle machine?

Say first: lowering nice takes CPU from other users' work, so it needs CAP_SYS_NICE or an explicit limit.

Proof: renice -n 2 after renice -n 5 fails with Permission denied.

Follow-up: Where do you grant a group permission to use negative nice values?

Say first: a controlled request from a user-space process to the kernel, such as openat, read, clone or connect; it switches the CPU to kernel mode and back.

Proof: strace cat /etc/hostname lists every call and its return value.

Follow-up: Why can't a program open a file without one?

Say first: the system calls and signals of a process with their arguments, results and errno; it answers "what is this program trying to do and what fails".

Proof: strace -e trace=%file -e status=failed <cmd> lists the paths a program could not find.

Follow-up: What is the risk of running it on a production service?

Say first: the path does not exist; the permission check on the path failed; the operation is not allowed for this identity even though the path is fine.

Proof: strace cat /nonexistent gives ENOENT; strace cat /etc/shadow as a user gives EACCES; kill 1 as a user gives EPERM.

Follow-up: Which one does chattr +i cause for root?

Say first: a runlevel was a numbered SysV state; a target is a named systemd unit that groups other units, and several targets can be active at once.

Proof: ls -l /usr/lib/systemd/system/runlevel3.target points to multi-user.target.

Follow-up: Which target does a headless server use?

Say first: parallel start from declared dependencies, reliable process tracking with cgroups, built-in restart and logging, and on-demand activation.

Proof: systemd-cgls -u <service> lists every process of a service, including forked children.

Follow-up: What does a PID file miss that a cgroup does not?

Say first: start runs the service now; enable creates the symlinks that start it at boot. Neither implies the other; enable --now does both.

Proof: after enable, is-enabled says enabled while is-active can still say inactive.

Follow-up: What does enable create on disk?

Say first: reload tells the running process to reread its configuration and keeps its PID and connections; restart stops and starts it.

Proof: systemctl show -p MainPID nginx stays the same after reload and changes after restart.

Follow-up: What happens on reload for a unit without ExecReload=?

Say first: disable removes boot links, but the unit can still be started manually or as a dependency; mask links it to /dev/null so nothing can start it.

Proof: systemctl start on a masked unit fails with Unit ... is masked.

Follow-up: Give a case where masking is the right choice.

Say first: vendor units are in /usr/lib/systemd/system, runtime units in /run/systemd/system, and administrator units and drop-ins in /etc/systemd/system, which takes precedence.

Proof: systemd-analyze unit-paths; systemctl show -p FragmentPath -p DropInPaths <unit>

Follow-up: Why should the vendor file not be edited?

Say first: Wants= makes systemd start the other unit too; After= only orders the start. A dependency usually needs both.

Proof: with only Wants=, the journal shows both units starting within milliseconds.

Follow-up: When do you use Requires= instead of Wants=?

Say first: simple treats the started process as the service; forking expects it to fork a daemon and exit, and tracks the child.

Proof: a backgrounding script under Type=simple ends as inactive (dead) and systemd kills its child.

Follow-up: Why is Type=exec safer than simple?

Say first: a [Service] section with ExecStart= and an absolute path to a program that stays in the foreground, plus [Install] with WantedBy= if it should start at boot.

Proof: systemd-analyze verify /etc/systemd/system/<name>.service, then systemctl enable --now <name>.

Follow-up: What changes if the program forks into the background?

Say first: systemd could not execute the program in ExecStart=: the path is wrong, the file is not executable, or its interpreter is missing.

Proof: journalctl -u <unit> shows Failed at step EXEC spawning <path>.

Follow-up: Which other step names appear in such messages? (USER, CHDIR, NAMESPACE.)

Say first: how long each unit took to start; units start in parallel, so the times overlap and a slow unit may not delay boot.

Proof: systemd-analyze critical-chain shows the path that actually delayed a target.

Follow-up: How do you make a slow unit stop blocking boot?

Say first: /var/log/secure on RHEL and /var/log/auth.log on Ubuntu; both also sit in the journal.

Proof: sudo grep "Failed password" /var/log/secure; journalctl -u sshd -g "Failed password".

Follow-up: Which binary file records failed logins, and which command reads it?

Say first: The journal is journald's indexed binary store with structured fields; messages is a text copy that rsyslog writes from it using facility and priority rules.

Proof: journalctl -u crond -o verbose -n 1 shows fields such as _SYSTEMD_UNIT that the text file does not keep.

Follow-up: Which one survives a reboot on a host with no /var/log/journal?

Say first: journalctl -u <unit> -b.

Proof: journalctl -u nginx -b --no-pager

Follow-up: And for the boot before the last one?

Say first: The journal is volatile: /var/log/journal does not exist, so only the current boot is kept in /run.

Proof: journalctl --header | grep "File path" shows /run/log/journal.

Follow-up: How do you make it persistent without a reboot?

Say first: All priorities of the authpriv facility; messages excludes it with authpriv.none because those lines are sensitive and go to the root-only secure file.

Proof: grep authpriv /etc/rsyslog.conf

Follow-up: Which facilities can applications use freely? (local0 to local7.)

Say first: @ sends over UDP, which can lose messages silently; @@ sends over TCP.

Proof: *.* @@logs.example.test:514

Follow-up: What does TCP still not give you? (Encryption.)

Say first: create renames the file and makes a new one, so the program must reopen it; copytruncate copies and truncates in place, so the program keeps writing but a few lines can be lost.

Proof: ls -l /proc/<pid>/fd shows app.log.1 after create without a reload.

Follow-up: Which one does the nginx package use, and how does nginx reopen?

Say first: By logrotate.timer, daily, with Persistent=true.

Proof: systemctl list-timers logrotate.timer

Follow-up: Why does a weekly rule still wait a week if the timer runs daily? (The state file.)

Say first: Print the first field, sort, count duplicates, sort by count descending.

Proof: awk '{print $1}' access.log | sort | uniq -c | sort -rn | head

Follow-up: Why does uniq -c need sorted input?

Say first: grep block-buffers its output when writing to a pipe; --line-buffered flushes each line.

Proof: tail -f access.log | grep --line-buffered " 500 " | awk '{print $7}'

Follow-up: Which other tools need a similar flag? (stdbuf -oL, sed -u.)

Say first: 30 2 * * 1-5 /usr/local/bin/db-backup.

Proof: crontab -l after installing it; journalctl -u crond the next morning.

Follow-up: What changes in /etc/cron.d? (A user field after the schedule.)

Say first: cron runs it with /bin/sh, a minimal PATH, no profile files, no terminal and a different working directory.

Proof: * * * * * env > /tmp/cron-env.txt, then compare with env in the shell.

Follow-up: Which character in a crontab line has a special meaning? (%.)

Say first: cron runs at exact times and skips runs while the host is off; anacron runs daily, weekly and monthly jobs that were missed, once per period.

Proof: cat /etc/anacrontab; ls /var/spool/anacron.

Follow-up: Which one would you use on a laptop?

Say first: A .timer and the .service it starts; you enable and start the timer.

Proof: systemctl enable --now db-dump.timer; systemctl list-timers.

Follow-up: Why does the service have no [Install] section?

Say first: If the host was off at the scheduled time, the job runs once soon after the next boot.

Proof: ls /var/lib/systemd/timers/ holds the stamp files that record the last run.

Follow-up: Which cron component provides the same behavior? (anacron.)

Say first: A virtual filesystem generated by the kernel on each read; nothing is stored on disk, so there is no size to report.

Proof: ls -l /proc/meminfo shows 0; wc -c /proc/meminfo shows the real length.

Follow-up: Which command-line tools are wrappers around /proc?

Say first: /proc is process-centric with legacy system files; /sys models devices, drivers and modules with one value per file.

Proof: ls /proc/1; ls /sys/class/net/eth0.

Follow-up: Where do sysctl parameters live? (/proc/sys.)

Say first: sysctl -w changes it now; a file in /etc/sysctl.d/ plus sysctl --system makes it permanent.

Proof: sudo sysctl -w net.ipv4.ip_forward=1; echo "net.ipv4.ip_forward = 1" | sudo tee /etc/sysctl.d/90-forward.conf.

Follow-up: Which service applies the file at boot?

Say first: It lets the kernel route packets between interfaces; routers, NAT gateways, VPN servers and container or Kubernetes hosts need it.

Proof: sysctl net.ipv4.ip_forward

Follow-up: What breaks for containers when it is 0?

Say first: modprobe takes a name and loads dependencies from modules.dep, and reads /etc/modprobe.d; insmod loads one file by path with no dependency handling.

Proof: modprobe --show-depends <name>; insmod sctp fails without a path.

Follow-up: Which command rebuilds the dependency index?

Say first: Only from automatic loading; an explicit modprobe or a dependency still loads it. install <name> /bin/false blocks it completely.

Proof: The nbd demo above: loaded after blacklist, refused after install.

Follow-up: Why can a blacklisted storage driver still load at boot? (The initramfs.)

Say first: The major number selects the driver and the minor number selects the device that driver handles.

Proof: ls -l /dev/null shows 1, 3; grep -w 1 /proc/devices shows mem.

Follow-up: Why does fstab use UUIDs instead of /dev/sdb1?

Say first: The kernel log, for an OOM kill or a segfault.

Proof: journalctl -k -g "Killed process|segfault"; dmesg -T | grep -i oom.

Follow-up: How do you tell a cgroup OOM from a host OOM?

Say first: dmesg reads the current ring buffer; journalctl -k reads the kernel messages the journal stored, with real timestamps and earlier boots.

Proof: journalctl -k -b -1

Follow-up: Why can dmesg -T show the wrong time?

Say first: they are disks on different drivers: SCSI layer (SATA, SAS, USB), Virtio in a KVM guest, and NVMe controller 0, namespace 1.

Proof: lsblk -o NAME,TRAN shows sata, virtio or nvme.

Follow-up: How are partitions on an NVMe disk named? (nvme0n1p1, because the name ends in a digit.)

Say first: above 2 TiB, with more than four primary partitions, and when the machine boots in UEFI mode.

Proof: sudo fdisk -l /dev/sda shows Disklabel type: gpt or dos.

Follow-up: Where does GPT keep its backup table? (In the last sectors of the disk.)

Say first: device names follow detection order and can change, while the filesystem UUID stays with the data.

Proof: ls -l /dev/disk/by-uuid; lsblk -f.

Follow-up: What changes the UUID? (Recreating the filesystem, or tune2fs -U and xfs_admin -U.)

Say first: fdisk and gdisk are interactive and write on w (gdisk handles GPT only), while parted applies each command immediately and scripts well with -s.

Proof: sudo fdisk -l, sudo gdisk -l, sudo parted -s /dev/sdb print.

Follow-up: Which tool dumps a table as text for backup? (sfdisk -d.)

Say first: a 1 MiB start aligns partitions with the erase blocks of SSDs and the stripes of RAID arrays, and leaves room for boot code on MBR disks.

Proof: sudo fdisk -l shows Start 2048; sudo parted /dev/sdb align-check optimal 1.

Follow-up: What does misalignment cost? (Extra writes and lower performance on every I/O.)

Say first: both are journaling filesystems; ext4 can shrink and fixes its inode count at creation, while XFS scales better for large files and parallel I/O, allocates inodes dynamically and cannot shrink.

Proof: df -i on both; xfs_info shows allocation groups.

Follow-up: Which one does RHEL install by default, and what does that mean for reducing a logical volume?

Say first: it records metadata changes before they are applied, so after a crash the kernel replays the journal instead of scanning the whole filesystem.

Proof: tune2fs -l lists has_journal; xfs_info shows the internal log.

Follow-up: Why does ext4's default data=ordered mode not journal file contents?

Say first: device, mount point, filesystem type, options, dump flag and fsck pass number.

Proof: grep -v '^#' /etc/fstab; man 5 fstab.

Follow-up: Why is the pass number 0 for XFS? (XFS checks itself at mount time; fsck.xfs does nothing.)

Say first: device names follow detection order and can change after a reboot or disk change, while the UUID belongs to the filesystem.

Proof: sudo blkid; ls -l /dev/disk/by-uuid.

Follow-up: When does the UUID change?

Say first: the mount becomes wanted instead of required by local-fs.target, so boot continues when the device is missing.

Proof: systemctl show -p RequiredBy,WantedBy <unit>.mount.

Follow-up: Which option waits for the network before mounting? (_netdev.)

Say first: swap holds memory pages the kernel moved out of RAM; some used swap is normal, while constant swapping in and out means the host needs more memory.

Proof: free -h; vmstat 1 shows si and so.

Follow-up: What does vm.swappiness change?

Say first: performance is the same on current kernels; a file is easier to resize, while a partition needs no filesystem and suits hibernation setups.

Proof: swapon --show lists partition or file.

Follow-up: Why must a swap file have no holes?

Say first: physical volumes are disks or partitions initialized for LVM, a volume group pools them into extents, and logical volumes are allocations of those extents used like partitions.

Proof: sudo pvs; sudo vgs; sudo lvs; vgdisplay shows PE Size.

Follow-up: How does LVM map an LV to disk blocks? (Through device-mapper tables: sudo dmsetup table.)

Say first: volumes can grow across disks, move between disks online and be snapshotted, without repartitioning.

Proof: lvextend -r, pvmove, lvcreate -s.

Follow-up: What does LVM not protect against? (Disk failure, unless the LV uses RAID or mirroring.)

Say first: the volume grew, but the partition and filesystem on it keep their old size until they are grown too.

Proof: lsblk shows a 50 GiB disk with a 20 GiB partition; df -h shows the filesystem.

Follow-up: Which commands finish the job for XFS on a partition?

Say first: no; the data must be copied to a new, smaller volume.

Proof: the provider rejects a smaller size in modify-volume.

Follow-up: Which filesystem also rules out shrinking in place?

Say first: df asks the filesystem how many blocks are allocated; du walks the directory tree and adds up the files it can reach.

Proof: df -h /var; sudo du -sh /var.

Follow-up: Name two reasons they can disagree.

Say first: inode usage; a filesystem with no free inodes cannot create files even with free space.

Proof: df -i; many small files in one tree.

Follow-up: Which filesystem fixes the inode count at creation?

Say first: the hard limit is never exceeded; the soft limit may be exceeded for the grace period, after which it is enforced like a hard limit.

Proof: quota -s <user> shows usage, both limits and the grace time.

Follow-up: What error does a program get at the limit? (EDQUOT, Disk quota exceeded.)

Say first: a full copies everything, an incremental copies changes since the last backup of any kind, and a differential copies changes since the last full.

Proof: tar --listed-incremental; restore order full, then each incremental.

Follow-up: Which one restores fastest, and which uses least space?

Say first: both live on the same system; RAID copies deletions and corruption instantly, and an LVM snapshot is lost with its volume group.

Proof: lvs shows the snapshot in the same VG as its origin.

Follow-up: What is the 3-2-1 rule?

Say first: 0 stripes for speed with no redundancy, 1 mirrors, 5 stripes with one parity block and survives one failure, 10 stripes across mirrors for speed and redundancy.

Proof: cat /proc/mdstat; sudo mdadm --detail /dev/md0.

Follow-up: Why is RAID not a backup?

Say first: UP means an administrator enabled the interface; LOWER_UP means the driver sees a carrier, so the physical or virtual link is connected.

Proof: ip link show eth0; a pulled cable shows NO-CARRIER with UP still set.

Follow-up: Why does a dummy interface report state UNKNOWN?

Say first: ip uses netlink and shows everything the kernel supports (multiple addresses, policy routing, tunnels, namespaces); net-tools used old ioctls and is no longer developed.

Proof: ip addr add a second address; ifconfig shows only the primary one.

Follow-up: Which ip commands replace route -n and arp -n?

Say first: a device is the interface; a connection is a saved profile of settings that NetworkManager activates on a device.

Proof: nmcli device status; nmcli connection show.

Follow-up: Why does nmcli con mod not change the running address?

Say first: Ubuntu uses netplan YAML that generates systemd-networkd files; RHEL uses NetworkManager keyfiles managed with nmcli.

Proof: ls /etc/netplan; ls /etc/NetworkManager/system-connections.

Follow-up: Which tool writes /etc/resolv.conf on each?

Say first: it checks the policy rules, then picks the most specific (longest prefix) matching route, and uses the metric to break ties.

Proof: ip rule show; ip route get 10.20.5.9 with a /16 and a /24 present.

Follow-up: What happens when no route matches?

Say first: it lets the kernel pass packets between interfaces; without it the host drops traffic not addressed to itself.

Proof: sysctl net.ipv4.ip_forward; tcpdump on the router shows requests arriving and nothing leaving.

Follow-up: Which container and Kubernetes components need it turned on?

Say first: glibc reads the hosts: line in nsswitch.conf, checks /etc/hosts, then asks the servers in /etc/resolv.conf, which on Ubuntu is the local systemd-resolved stub.

Proof: grep ^hosts /etc/nsswitch.conf; cat /etc/resolv.conf; resolvectl status.

Follow-up: Which tools skip /etc/hosts?

Say first: NXDOMAIN is a definite answer that the name does not exist; SERVFAIL means the server could not produce an answer.

Proof: dig @172.16.1.3 nosuch.shop.internal gives NXDOMAIN; with named stopped, the resolved stub answers SERVFAIL for the same zone.

Follow-up: Which one is cached, and for how long? (Negative answers, for the SOA minimum.)

Say first: 0.0.0.0 accepts connections on every address of the host; 127.0.0.1 accepts only connections from the host itself.

Proof: sudo ss -tlpn; nc -vz <server ip> 9000 from another host is refused.

Follow-up: Why does a container publishing a port to 127.0.0.1 behave the same way?

Say first: refused means the host replied with a TCP reset because nothing listens; a timeout means packets or replies are dropped on the way.

Proof: nc -vz -w2 host 9000 returns at once; a firewalled port waits for -w.

Follow-up: Which firewall action produces a refusal instead of a timeout? (reject.)

Say first: CLOSE_WAIT is on the side that received a FIN and has not closed yet (an application issue); TIME_WAIT is on the side that closed first and lasts 60 seconds (normal).

Proof: ss -tan state close-wait; ss -tan state time-wait.

Follow-up: Which one should never pile up?

Say first: it sends probes with increasing TTL; each router that decrements TTL to zero returns ICMP Time Exceeded, which reveals its address.

Proof: traceroute -n <host>; tcpdump -n icmp on the client shows the time exceeded replies.

Follow-up: Why do some hops show * * * while later hops answer?

Say first: ICMP is filtered while TCP 80/443 is allowed; ping only tests ICMP.

Proof: ping -c2 <host> fails; curl -sI https://<host> succeeds.

Follow-up: Which test would you use instead of ping in a runbook?

Say first: SYN, SYN-ACK, FIN with ACK, and reset with ACK.

Proof: sudo tcpdump -ni eth0 'tcp port 8080' during a curl.

Follow-up: Which packet does the client see when a port is closed?

Say first: a bridge switches frames between several segments; a bond presents several NICs as one link to one segment.

Proof: bridge link show; cat /proc/net/bonding/bond0.

Follow-up: Which one does Docker use for its default network?

Say first: clocks drift, and wrong time breaks TLS validity checks, Kerberos, TOTP, scheduled jobs, log correlation and distributed consensus.

Proof: openssl verify -attime with a shifted time fails; chronyc tracking shows the offset.

Follow-up: Which component in Kubernetes is most sensitive to clock skew?

Say first: stepping jumps the time at once; slewing speeds up or slows down the clock until it is correct, so time never goes backwards.

Proof: makestep 1.0 3 in chrony.conf; journalctl -u chronyd shows System clock was stepped.

Follow-up: Why can a backwards step hurt a database or a log pipeline?

Say first: 502 means the proxy got no valid response (refused, reset, crashed backend); 504 means the backend did not answer in time.

Proof: the nginx error log shows connect() failed for a 502 and upstream timed out for a 504.

Follow-up: Which timeout setting controls the 504?

Say first: the proxy opens its own connection to the backend; the client address travels in X-Forwarded-For.

Proof: the backend response shows client=172.16.1.2 xff=172.16.0.2.

Follow-up: Why must the application not trust that header from everyone?

Say first: it is both the routing decision for outgoing packets (which peer gets them) and the filter for incoming packets (which source addresses a peer may use).

Proof: sudo wg show wg0 allowed-ips; a ping outside every range fails with Required key not available.

Follow-up: What happens if two peers list overlapping ranges?

Say first: clarify the scope, then test from the bottom up: link, address, route, gateway, reachability, DNS, port, firewall, application, stopping at the first failure.

Proof: ip -br link; ip route get; ping; getent hosts; nc -vz; curl -sv.

Follow-up: Which one question would you ask first, and why?

Say first: the server holds the public key in authorized_keys, and the client proves it owns the matching private key by signing session data; the private key never crosses the network.

Proof: ssh -v host shows Offering public key and Server accepts key.

Follow-up: What does a passphrase add, and how does ssh-agent keep that practical?

Say first: it records each server's host key, so the client detects a different server answering on the same address.

Proof: ssh-keygen -F 172.16.1.3

Follow-up: When is StrictHostKeyChecking accept-new acceptable, and why not no?

Say first: -L listens on the client and connects from the server, -R listens on the server and connects from the client, and -D is a SOCKS proxy on the client whose destinations the application chooses.

Proof: ss -tlnp on the side that listens shows the ssh or sshd process.

Follow-up: In -L 9000:127.0.0.1:8008, whose loopback is 127.0.0.1?

Say first: check the syntax with sshd -t, confirm the effective value with sshd -T, reload instead of restart, and test a new login while the old session stays open.

Proof: sudo sshd -t && sudo systemctl reload sshd

Follow-up: Why can a drop-in file silently override your change?

Say first: for repeated copies, large files and directory trees, because rsync sends only differences, can resume and can delete extra files.

Proof: a second rsync -a --stats of an unchanged file reports Number of regular files transferred: 0.

Follow-up: What does the trailing slash on the source change?

Say first: refused means the host answered but nothing listens (or a firewall rejects); timed out means nothing answered, usually a dropping firewall or a routing problem.

Proof: nc -vz -w3 host 22; ss -tlnp on the server for refused, security groups and nft list ruleset for timeouts.

Follow-up: Which one does a cloud security group produce?

Say first: StrictModes refuses keys that another user could have written, because that user could add their own key.

Proof: the log line Authentication refused: bad ownership or modes for directory.

Follow-up: Which files and directories does it check?

Say first: runtime rules apply at once and are lost on reload or reboot; permanent rules are saved and apply after a reload.

Proof: sudo firewall-cmd --add-port=8080/tcp disappears after sudo firewall-cmd --reload.

Follow-up: How do you keep a tested runtime rule?

Say first: a named rule set applied to traffic by source address or interface, with the default zone for everything else.

Proof: sudo firewall-cmd --get-active-zones

Follow-up: A source is in internal and the interface is in public. Which zone applies?

Say first: input sees packets addressed to the host itself, forward sees packets the host routes to another host.

Proof: a drop policy in inet lab forward on gw blocked client to web traffic while gw's own services stayed reachable.

Follow-up: Which chain filters traffic to a Docker container published with -p?

Say first: replies and follow-up packets match it, so allow rules only need to describe new connections.

Proof: the port rule matched one packet per connection, the established rule matched the rest.

Follow-up: What does related add for ICMP errors and FTP?

Say first: chcon changes a label on disk only; semanage fcontext changes the policy's labeling rules, which restorecon and relabels apply.

Proof: after chcon, restorecon -Rv /data/www put default_t back and the page returned 403.

Follow-up: Why does mv from /tmp cause SELinux denials?

Say first: AppArmor confines programs by file paths in per-program profiles; SELinux labels every file and process and decides by type.

Proof: /etc/apparmor.d/usr.bin.tcpdump lists paths; ls -Z on RHEL shows labels.

Follow-up: Which one does Docker use on Ubuntu, and on RHEL?

Say first: they split root's privileges into separate bits, so a process gets only the ones it needs instead of full root.

Proof: sudo capsh --drop=cap_chown -- -c 'chown ...' failed even as root.

Follow-up: Which capabilities are nearly equal to root?

Say first: on Ubuntu it carries the file capability cap_net_raw; on RHEL it uses unprivileged ICMP sockets allowed by net.ipv4.ping_group_range.

Proof: getcap /usr/bin/ping; sysctl net.ipv4.ping_group_range.

Follow-up: Why is a file capability safer than setuid root?

Say first: it is the UID a user logged in with; it survives sudo and su, so audit records show the real person behind root actions.

Proof: ausearch -k root-exec --format text printed deploy, acting as root.

Follow-up: Why do services show unset?

Say first: that the content is unchanged since the holder of the private key signed it; whether the key belongs to the publisher is checked separately, by fingerprint.

Proof: after one appended line, gpg --verify printed BAD signature and exited 1.

Follow-up: What else must you check before trusting the key?

Say first: that the chain leads to a trusted root, that the requested name is in the SAN, and that every certificate is within its validity dates.

Proof: openssl s_client -connect host:443 -servername host shows the chain and Verify return code.

Follow-up: Which of these checks does a wrong system clock break?

Say first: not necessarily; RHEL backports fixes, so check the package changelog or errata, not the upstream version.

Proof: rpm -q --changelog openssh-server | grep CVE-...; dnf updateinfo list --security.

Follow-up: Where does Ubuntu publish the same information?

Say first: updates, SSH (keys only, no root), a default-deny firewall, only needed services and accounts, and MAC left enforcing.

Proof: sudo dnf updateinfo list --security; sudo sshd -T; sudo firewall-cmd --list-all; sudo ss -tulpn; getenforce.

Follow-up: Which of these would you automate first, and how?

Say first: firmware, bootloader, kernel with initramfs, switch to the real root, then systemd reaching the default target.

Proof: /proc/cmdline shows the bootloader's handoff; systemd-analyze shows kernel and userspace phases.

Follow-up: which stage does Cannot open root device fail at? (initramfs or root=.)

Say first: it is a temporary in-memory root holding the drivers needed to reach the real root, such as LVM, RAID, encryption or network storage.

Proof: lsinitrd lists its contents; /proc/cmdline root= names the real root it pivots to.

Follow-up: when must you rebuild it? (after a storage or driver change; see kernel updates.)

Say first: grub.cfg is generated from /etc/default/grub and /etc/grub.d, and a kernel update regenerates it, discarding hand edits.

Proof: the file header marks it auto-generated; changes belong in the source, applied with grub2-mkconfig or update-grub.

Follow-up: where do persistent kernel parameters go? (GRUB_CMDLINE_LINUX.)

Say first: rescue starts a minimal system with local filesystems mounted and a root shell; emergency is barer, with a read-only root and almost nothing started.

Proof: systemctl rescue versus systemctl emergency; boot with systemd.unit=rescue.target.

Follow-up: why can neither reset a lost root password? (both prompt for it; use rd.break.)

Say first: an oops is a recoverable kernel error that logs and may continue; a panic is an unrecoverable stop where the kernel halts.

Proof: an oops leaves a backtrace in dmesg; a panic prints to the console and stops, often with nothing on disk.

Follow-up: when does an oops become a panic? (in atomic or interrupt context, or with panic_on_oops=1.)

Say first: not until it reboots into the new kernel; installing changes /boot, not the running kernel.

Proof: uname -r still shows the old version; needs-restarting -r reports a reboot is required.

Follow-up: what technology applies some fixes without a reboot? (livepatch / kpatch.)

Say first: uname -r shows the running kernel; the package tools list the installed ones, which can be newer.

Proof: uname -r against rpm -q kernel or dpkg -l 'linux-image*'.

Follow-up: why do they differ right after an update? (the new kernel runs only after reboot.)

Say first: for every resource it checks Utilization, Saturation and Errors; utilization alone hides a queue building behind a busy resource.

Proof: a disk at %util 100 with aqu-sz near 1 is fine, while the same disk with aqu-sz 140 is the bottleneck.

Follow-up: which single command shows both utilization and saturation for disks? (iostat -xz.)

Say first: Linux load counts uninterruptible (D state) tasks as well as runnable ones, so processes blocked on I/O raise it without using CPU.

Proof: uptime high, mpstat %idle near 100, vmstat b and wa non-zero.

Follow-up: what commonly puts many tasks into D state at once? (a stalled disk or NFS mount.)

Say first: on average eight tasks wanted to run while only four cores existed, so the host was oversubscribed by roughly two times.

Proof: compare uptime against nproc; vmstat r shows the current run queue.

Follow-up: would the same load of 8 be a problem on a 16-CPU host? (no, there is headroom.)

Say first: us is user code, sy is kernel code, wa is idle time waiting on I/O, and st is time the hypervisor gave to another guest.

Proof: mpstat -P ALL 1 shows all four per CPU.

Follow-up: which of these means the CPU is not actually the bottleneck? (wa and st.)

Say first: Linux uses otherwise-idle RAM for page cache, which is reclaimable, so available matters more than free.

Proof: free -h shows large buff/cache and an available value well above free.

Follow-up: which metric should a memory alert watch? (available, or active swap-in.)

Say first: VSZ is reserved address space, RSS is physical RAM the process holds, and PSS is RSS with shared pages divided among the processes sharing them.

Proof: /proc/PID/status for VmSize and VmRSS; smaps_rollup for Pss.

Follow-up: why can the sum of RSS across processes exceed physical RAM? (shared pages counted multiple times.)

Say first: a minor fault is satisfied from memory (zero page, cache or copy-on-write); a major fault needs a disk read and is far slower.

Proof: /usr/bin/time -v reports both counts; /proc/vmstat pgmajfault tracks major faults system-wide.

Follow-up: which counter rising indicates the system is thrashing? (pgmajfault with vmstat si.)

Say first: %util is the fraction of time the device had at least one request in flight; on SSDs and arrays that serve requests in parallel, 100% does not mean saturated.

Proof: iostat -xz 1 with %util at 100 but low aqu-sz and await is still healthy.

Follow-up: which two fields better indicate saturation? (aqu-sz and await.)

Say first: the soft limit is enforced now; the hard limit is the ceiling a process may raise its own soft limit to without root.

Proof: ulimit -Sn and ulimit -Hn show both; a non-root process can set soft up to hard, not beyond.

Follow-up: who can raise the hard limit? (root, or a unit's LimitNOFILE.)

Say first: strace when the question is which syscalls a process makes and why they fail; perf when the question is where CPU time goes.

Proof: strace -e openat finds a missing file; perf top finds a hot function.

Follow-up: why is strace a poor choice on a busy production process? (it traps every syscall and slows it heavily.)

Say first: an SLI is the measured indicator, an SLO is the internal target for it, and an SLA is the external contract with consequences.

Proof: latency is the SLI, 99.9% under 300ms is the SLO, and service credits below 99.5% is the SLA.

Follow-up: what is an error budget? (the allowed failure under the SLO.)

Say first: liveness confirms the process is running; readiness confirms it can actually serve requests.

Proof: systemctl is-active for liveness; a curl to the endpoint returning 200 for readiness.

Follow-up: why can a process be alive but not ready? (still warming up, or a dependency is down.)

Say first: tuned applies a tested profile of many settings chosen together for a workload, avoiding conflicts between hand-copied values.

Proof: tuned-adm list shows profiles like throughput-performance; tuned-adm active shows the current one.

Follow-up: how do you make a profile survive a reboot? (it already does; tuned-adm profile persists it.)

Say first: tuning helps only after the bottleneck is measured; changing a knob before that adds variables without evidence.

Proof: confirm the limiting resource with vmstat/iostat, then apply one change and re-measure.

Follow-up: why change one setting at a time? (so the effect is attributable.)

Say first: a hard mount retries forever if the server stops, blocking the process but never losing data; a soft mount fails I/O after a few retries, risking corruption.

Proof: the mount options hard and soft; a hard-mounted hang shows processes in D state.

Follow-up: which is the safe default for data? (hard, with _netdev and nofail in fstab.)

Say first: it maps a client's root to nobody on the export, so a remote root cannot own or overwrite files as root on the server.

Proof: it is on by default; exportfs -s shows root_squash in the options.

Follow-up: when would you disable it? (rarely, for a trusted management host, with the risk understood.)

Say first: it mounts on demand and unmounts when idle, so shares are only connected when used and an absent server does not hang boot.

Proof: the path is unmounted until accessed, then findmnt shows the mount.

Follow-up: what is the default idle timeout? (300 seconds.)

Say first: a direct map uses absolute paths under the key /-; an indirect map has a base directory whose keys become subdirectories.

Proof: the master map references /- for direct, a base path for indirect.

Follow-up: which suits per-user home directories? (an indirect map with a wildcard key.)

Say first: NFS is the native Unix network filesystem; Samba serves the SMB protocol used by Windows, mounted on Linux as cifs.

Proof: NFS mounts are -t nfs; SMB mounts are -t cifs.

Follow-up: which would you pick for a mixed Windows and Linux environment? (Samba, for SMB compatibility.)

Say first: Samba keeps a separate password database from /etc/shadow, so a system user must also be given a Samba password.

Proof: smbpasswd -a user sets it; /etc/passwd alone is not enough to log in over SMB.

Follow-up: where is the SMB password stored? (Samba's own database, not /etc/shadow.)

Say first: iSCSI exports a raw block device that one host owns and formats; NFS exports a filesystem that many clients share.

Proof: an iSCSI login adds a /dev/sd* disk; an NFS mount is a filesystem of type nfs.

Follow-up: why can two hosts share one NFS mount but not one iSCSI LUN? (NFS coordinates access; a raw block device does not.)

Say first: the target is the server exporting the LUN; the initiator is the client that logs in and uses it.

Proof: targetcli builds the target; iscsiadm drives the initiator; each has an IQN.

Follow-up: what restricts which client may connect? (the target's ACL, by initiator IQN.)

Say first: a namespace gives a set of processes their own isolated instance of a global kernel resource, such as the process tree, network stack, or mount table.

Proof: ls /proc/self/ns/ lists the eight types; lsns shows which processes share each one.

Follow-up: which namespace makes rootless containers possible? (the user namespace, by mapping container root to an unprivileged host uid.)

Say first: a namespace controls what a process can see; a cgroup controls how much it can use.

Proof: namespaces isolate the PID, net and mount views; cgroups cap CPU, memory and pids, covered in Cgroups.

Follow-up: which one does docker run --memory use? (a cgroup.)

Say first: cgroups limit and account for a group of processes' resource use (CPU, memory, pids, I/O); namespaces isolate what those processes can see.

Proof: cpu.max and memory.max cap usage; /proc/self/ns/ shows the isolation, covered in Namespaces.

Follow-up: which one enforces docker run --memory? (a cgroup, via memory.max.)

Say first: v2 is a single unified hierarchy for all controllers; v1 had a separate tree per controller.

Proof: stat -fc %T /sys/fs/cgroup returns cgroup2fs; controllers are enabled per subtree with cgroup.subtree_control.

Follow-up: why did runtimes move to v2? (one consistent hierarchy, better memory and OOM handling.)

Say first: chroot only changes the root directory a process sees; a container adds a mount namespace, pivot_root, an overlay filesystem, cgroups and dropped capabilities.

Proof: chroot shares the host's namespaces; a container has its own, shown by /proc/PID/ns/.

Follow-up: why is chroot not a security boundary? (a privileged process can escape it.)

Say first: one or more read-only lowerdirs, a single writable upperdir, a workdir for scratch, and the merged view processes use.

Proof: mount -t overlay -o lowerdir=...,upperdir=...,workdir=...; reads prefer the upper, writes go to the upper.

Follow-up: which layer holds a container's changes? (the upperdir.)

Say first: a container is a host process isolated with namespaces and cgroups, sharing the host kernel; a VM runs its own guest kernel on virtual hardware.

Proof: ps on the host finds a container's process; uname -r inside a container reports the host kernel.

Follow-up: which gives a stronger isolation boundary? (a VM, because it has its own kernel.)

Say first: a container only forks a process into new namespaces, while a VM boots a full guest kernel and OS.

Proof: podman run returns in milliseconds; a VM boot runs an init sequence.

Follow-up: what does a container give up for that speed? (a shared kernel, so a weaker boundary.)

Say first: each podman command runs as its own process with no central daemon, and containers run as an ordinary user by default rather than as root.

Proof: podman info shows rootless: true; there is no dockerd-style service.

Follow-up: how does container root stay unprivileged? (a user namespace maps it to the user's uid and subuid range.)

Say first: container uid 0 maps to the running user's uid, and higher container ids map into the user's subuid range.

Proof: /proc/PID/uid_map shows 0 <user-uid> 1; /etc/subuid holds the range.

Follow-up: who owns a file the container creates as root? (a host subuid, not real root.)

Say first: KVM is the kernel's hardware virtualization, QEMU emulates the virtual hardware, and libvirt is the management layer tools like virsh use.

Proof: lsmod | grep kvm; qemu-system-x86_64 --version; virsh list.

Follow-up: what does a VM have that a container does not? (its own guest kernel.)

Say first: check for the CPU virtualization flag and /dev/kvm, or run virt-host-validate.

Proof: grep -E 'vmx|svm' /proc/cpuinfo; ls /dev/kvm; virt-host-validate qemu.

Follow-up: what happens without /dev/kvm? (QEMU emulates in software, much slower.)

Say first: qcow2 is thin and supports snapshots and backing files; raw is a flat full-size file that is faster with no thin provisioning.

Proof: qemu-img info shows qcow2's disk size far below its virtual size.

Follow-up: when would you convert qcow2 to raw? (when a platform requires raw, with qemu-img convert.)

Say first: the machine-id and the SSH host keys, at least.

Proof: /etc/machine-id and /etc/ssh/ssh_host_*_key; both must be regenerated on a clone.

Follow-up: what breaks if SSH host keys are shared? (clients cannot tell the hosts apart; host-key checks fail.)

Say first: it configures a cloud image on first boot from metadata the platform supplies, running once per instance.

Proof: cloud-init status; state under /var/lib/cloud/instance.

Follow-up: where does it read the config from? (user-data via a data source such as the cloud API or a NoCloud ISO.)

Say first: Kickstart automates a full OS install with Anaconda; cloud-init configures an already-installed cloud image on first boot.

Proof: Kickstart uses ks.cfg and inst.ks=; cloud-init uses #cloud-config user-data.

Follow-up: which do public cloud images use? (cloud-init.)