A production server reports `df -h` showing `/var` at 96% full, yet `du -sh /var` sums to only 40% of the partition. `du` was run as root. The app was recently restarted after a log-rotation script `truncate`d nothing but instead `rm`'d a 20GB active log file. What is the single most reliable command to confirm the root cause and reclaim the space without a reboot?
- A. Run `sync && echo 3 > /proc/sys/vm/drop_caches` to flush the page cache holding the deleted file's dirty pages
- B. Run `lsof +L1` (or `lsof | grep deleted`) to find the process holding the unlinked inode, then restart or signal that process to close the fd ✓
- C. Run `fsck` on the `/var` filesystem to reclaim orphaned inodes left by the failed `rm`
- D. Run `fstrim /var` to release blocks that the SSD firmware has not yet marked as free after the deletion
Correct answer: B. An unlinked file with an open file handle keeps its data blocks allocated until every fd is closed, so `lsof +L1` reveals the holding process and closing it (or the app was already restarted, releasing it) reclaims the space—no reboot or fsck needed.
A high-throughput reverse proxy under load starts refusing new outbound connections to a backend with `EADDRNOTAVAIL`, while `ss -s` shows tens of thousands of sockets in TIME_WAIT toward that single backend IP:port. Which change most directly addresses the exhaustion without risking data corruption from delayed duplicate segments?
- A. Set `net.ipv4.tcp_tw_recycle=1` to aggressively recycle TIME_WAIT sockets globally
- B. Lower `net.ipv4.tcp_fin_timeout` to 5 seconds so sockets leave TIME_WAIT much faster
- C. Enable `net.ipv4.tcp_tw_reuse=1` and widen `net.ipv4.ip_local_port_range` so the client can safely reuse TIME_WAIT sockets for new outbound connections ✓
- D. Increase `net.core.somaxconn` and the backend's listen backlog to accept the pending connections
Correct answer: C. `tcp_tw_reuse` safely reuses TIME_WAIT sockets for new outbound connections using timestamps (unlike the removed/unsafe `tcp_tw_recycle`), and widening the ephemeral port range increases the 4-tuple space, directly relieving source-port exhaustion.
A Kubernetes Service has the correct label selector but `kubectl get endpoints my-svc` returns no addresses, even though three Pods with matching labels are `Running`. The Pods' `readinessProbe` is an HTTP GET on `/healthz` port 8080. Which is the most likely cause of the empty endpoint list?
- A. The Pods are Running but not Ready because the readiness probe is failing, so the endpoints controller excludes them ✓
- B. The Service `targetPort` does not match `containerPort`, which prevents the endpoints controller from populating addresses
- C. kube-proxy has not yet programmed iptables rules, so the endpoints object stays empty until it syncs
- D. The Pods lack a `podIP` because the CNI plugin has not assigned addresses to Running Pods
Correct answer: A. The EndpointSlice/endpoints controller only includes Pod IPs once the Pod passes its readiness probe; a failing `/healthz` keeps Pods Running-but-not-Ready and thus absent from endpoints, while targetPort mismatch or kube-proxy syncing would not empty the endpoints list.
You must import an existing resource into Terraform state, but two engineers ran `terraform apply` simultaneously against an S3 backend and the DynamoDB lock table now shows a stale lock from a crashed run. What is the correct and safest recovery?
- A. Delete the `.terraform/terraform.tfstate` local cache and re-run apply so Terraform re-acquires a fresh lock
- B. Run `terraform force-unlock <LOCK_ID>` using the ID reported in the error after confirming no apply is actually running ✓
- C. Manually delete the lock item from the DynamoDB table and immediately run `terraform apply`
- D. Set `-lock=false` on the next apply to bypass the stale lock and let Terraform overwrite it
Correct answer: B. `terraform force-unlock` with the reported LOCK_ID is the supported, auditable way to clear a confirmed-stale lock, whereas `-lock=false` or hand-deleting the DynamoDB item risks two concurrent writers corrupting state.
An application intermittently sees ~5 second stalls on outbound requests to an internal API. `dig api.internal` is fast, but tcpdump shows the app sending two simultaneous A and AAAA queries and one response is consistently lost, after which the resolver waits before retrying. On a musl/Alpine-based container, which mechanism best explains and fixes this?
- A. CoreDNS negative caching is returning NXDOMAIN for AAAA, so raising the negative TTL fixes the stall
- B. The 5s stall is the glibc/musl resolver retransmit timeout triggered by a dropped parallel A/AAAA UDP reply (a conntrack race); set `single-request-reopen` / use `use-vc` or disable parallel lookups ✓
- C. The MSS is too large for the DNS UDP packet, causing fragmentation drops; lowering MTU to 1400 resolves it
- D. CoreDNS is rate-limiting the pod, and increasing its cache size eliminates the retries
Correct answer: B. Parallel A/AAAA queries sharing a socket can hit a conntrack insert race that drops one reply, and the resolver's ~5s retransmit timeout produces the stall—mitigated by serializing queries (single-request/single-request-reopen) or forcing TCP, not by TTL or MTU changes.
Consider this systemd unit fragment for a service that must start only after the network is truly up and its data mount is available:
```
[Unit]
Wants=network-online.target
After=network.target
RequiresMountsFor=/data
```
The service still occasionally starts before connectivity exists. What is the precise fix?
- A. Change `Wants=network-online.target` to `Requires=network.target` so the dependency becomes mandatory
- B. Add `After=network-online.target` because `Wants` only pulls the target in but does not order the service after it ✓
- C. Replace `RequiresMountsFor=/data` with `After=data.mount` since RequiresMountsFor does not enforce ordering
- D. Add `Before=network.target` so the service is sequenced relative to the network stack
Correct answer: B. `Wants=` establishes a dependency (pulls the target in) but not ordering; you must also add `After=network-online.target` so the service is actually sequenced after connectivity is established.
A memory-sensitive service on a node with 32GB RAM and no swap is being OOM-killed while `free -m` shows ~8GB in `buff/cache`. The container has a cgroup v2 `memory.max` of 4GB. `memory.current` sits near 4GB and `memory.stat` shows large `active_file`/`inactive_file`. What is happening?
- A. Page cache counts against the cgroup's `memory.max`; under pressure the kernel reclaims reclaimable file pages first, but the workload's anonymous memory exceeds what's left, triggering the cgroup OOM killer ✓
- B. The node has 8GB of unreclaimable cache, so the global OOM killer fires despite the cgroup limit being irrelevant
- C. cgroup v2 does not account page cache, so the OOM must be caused by a kernel memory leak in slab
- D. Because there is no swap, `memory.max` is ignored and the process is killed by the node-level OOM killer at 32GB
Correct answer: A. In cgroup v2, page cache is charged to the cgroup and counts toward `memory.max`; the kernel reclaims clean/reclaimable file pages under pressure, but if anonymous (unreclaimable, unswappable) memory still exceeds the limit, the cgroup-level OOM killer fires.
You are designing delivery guarantees for a payment event consumer reading from a partitioned log (e.g., Kafka). The business requires that a duplicate delivery never double-charges. Which design is both correct and the least brittle at scale?
- A. Use exactly-once by committing the consumer offset before processing so a message is never reprocessed
- B. Rely on at-least-once delivery plus an idempotent handler keyed by a stable business idempotency key persisted transactionally with the side effect ✓
- C. Enable at-most-once delivery so duplicates are impossible, accepting occasional lost charges
- D. Deduplicate using an in-memory LRU set of recently seen message IDs on each consumer instance
Correct answer: B. True end-to-end exactly-once across external side effects is generally unachievable, so the robust pattern is at-least-once delivery with an idempotent handler that records a durable idempotency key in the same transaction as the effect; committing offsets first causes lost messages and an in-memory set fails across restarts and rebalances.
A read-heavy service fronts a database with a cache. When a hot key's TTL expires, thousands of concurrent requests all miss and stampede the database, briefly saturating it. Which mitigation most directly prevents the recomputation storm while keeping data reasonably fresh?
- A. Set the hot key's TTL to a much larger value so it expires far less often
- B. Add jitter to all TTLs so keys do not expire at the same wall-clock moment
- C. Use a per-key mutex/single-flight so only one request recomputes and repopulates while others wait or serve stale ✓
- D. Switch the cache eviction policy from LRU to LFU so hot keys are never evicted
Correct answer: C. A single-flight lock (or request coalescing) ensures exactly one caller recomputes the expired hot key while the rest wait or serve stale, directly stopping the thundering herd; TTL jitter helps correlated expiry across many keys but not a single hot key stampede.
During an incident, p50 latency is normal but p99 spiked 20x. CPU utilization averages 55%, and the service uses a fixed thread pool of 50 with a synchronous downstream call. A downstream dependency's p99 rose from 20ms to 800ms. Which explanation best accounts for the tail blowup despite modest average CPU?
- A. Average 55% CPU proves headroom exists, so the tail must be caused by GC pauses unrelated to the downstream
- B. Slow downstream calls occupy pool threads longer, so under bursty arrivals the queue builds and requests wait behind blocked threads—Little's Law/queueing means utilization of the pool, not CPU, is saturated at the tail ✓
- C. The load balancer is using round-robin instead of least-connections, which only affects p50
- D. The p99 spike is a measurement artifact of averaging CPU across cores and disappears with per-core metrics
Correct answer: B. With a bounded thread pool and synchronous blocking calls, slow downstream responses hold threads longer, so effective concurrency capacity (not CPU) saturates and queueing delay explodes the tail—classic Little's Law behavior where average CPU can look modest while the pool is the bottleneck.
What is the correct order of the TCP three-way handshake?
- A. SYN, ACK, SYN-ACK
- B. SYN, SYN-ACK, ACK ✓
- C. ACK, SYN, SYN-ACK
- D. SYN-ACK, SYN, ACK
Correct answer: B. The client sends SYN, the server replies SYN-ACK, and the client completes with ACK.
On a busy server, which side of a TCP connection accumulates sockets in the TIME_WAIT state?
- A. The side that passively closes the connection
- B. The side that actively closes, holding sockets for about 2×MSL ✓
- C. Any side experiencing SYN retransmissions
- D. The side receiving a SYN flood
Correct answer: B. The active closer enters TIME_WAIT for roughly twice the maximum segment lifetime to absorb stray packets.
The Linux OOM killer is invoked when:
- A. A process exceeds its file-descriptor limit
- B. The system is critically low on memory and must reclaim RAM ✓
- C. Swap is disabled at boot
- D. A process forks too many children
Correct answer: B. Under severe memory pressure the kernel kills a process (by oom_score) to free RAM.
Compared with a write-back cache, a write-through cache mainly provides:
- A. Higher write throughput
- B. Stronger consistency because cache and backing store are updated together, reducing data-loss risk ✓
- C. Lower read latency
- D. Reduced memory usage
Correct answer: B. Write-through writes to cache and storage simultaneously, so a cache failure does not lose committed data.
In Kubernetes, what does a pod's readiness probe control?
- A. Whether the container is restarted
- B. Whether the pod is added to Service endpoints and receives traffic ✓
- C. Whether the node is schedulable
- D. Whether the container image is pulled
Correct answer: B. A failing readiness probe removes the pod from Service endpoints so it stops receiving traffic.
How do SIGTERM and SIGKILL differ?
- A. Both can be caught by the process
- B. SIGTERM can be caught or handled for graceful shutdown; SIGKILL cannot be caught or ignored ✓
- C. SIGKILL allows cleanup while SIGTERM does not
- D. SIGTERM kills instantly while SIGKILL asks politely
Correct answer: B. SIGTERM is catchable for graceful exit, whereas SIGKILL is enforced by the kernel and uncatchable.
Setting `vm.swappiness` to 0 causes the kernel to:
- A. Disable virtual memory entirely
- B. Avoid swapping process pages out until it is nearly unavoidable ✓
- C. Force all memory into swap
- D. Enlarge the swap partition
Correct answer: B. A swappiness of 0 makes the kernel strongly prefer reclaiming page cache over swapping application memory.
During a TLS handshake, the server's certificate primarily lets the client:
- A. Encrypt the whole session using the certificate directly
- B. Verify the server's identity and obtain its public key ✓
- C. Generate the symmetric session key by itself
- D. Compress the handshake messages
Correct answer: B. The certificate authenticates the server and carries the public key used to establish the session key.
What is 'split-brain' in a high-availability cluster?
- A. A single node handling all traffic
- B. A network partition causing multiple nodes to each act as primary, risking data divergence ✓
- C. A CPU cache-coherency failure
- D. Memory being divided between two processes
Correct answer: B. When nodes lose communication, more than one may assume the primary role, corrupting shared state.
When multiple routes match a destination IP, the router selects the:
- A. Route with the lowest metric regardless of prefix
- B. Most specific route, i.e. the longest prefix match ✓
- C. First route in the table
- D. Route with the highest metric
Correct answer: B. IP forwarding uses longest-prefix match, choosing the most specific matching route.
The Linux OOM killer selects a process to terminate primarily based on:
- A. The lowest PID on the system
- B. A badness score derived from memory usage and oom_score_adj ✓
- C. The oldest running process
- D. The process consuming the most CPU
Correct answer: B. The kernel computes an oom_score from memory footprint and the tunable oom_score_adj to choose a victim.
A large number of sockets stuck in TIME_WAIT on a busy server is caused by:
- A. The remote end failing to send SYN
- B. The local end that actively closed the connection waiting to ensure the final ACK was received ✓
- C. Running out of inodes
- D. A misconfigured default gateway
Correct answer: B. The peer performing the active close enters TIME_WAIT to absorb delayed packets and confirm the final ACK.
A filesystem shows plenty of free space in `df` but writes fail with 'No space left on device'. The most likely cause is:
- A. A corrupted superblock
- B. Inode exhaustion (all inodes consumed) ✓
- C. Swap being full
- D. The disk mounted read-only
Correct answer: B. Many small files can exhaust inodes even when block space remains, blocking new file creation.
In RAID 5, the write penalty for a single small random write is approximately:
- A. 1 I/O operation
- B. 2 I/O operations
- C. 4 I/O operations ✓
- D. 6 I/O operations
Correct answer: C. A small write requires reading old data and parity, then writing new data and parity — four I/Os.
Linux containers achieve process and resource isolation primarily through:
- A. Hardware virtualization via a hypervisor
- B. Kernel namespaces and cgroups ✓
- C. A separate guest kernel per container
- D. chroot used alone
Correct answer: B. Namespaces isolate resource views while cgroups limit and account for resource usage, sharing the host kernel.
On a NUMA system, performance degrades most when:
- A. All memory accessed is local to the CPU
- B. A CPU frequently accesses memory attached to a remote node ✓
- C. Hyper-threading is disabled
- D. Swap is turned off
Correct answer: B. Remote-node memory access incurs higher latency than local access, hurting NUMA performance.
Lowering a DNS record's TTL shortly before a planned migration is done to:
- A. Increase resolver cache duration
- B. Reduce how long resolvers cache the old record so changes propagate faster ✓
- C. Improve DNSSEC validation
- D. Reduce authoritative server load
Correct answer: B. A lower TTL shortens caching so the new IP is picked up quickly after the cutover.
An LVM snapshot of a logical volume:
- A. Creates a full independent copy immediately
- B. Uses copy-on-write, storing only blocks changed since the snapshot ✓
- C. Cannot be created while the volume is mounted
- D. Requires the volume to be reformatted first
Correct answer: B. LVM snapshots are copy-on-write, capturing original blocks only as they change on the source volume.
The TCP three-way handshake sequence is:
- A. SYN, ACK, SYN
- B. SYN, SYN-ACK, ACK ✓
- C. ACK, SYN, ACK
- D. SYN, FIN, ACK
Correct answer: B. The client sends SYN, the server replies SYN-ACK, and the client completes with ACK.
In Kubernetes, a failing readiness probe on a pod causes:
- A. The pod to be restarted immediately
- B. The pod's endpoints to be removed from Service load balancing until it passes ✓
- C. The node to be cordoned
- D. The container image to be re-pulled
Correct answer: B. A failed readiness probe pulls the pod out of Service endpoints without restarting it, unlike a liveness probe.