A production server reports `df -h` showing `/var` at 96% full, yet `du -sh /var` sums to only 40% of the partition. `du` was run as root. The app was recently restarted after a log-rotation script `truncate`d nothing but instead `rm`'d a 20GB active log file. What is the single most reliable command to confirm the root cause and reclaim the space without a reboot?
- A. Run `sync && echo 3 > /proc/sys/vm/drop_caches` to flush the page cache holding the deleted file's dirty pages
- B. Run `lsof +L1` (or `lsof | grep deleted`) to find the process holding the unlinked inode, then restart or signal that process to close the fd ✓
- C. Run `fsck` on the `/var` filesystem to reclaim orphaned inodes left by the failed `rm`
- D. Run `fstrim /var` to release blocks that the SSD firmware has not yet marked as free after the deletion
Correct answer: B. An unlinked file with an open file handle keeps its data blocks allocated until every fd is closed, so `lsof +L1` reveals the holding process and closing it (or the app was already restarted, releasing it) reclaims the space—no reboot or fsck needed.
A high-throughput reverse proxy under load starts refusing new outbound connections to a backend with `EADDRNOTAVAIL`, while `ss -s` shows tens of thousands of sockets in TIME_WAIT toward that single backend IP:port. Which change most directly addresses the exhaustion without risking data corruption from delayed duplicate segments?
- A. Set `net.ipv4.tcp_tw_recycle=1` to aggressively recycle TIME_WAIT sockets globally
- B. Lower `net.ipv4.tcp_fin_timeout` to 5 seconds so sockets leave TIME_WAIT much faster
- C. Enable `net.ipv4.tcp_tw_reuse=1` and widen `net.ipv4.ip_local_port_range` so the client can safely reuse TIME_WAIT sockets for new outbound connections ✓
- D. Increase `net.core.somaxconn` and the backend's listen backlog to accept the pending connections
Correct answer: C. `tcp_tw_reuse` safely reuses TIME_WAIT sockets for new outbound connections using timestamps (unlike the removed/unsafe `tcp_tw_recycle`), and widening the ephemeral port range increases the 4-tuple space, directly relieving source-port exhaustion.
A Kubernetes Service has the correct label selector but `kubectl get endpoints my-svc` returns no addresses, even though three Pods with matching labels are `Running`. The Pods' `readinessProbe` is an HTTP GET on `/healthz` port 8080. Which is the most likely cause of the empty endpoint list?
- A. The Pods are Running but not Ready because the readiness probe is failing, so the endpoints controller excludes them ✓
- B. The Service `targetPort` does not match `containerPort`, which prevents the endpoints controller from populating addresses
- C. kube-proxy has not yet programmed iptables rules, so the endpoints object stays empty until it syncs
- D. The Pods lack a `podIP` because the CNI plugin has not assigned addresses to Running Pods
Correct answer: A. The EndpointSlice/endpoints controller only includes Pod IPs once the Pod passes its readiness probe; a failing `/healthz` keeps Pods Running-but-not-Ready and thus absent from endpoints, while targetPort mismatch or kube-proxy syncing would not empty the endpoints list.
You must import an existing resource into Terraform state, but two engineers ran `terraform apply` simultaneously against an S3 backend and the DynamoDB lock table now shows a stale lock from a crashed run. What is the correct and safest recovery?
- A. Delete the `.terraform/terraform.tfstate` local cache and re-run apply so Terraform re-acquires a fresh lock
- B. Run `terraform force-unlock <LOCK_ID>` using the ID reported in the error after confirming no apply is actually running ✓
- C. Manually delete the lock item from the DynamoDB table and immediately run `terraform apply`
- D. Set `-lock=false` on the next apply to bypass the stale lock and let Terraform overwrite it
Correct answer: B. `terraform force-unlock` with the reported LOCK_ID is the supported, auditable way to clear a confirmed-stale lock, whereas `-lock=false` or hand-deleting the DynamoDB item risks two concurrent writers corrupting state.
An application intermittently sees ~5 second stalls on outbound requests to an internal API. `dig api.internal` is fast, but tcpdump shows the app sending two simultaneous A and AAAA queries and one response is consistently lost, after which the resolver waits before retrying. On a musl/Alpine-based container, which mechanism best explains and fixes this?
- A. CoreDNS negative caching is returning NXDOMAIN for AAAA, so raising the negative TTL fixes the stall
- B. The 5s stall is the glibc/musl resolver retransmit timeout triggered by a dropped parallel A/AAAA UDP reply (a conntrack race); set `single-request-reopen` / use `use-vc` or disable parallel lookups ✓
- C. The MSS is too large for the DNS UDP packet, causing fragmentation drops; lowering MTU to 1400 resolves it
- D. CoreDNS is rate-limiting the pod, and increasing its cache size eliminates the retries
Correct answer: B. Parallel A/AAAA queries sharing a socket can hit a conntrack insert race that drops one reply, and the resolver's ~5s retransmit timeout produces the stall—mitigated by serializing queries (single-request/single-request-reopen) or forcing TCP, not by TTL or MTU changes.
Consider this systemd unit fragment for a service that must start only after the network is truly up and its data mount is available:
```
[Unit]
Wants=network-online.target
After=network.target
RequiresMountsFor=/data
```
The service still occasionally starts before connectivity exists. What is the precise fix?
- A. Change `Wants=network-online.target` to `Requires=network.target` so the dependency becomes mandatory
- B. Add `After=network-online.target` because `Wants` only pulls the target in but does not order the service after it ✓
- C. Replace `RequiresMountsFor=/data` with `After=data.mount` since RequiresMountsFor does not enforce ordering
- D. Add `Before=network.target` so the service is sequenced relative to the network stack
Correct answer: B. `Wants=` establishes a dependency (pulls the target in) but not ordering; you must also add `After=network-online.target` so the service is actually sequenced after connectivity is established.
A memory-sensitive service on a node with 32GB RAM and no swap is being OOM-killed while `free -m` shows ~8GB in `buff/cache`. The container has a cgroup v2 `memory.max` of 4GB. `memory.current` sits near 4GB and `memory.stat` shows large `active_file`/`inactive_file`. What is happening?
- A. Page cache counts against the cgroup's `memory.max`; under pressure the kernel reclaims reclaimable file pages first, but the workload's anonymous memory exceeds what's left, triggering the cgroup OOM killer ✓
- B. The node has 8GB of unreclaimable cache, so the global OOM killer fires despite the cgroup limit being irrelevant
- C. cgroup v2 does not account page cache, so the OOM must be caused by a kernel memory leak in slab
- D. Because there is no swap, `memory.max` is ignored and the process is killed by the node-level OOM killer at 32GB
Correct answer: A. In cgroup v2, page cache is charged to the cgroup and counts toward `memory.max`; the kernel reclaims clean/reclaimable file pages under pressure, but if anonymous (unreclaimable, unswappable) memory still exceeds the limit, the cgroup-level OOM killer fires.
You are designing delivery guarantees for a payment event consumer reading from a partitioned log (e.g., Kafka). The business requires that a duplicate delivery never double-charges. Which design is both correct and the least brittle at scale?
- A. Use exactly-once by committing the consumer offset before processing so a message is never reprocessed
- B. Rely on at-least-once delivery plus an idempotent handler keyed by a stable business idempotency key persisted transactionally with the side effect ✓
- C. Enable at-most-once delivery so duplicates are impossible, accepting occasional lost charges
- D. Deduplicate using an in-memory LRU set of recently seen message IDs on each consumer instance
Correct answer: B. True end-to-end exactly-once across external side effects is generally unachievable, so the robust pattern is at-least-once delivery with an idempotent handler that records a durable idempotency key in the same transaction as the effect; committing offsets first causes lost messages and an in-memory set fails across restarts and rebalances.
A read-heavy service fronts a database with a cache. When a hot key's TTL expires, thousands of concurrent requests all miss and stampede the database, briefly saturating it. Which mitigation most directly prevents the recomputation storm while keeping data reasonably fresh?
- A. Set the hot key's TTL to a much larger value so it expires far less often
- B. Add jitter to all TTLs so keys do not expire at the same wall-clock moment
- C. Use a per-key mutex/single-flight so only one request recomputes and repopulates while others wait or serve stale ✓
- D. Switch the cache eviction policy from LRU to LFU so hot keys are never evicted
Correct answer: C. A single-flight lock (or request coalescing) ensures exactly one caller recomputes the expired hot key while the rest wait or serve stale, directly stopping the thundering herd; TTL jitter helps correlated expiry across many keys but not a single hot key stampede.
During an incident, p50 latency is normal but p99 spiked 20x. CPU utilization averages 55%, and the service uses a fixed thread pool of 50 with a synchronous downstream call. A downstream dependency's p99 rose from 20ms to 800ms. Which explanation best accounts for the tail blowup despite modest average CPU?
- A. Average 55% CPU proves headroom exists, so the tail must be caused by GC pauses unrelated to the downstream
- B. Slow downstream calls occupy pool threads longer, so under bursty arrivals the queue builds and requests wait behind blocked threads—Little's Law/queueing means utilization of the pool, not CPU, is saturated at the tail ✓
- C. The load balancer is using round-robin instead of least-connections, which only affects p50
- D. The p99 spike is a measurement artifact of averaging CPU across cores and disappears with per-core metrics
Correct answer: B. With a bounded thread pool and synchronous blocking calls, slow downstream responses hold threads longer, so effective concurrency capacity (not CPU) saturates and queueing delay explodes the tail—classic Little's Law behavior where average CPU can look modest while the pool is the bottleneck.