A 3-node StatefulSet (Cassandra) runs one pod per node. Node-2 suddenly loses network connectivity and its kubelet stops reporting. After the node goes NotReady, an operator notices the pod for node-2 stays in `Terminating` indefinitely and no replacement pod is created on a healthy node. Why does Kubernetes deliberately NOT reschedule this StatefulSet pod automatically?
- A. The StatefulSet controller lacks a pod-eviction reconciler, so only Deployments auto-reschedule on node failure
- B. The API server cannot confirm the old pod is truly gone; creating a same-identity replacement risks two pods with the same stable identity/volume, so it waits for confirmed deletion (force-delete or node removal) ✓
- C. PodDisruptionBudget blocks the eviction because minAvailable is set to 3
- D. The default-scheduler has no other node with sufficient CPU, so the pod stays Pending as Terminating
Correct answer: B. For StatefulSets a partitioned/unreachable node's pod is only marked Terminating; Kubernetes will not create a replacement with the same identity until deletion is confirmed, to avoid split-brain on the stable network ID and volume.
A pod has `requests.memory: 256Mi` and `limits.memory: 512Mi`. Under a memory-pressure event the node's kernel OOM killer fires and this pod's container is killed (exit 137) even though it was using ~300Mi, while a neighboring pod using 1Gi survived. The neighbor has `requests.memory: 1Gi, limits.memory: 1Gi`. What best explains the kernel's choice?
- A. The neighbor was killed first but restarted faster, so it appears to have survived
- B. The OOM killer always kills the pod with the highest absolute RSS, so the 300Mi pod should not have died
- C. Guaranteed QoS (requests==limits) pods get a lower oom_score_adj, while Burstable pods exceeding their requests get a higher oom_score_adj, making the Burstable pod a preferred kill target ✓
- D. cgroup v2 disables per-container OOM scoring, so kills are purely random across the node
Correct answer: C. Kubernetes sets oom_score_adj by QoS class; Guaranteed pods are most protected while Burstable pods using memory above their request receive a higher (worse) score, so they are killed first under node pressure regardless of absolute usage.
Your service has a 99.9% availability SLO over a 30-day window. Multi-window multi-burn-rate alerting uses a fast page when a 1-hour burn rate is high AND a 5-minute burn rate confirms it. If you page at a burn rate of 14.4, roughly how much of the 30-day error budget would be consumed in that 1-hour window if the burn continued, and why is the 14.4 threshold chosen?
- A. About 2% of the budget; 14.4 is chosen because it equals 1000 minutes of downtime
- B. About 2% of the budget; 14.4x burn over 1 hour spends ~2% of a 30-day budget, giving a fast but still-meaningful page while the short second window suppresses spikes ✓
- C. About 50% of the budget; 14.4 corresponds to burning half the month in one hour
- D. About 0.1% of the budget; 14.4 is simply 99.9 divided by 6.9
Correct answer: B. A 14.4x burn rate consumes ~2% of the monthly budget in one hour (14.4 x (1h/720h) ≈ 2%), the canonical Google SRE fast-burn threshold, with a short secondary window to avoid alerting on transient spikes.
Two engineers run `terraform apply` against the same S3-backed state with DynamoDB locking. Engineer A's apply crashes (laptop dies) mid-write. Engineer B now gets `Error acquiring the state lock` on every command. What is the correct and safest recovery?
- A. Run `terraform force-unlock <LOCK_ID>` after confirming A's process is dead and no apply is in flight, then re-plan to check state consistency ✓
- B. Delete the DynamoDB lock table so future runs never lock again
- C. Run `terraform apply -lock=false` immediately to bypass the stuck lock
- D. Manually edit terraform.tfstate in S3 to remove the lock field, then apply
Correct answer: A. The lock is a DynamoDB item, not part of state JSON; after verifying no apply is actually running you use `terraform force-unlock <ID>` and re-plan, whereas disabling locking or deleting the table invites concurrent-write corruption.
A 3-node etcd cluster backs your Kubernetes control plane. One etcd member's disk fills and it starts returning slow/failed writes. You observe API server latency spikes and some writes failing, but reads mostly work. Why does losing (or degrading) just ONE of THREE members cause write problems, and what is the quorum math?
- A. Three members need all 3 for any write (quorum = N); losing one halts all writes
- B. Quorum is majority = 2 of 3; the cluster can tolerate 1 full failure, but a degraded (slow-but-alive) member can stall the Raft leader's commit latency and cause write timeouts even with quorum technically available ✓
- C. etcd uses primary-backup, so the single degraded node is the leader and no failover is possible
- D. Reads require quorum but writes do not, which is why reads fail and writes succeed
Correct answer: B. Quorum for 3 members is 2, so one dead member is tolerable, but a slow-but-alive member can drag down Raft commit latency and leader elections, producing write timeouts even though a bare majority exists.
You must implement a token-bucket rate limiter (capacity C, refill rate R tokens/sec) that allows bursts up to C and is checked on every request. Which implementation correctly refills lazily without a background timer and handles bursts?
- A. On each request, set tokens = min(C, R); if tokens >= 1 allow and decrement
- B. On each request, compute elapsed = now - last; tokens = min(C, tokens + elapsed*R); last = now; if tokens >= 1 then tokens -= 1 and allow, else reject ✓
- C. On each request, if request_count in the current 1-second wall-clock window < R, allow; reset counter each second
- D. On each request, tokens += R; if tokens > 1 allow and set tokens = 0
Correct answer: B. Correct lazy token-bucket accrues tokens = min(capacity, tokens + elapsed*rate) since the last check, then spends one, which permits bursts up to C and needs no timer thread; the fixed-window counter is a different algorithm with edge-burst problems.
A microservice intermittently fails with `dial tcp: i/o timeout` connecting to an upstream, but only for payloads larger than ~1400 bytes; small requests succeed. `ping` works, DNS resolves, TCP handshake completes. Which root cause is most consistent with these symptoms?
- A. The upstream's readiness probe is failing, dropping large requests
- B. A path MTU / MTU mismatch (e.g., a tunnel/overlay reducing MTU) is black-holing large packets because ICMP 'fragmentation needed' is being filtered, so only sub-MTU payloads pass ✓
- C. DNS TTL expired mid-connection, forcing re-resolution only on large payloads
- D. The connection pool is exhausted only when payloads exceed 1400 bytes
Correct answer: B. Handshake succeeds (small packets) but large payloads hang — the classic PMTU black hole where ICMP 'fragmentation needed' is filtered, so packets above the path MTU are silently dropped.
You run Prometheus with a metric `http_requests_total{path, method, status, user_id}`. Ingestion is fine at first but after weeks Prometheus OOMs and queries slow drastically. The `user_id` label has ~2 million distinct values. What is the core problem and the correct fix?
- A. Retention is too long; reduce retention to 7 days to fix memory
- B. High cardinality: each unique label-set is a separate time series, so `user_id` explodes series count and index memory; remove/aggregate the unbounded label (drop user_id) or move it to logs/traces ✓
- C. Scrape interval is too short; increase it to 5 minutes to reduce series
- D. Prometheus needs remote_write enabled; that offloads all series memory automatically
Correct answer: B. An unbounded label like user_id multiplies the number of distinct time series (each stored/indexed separately), so the fix is to drop the high-cardinality label and put per-user detail in logs/traces, not metrics.
During an incident, retries with a fixed 1s interval from thousands of clients hitting a briefly-degraded backend cause a sustained overload that outlasts the original fault (the backend never recovers). Which combination most directly breaks this thundering-herd/retry-storm dynamic?
- A. Increase the backend's connection timeout and add more retries
- B. Exponential backoff WITH jitter plus a circuit breaker to stop hammering an unhealthy dependency, optionally with load shedding at the backend ✓
- C. Switch clients to synchronous retries so they serialize naturally
- D. Raise the client-side retry count so eventually one succeeds
Correct answer: B. Fixed-interval synchronized retries create a self-sustaining herd; exponential backoff with jitter de-synchronizes clients and a circuit breaker halts requests to an unhealthy backend so it can recover, while load-shedding protects it under overload.
You need the top-10 most frequent failing endpoints from a 200GB access log that cannot fit in memory, and you want bounded memory. Which approach is correct and memory-bounded?
- A. Load the whole file into a hash map of endpoint->count, then sort — it's O(n) so memory is fine
- B. Stream line-by-line, maintain a hash map keyed only by endpoint (bounded by the number of distinct endpoints, which is small) incrementing counts, then take the top 10; or use a bounded count-min sketch + heap if distinct keys are huge ✓
- C. Sort the entire 200GB file first with an external merge sort, then scan — this is the only correct method
- D. Sample 1% of lines randomly; the top-10 from the sample is always exactly correct
Correct answer: B. Distinct endpoints are typically bounded and small, so a streaming pass with a per-endpoint counter uses memory proportional to distinct keys (not file size); if the keyspace were huge, a count-min sketch plus a top-k heap keeps memory bounded.