A 3-node StatefulSet (Cassandra) runs one pod per node. Node-2 suddenly loses network connectivity and its kubelet stops reporting. After the node goes NotReady, an operator notices the pod for node-2 stays in `Terminating` indefinitely and no replacement pod is created on a healthy node. Why does Kubernetes deliberately NOT reschedule this StatefulSet pod automatically?
- A. The StatefulSet controller lacks a pod-eviction reconciler, so only Deployments auto-reschedule on node failure
- B. The API server cannot confirm the old pod is truly gone; creating a same-identity replacement risks two pods with the same stable identity/volume, so it waits for confirmed deletion (force-delete or node removal) ✓
- C. PodDisruptionBudget blocks the eviction because minAvailable is set to 3
- D. The default-scheduler has no other node with sufficient CPU, so the pod stays Pending as Terminating
Correct answer: B. For StatefulSets a partitioned/unreachable node's pod is only marked Terminating; Kubernetes will not create a replacement with the same identity until deletion is confirmed, to avoid split-brain on the stable network ID and volume.
A pod has `requests.memory: 256Mi` and `limits.memory: 512Mi`. Under a memory-pressure event the node's kernel OOM killer fires and this pod's container is killed (exit 137) even though it was using ~300Mi, while a neighboring pod using 1Gi survived. The neighbor has `requests.memory: 1Gi, limits.memory: 1Gi`. What best explains the kernel's choice?
- A. The neighbor was killed first but restarted faster, so it appears to have survived
- B. The OOM killer always kills the pod with the highest absolute RSS, so the 300Mi pod should not have died
- C. Guaranteed QoS (requests==limits) pods get a lower oom_score_adj, while Burstable pods exceeding their requests get a higher oom_score_adj, making the Burstable pod a preferred kill target ✓
- D. cgroup v2 disables per-container OOM scoring, so kills are purely random across the node
Correct answer: C. Kubernetes sets oom_score_adj by QoS class; Guaranteed pods are most protected while Burstable pods using memory above their request receive a higher (worse) score, so they are killed first under node pressure regardless of absolute usage.
Your service has a 99.9% availability SLO over a 30-day window. Multi-window multi-burn-rate alerting uses a fast page when a 1-hour burn rate is high AND a 5-minute burn rate confirms it. If you page at a burn rate of 14.4, roughly how much of the 30-day error budget would be consumed in that 1-hour window if the burn continued, and why is the 14.4 threshold chosen?
- A. About 2% of the budget; 14.4 is chosen because it equals 1000 minutes of downtime
- B. About 2% of the budget; 14.4x burn over 1 hour spends ~2% of a 30-day budget, giving a fast but still-meaningful page while the short second window suppresses spikes ✓
- C. About 50% of the budget; 14.4 corresponds to burning half the month in one hour
- D. About 0.1% of the budget; 14.4 is simply 99.9 divided by 6.9
Correct answer: B. A 14.4x burn rate consumes ~2% of the monthly budget in one hour (14.4 x (1h/720h) ≈ 2%), the canonical Google SRE fast-burn threshold, with a short secondary window to avoid alerting on transient spikes.
Two engineers run `terraform apply` against the same S3-backed state with DynamoDB locking. Engineer A's apply crashes (laptop dies) mid-write. Engineer B now gets `Error acquiring the state lock` on every command. What is the correct and safest recovery?
- A. Run `terraform force-unlock <LOCK_ID>` after confirming A's process is dead and no apply is in flight, then re-plan to check state consistency ✓
- B. Delete the DynamoDB lock table so future runs never lock again
- C. Run `terraform apply -lock=false` immediately to bypass the stuck lock
- D. Manually edit terraform.tfstate in S3 to remove the lock field, then apply
Correct answer: A. The lock is a DynamoDB item, not part of state JSON; after verifying no apply is actually running you use `terraform force-unlock <ID>` and re-plan, whereas disabling locking or deleting the table invites concurrent-write corruption.
A 3-node etcd cluster backs your Kubernetes control plane. One etcd member's disk fills and it starts returning slow/failed writes. You observe API server latency spikes and some writes failing, but reads mostly work. Why does losing (or degrading) just ONE of THREE members cause write problems, and what is the quorum math?
- A. Three members need all 3 for any write (quorum = N); losing one halts all writes
- B. Quorum is majority = 2 of 3; the cluster can tolerate 1 full failure, but a degraded (slow-but-alive) member can stall the Raft leader's commit latency and cause write timeouts even with quorum technically available ✓
- C. etcd uses primary-backup, so the single degraded node is the leader and no failover is possible
- D. Reads require quorum but writes do not, which is why reads fail and writes succeed
Correct answer: B. Quorum for 3 members is 2, so one dead member is tolerable, but a slow-but-alive member can drag down Raft commit latency and leader elections, producing write timeouts even though a bare majority exists.
You must implement a token-bucket rate limiter (capacity C, refill rate R tokens/sec) that allows bursts up to C and is checked on every request. Which implementation correctly refills lazily without a background timer and handles bursts?
- A. On each request, set tokens = min(C, R); if tokens >= 1 allow and decrement
- B. On each request, compute elapsed = now - last; tokens = min(C, tokens + elapsed*R); last = now; if tokens >= 1 then tokens -= 1 and allow, else reject ✓
- C. On each request, if request_count in the current 1-second wall-clock window < R, allow; reset counter each second
- D. On each request, tokens += R; if tokens > 1 allow and set tokens = 0
Correct answer: B. Correct lazy token-bucket accrues tokens = min(capacity, tokens + elapsed*rate) since the last check, then spends one, which permits bursts up to C and needs no timer thread; the fixed-window counter is a different algorithm with edge-burst problems.
A microservice intermittently fails with `dial tcp: i/o timeout` connecting to an upstream, but only for payloads larger than ~1400 bytes; small requests succeed. `ping` works, DNS resolves, TCP handshake completes. Which root cause is most consistent with these symptoms?
- A. The upstream's readiness probe is failing, dropping large requests
- B. A path MTU / MTU mismatch (e.g., a tunnel/overlay reducing MTU) is black-holing large packets because ICMP 'fragmentation needed' is being filtered, so only sub-MTU payloads pass ✓
- C. DNS TTL expired mid-connection, forcing re-resolution only on large payloads
- D. The connection pool is exhausted only when payloads exceed 1400 bytes
Correct answer: B. Handshake succeeds (small packets) but large payloads hang — the classic PMTU black hole where ICMP 'fragmentation needed' is filtered, so packets above the path MTU are silently dropped.
You run Prometheus with a metric `http_requests_total{path, method, status, user_id}`. Ingestion is fine at first but after weeks Prometheus OOMs and queries slow drastically. The `user_id` label has ~2 million distinct values. What is the core problem and the correct fix?
- A. Retention is too long; reduce retention to 7 days to fix memory
- B. High cardinality: each unique label-set is a separate time series, so `user_id` explodes series count and index memory; remove/aggregate the unbounded label (drop user_id) or move it to logs/traces ✓
- C. Scrape interval is too short; increase it to 5 minutes to reduce series
- D. Prometheus needs remote_write enabled; that offloads all series memory automatically
Correct answer: B. An unbounded label like user_id multiplies the number of distinct time series (each stored/indexed separately), so the fix is to drop the high-cardinality label and put per-user detail in logs/traces, not metrics.
During an incident, retries with a fixed 1s interval from thousands of clients hitting a briefly-degraded backend cause a sustained overload that outlasts the original fault (the backend never recovers). Which combination most directly breaks this thundering-herd/retry-storm dynamic?
- A. Increase the backend's connection timeout and add more retries
- B. Exponential backoff WITH jitter plus a circuit breaker to stop hammering an unhealthy dependency, optionally with load shedding at the backend ✓
- C. Switch clients to synchronous retries so they serialize naturally
- D. Raise the client-side retry count so eventually one succeeds
Correct answer: B. Fixed-interval synchronized retries create a self-sustaining herd; exponential backoff with jitter de-synchronizes clients and a circuit breaker halts requests to an unhealthy backend so it can recover, while load-shedding protects it under overload.
You need the top-10 most frequent failing endpoints from a 200GB access log that cannot fit in memory, and you want bounded memory. Which approach is correct and memory-bounded?
- A. Load the whole file into a hash map of endpoint->count, then sort — it's O(n) so memory is fine
- B. Stream line-by-line, maintain a hash map keyed only by endpoint (bounded by the number of distinct endpoints, which is small) incrementing counts, then take the top 10; or use a bounded count-min sketch + heap if distinct keys are huge ✓
- C. Sort the entire 200GB file first with an external merge sort, then scan — this is the only correct method
- D. Sample 1% of lines randomly; the top-10 from the sample is always exactly correct
Correct answer: B. Distinct endpoints are typically bounded and small, so a streaming pass with a per-endpoint counter uses memory proportional to distinct keys (not file size); if the keyspace were huge, a count-min sketch plus a top-k heap keeps memory bounded.
A Kubernetes pod is OOMKilled repeatedly even though node memory appears free. The container limit is 512Mi. Which explanation is most accurate?
- A. The node kernel randomly kills pods
- B. The container's cgroup memory limit (512Mi) was exceeded, independent of node free memory ✓
- C. requests must equal limits or pods are killed
- D. The scheduler evicted it due to node pressure
Correct answer: B. A container's memory limit sets its cgroup limit; exceeding it triggers an OOMKill regardless of overall node free memory.
During a rolling update you notice brief 502s despite readiness probes. What is the most likely root cause of dropped in-flight requests on old pods?
- A. Liveness probe misconfigured
- B. No preStop hook/graceful shutdown, so the pod stops serving before it is removed from Service endpoints ✓
- C. HPA scaled down too fast
- D. Image digest mismatch
Correct answer: B. Without a preStop delay/graceful shutdown, a pod can stop serving on SIGTERM before endpoints are drained, dropping in-flight requests.
Your 99.9% SLO error budget is 80% consumed by mid-month. Per SRE practice, what is the appropriate action?
- A. Immediately raise the SLO to 99.99%
- B. Slow or freeze risky feature releases and prioritize reliability work ✓
- C. Ignore it since the budget isn't fully spent
- D. Delete the alerting rules
Correct answer: B. A rapidly depleting error budget signals the team should throttle risky changes and shift effort to reliability.
Two engineers run 'terraform apply' against the same remote state simultaneously without state locking. What is the primary risk?
- A. Slower plans
- B. State corruption/conflicting writes producing drift or lost resources ✓
- C. Provider version conflict
- D. Duplicate variable definitions
Correct answer: B. Concurrent writes to shared state without locking can corrupt the state file and desync it from real infrastructure.
A Prometheus query 'rate(http_requests_total[1m])' returns misleadingly low values during a spike while '[5m]' looks correct. Likely cause?
- A. Counter reset detection failed
- B. The scrape interval is close to the 1m window, giving too few samples for an accurate rate ✓
- C. Histograms are not supported
- D. The metric is a gauge
Correct answer: B. rate() needs at least two samples in the window; a short window near the scrape interval yields too few points for a reliable rate.
You must give an EKS pod temporary AWS S3 access without long-lived credentials. Which approach is recommended?
- A. Hardcode access keys in a ConfigMap
- B. IAM Roles for Service Accounts (IRSA) via OIDC ✓
- C. Attach the node's instance role broadly to all pods
- D. Store keys in an environment variable in the image
Correct answer: B. IRSA maps a Kubernetes service account to an IAM role via OIDC, giving pods scoped, short-lived credentials.
A blue-green cutover to green left a subset of users still hitting blue for several minutes. Most probable cause?
- A. Green pods never became ready
- B. DNS TTL or connection keep-alive caching kept routing some clients to blue ✓
- C. The database migration failed
- D. The load balancer was deleted
Correct answer: B. DNS TTL caching and persistent keep-alive connections can keep some clients on the old (blue) target after the switch.
In etcd, why is running an even number of members (e.g., 4) discouraged?
- A. It doubles storage cost
- B. It provides no better fault tolerance than the next-lower odd count and worsens quorum math ✓
- C. etcd only supports 3 members
- D. It disables TLS
Correct answer: B. Quorum requires a majority; an even count tolerates the same failures as the next-lower odd count while raising split-brain risk, so odd counts are preferred.
A Linux server shows high load average but low CPU utilization. Most likely explanation?
- A. CPU is thermally throttled
- B. Many processes are blocked in uninterruptible I/O wait (D state) ✓
- C. Swap is disabled
- D. The kernel is idle
Correct answer: B. Linux load average counts uninterruptible-sleep (D-state) processes, so heavy I/O wait raises load without raising CPU usage.
You enable a CPU-based HPA but pods never scale up during real overload. Which misconfiguration commonly causes this?
- A. The Deployment has too many replicas
- B. No CPU 'requests' are set, so HPA cannot compute a utilization percentage ✓
- C. The image is too large
- D. The Service is of type NodePort
Correct answer: B. CPU-based HPA computes utilization relative to the pod's CPU request; without a request it has no baseline and cannot scale.
A Kubernetes pod using a PersistentVolumeClaim with an EBS-backed StorageClass fails to schedule onto a node in another availability zone. What is the most likely cause?
- A. The image pull secret is missing
- B. EBS volumes are AZ-scoped, so the pod must schedule in the volume's AZ ✓
- C. The CNI plugin is misconfigured
- D. The readiness probe is failing
Correct answer: B. EBS volumes are bound to a single Availability Zone, so a pod using that volume can only schedule on nodes in the same AZ.
You observe that a Kubernetes Deployment rollout updates all pods at once causing downtime, despite RollingUpdate strategy. Which field most likely caused this?
- A. maxUnavailable set to 100% (or equal to replicas) ✓
- B. revisionHistoryLimit set too low
- C. terminationGracePeriodSeconds too high
- D. progressDeadlineSeconds too low
Correct answer: A. A maxUnavailable equal to the replica count lets all old pods be taken down simultaneously, causing downtime.
In Prometheus, rate() on a counter shows correct values even after a target restart resets the counter to zero. Why?
- A. It ignores restarts entirely
- B. rate() automatically handles counter resets by treating a decrease as a reset ✓
- C. Counters never reset
- D. It uses the raw delta without adjustment
Correct answer: B. Prometheus rate()/increase() detect counter resets (a drop) and compensate, so restarts don't produce false negatives.
A Terraform apply intermittently fails with a state lock error in a team using an S3 backend. What is the correct fix?
- A. Set parallelism to 1
- B. Enable a DynamoDB table for state locking in the backend config ✓
- C. Delete the .terraform directory
- D. Use terraform refresh before apply
Correct answer: B. An S3 backend needs a DynamoDB lock table to serialize concurrent state operations and prevent corruption.
Your service behind an L7 load balancer shows healthy targets but users get intermittent 502s during deploys. Which mechanism most directly prevents this?
- A. Increasing the health check interval
- B. Connection draining / graceful shutdown with preStop hook and matching timeouts ✓
- C. Lowering the pod CPU request
- D. Disabling keep-alive
Correct answer: B. Connection draining plus a preStop hook lets in-flight requests finish before the pod terminates, avoiding 502s.
In a Linux system, a process is stuck in uninterruptible sleep (state D) and cannot be killed with SIGKILL. What does this indicate?
- A. The process is a zombie
- B. It is blocked in a kernel I/O call, often on unresponsive storage/NFS ✓
- C. Its niceness is too high
- D. It is being traced by a debugger
Correct answer: B. State D means uninterruptible sleep, typically waiting on I/O; signals are deferred until the kernel call returns.
You run a multi-stage Docker build but the final image is still large. Which practice most reliably shrinks it?
- A. Add more RUN layers
- B. Copy only the compiled artifacts into a minimal final base image such as distroless or alpine ✓
- C. Use ADD instead of COPY
- D. Set WORKDIR to /
Correct answer: B. Multi-stage builds shrink images by copying only needed artifacts into a minimal final base, discarding build tooling.
A Kubernetes HorizontalPodAutoscaler is not scaling up despite high load. metrics-server is running. What is the most common misconfiguration?
- A. The pods have no CPU/memory requests set, so utilization can't be computed ✓
- B. The HPA minReplicas is too high
- C. The Deployment uses RollingUpdate
- D. The Service is of type NodePort
Correct answer: A. CPU-based HPAs compute utilization as a percentage of the request; without requests set, the HPA cannot scale.
During an incident, etcd latency spikes and the whole Kubernetes control plane slows down. Which is the most appropriate long-term mitigation?
- A. Increase pod replica counts
- B. Provide low-latency SSD disks and isolate etcd, keeping it well below its DB size limit ✓
- C. Disable RBAC
- D. Increase kube-proxy sync period
Correct answer: B. etcd is disk-latency sensitive; fast dedicated SSDs and staying under the DB size limit keep the control plane responsive.
In a GitOps workflow with a controller like Argo CD, someone manually edits a live resource with kubectl. What happens by default with auto-sync and self-heal enabled?
- A. The change persists permanently
- B. The controller detects drift and reverts the resource to match the Git-declared state ✓
- C. The controller deletes the namespace
- D. The change is committed back to Git automatically
Correct answer: B. With self-heal, the GitOps controller treats Git as the source of truth and reverts out-of-band manual changes.