A fraud model on a dataset with 0.5% positives reports 99.4% accuracy, ROC-AUC 0.91, but the fraud team says it misses most fraud in production at the default 0.5 threshold. Which single diagnostic best explains the gap and points to the fix?
- A. ROC-AUC is invalid on imbalanced data, so retrain with balanced class weights before trusting any metric
- B. The PR curve / PR-AUC shows precision-recall tradeoff at operating thresholds; 0.5 sits where recall collapses, so tune the threshold on the PR curve to the business cost ratio ✓
- C. Accuracy is high, so the model is fine and the issue is label noise in the production fraud reports
- D. ROC-AUC of 0.91 proves the model ranks well, so the only fix is to oversample the minority class with SMOTE until classes are balanced
Correct answer: B. ROC-AUC can look strong under imbalance because true negatives dominate; the PR curve exposes the collapse in recall at 0.5 and the correct fix is choosing a cost-driven threshold rather than blindly rebalancing.
You run this query to compute month-over-month revenue growth:
SELECT month, revenue,
(revenue - LAG(revenue) OVER (ORDER BY month)) / LAG(revenue) OVER (ORDER BY month) AS growth
FROM monthly_rev;
The table has multiple months per region (region column present) but you forgot to partition. What happens?
- A. It errors because LAG requires a PARTITION BY clause when the table has a grouping column
- B. It silently computes growth across region-interleaved rows, so LAG pulls the prior row which may belong to a different region, producing meaningless growth values ✓
- C. It automatically partitions by region because region is a column in the table
- D. It returns NULL for every row because ORDER BY month alone is not a valid window frame
Correct answer: B. Without PARTITION BY region, the window is the whole result set ordered by month, so LAG crosses region boundaries and yields incorrect cross-region growth rather than erroring.
An XGBoost model's training logloss keeps decreasing across 2000 boosting rounds, but validation logloss bottomed at round 300 and has risen since. A teammate proposes lowering the learning rate from 0.1 to 0.01 while keeping n_estimators=2000 fixed and no early stopping. What is the most likely result?
- A. Lower learning rate alone eliminates overfitting because each tree contributes less, so validation loss will keep improving to round 2000
- B. It still overfits: a smaller learning rate needs MORE rounds to fit, but with 2000 rounds and no early stopping it will again overshoot the optimal round and overfit ✓
- C. Training loss will stop decreasing, proving the model can no longer fit the data
- D. The two hyperparameters are independent, so learning rate has no effect on where validation loss bottoms out
Correct answer: B. Learning rate and n_estimators trade off; shrinking the rate slows fitting but with a fixed large round count and no early stopping the ensemble still passes the optimal point and overfits, so early stopping on validation is the real fix.
A deep network trains stably at batch size 256 with Adam and lr=1e-3. After switching to SGD with the SAME lr=1e-3 and batch size 2048, loss diverges to NaN within a few steps. Which explanation is most consistent with this?
- A. Adam adapts per-parameter step sizes so 1e-3 was safe, but raw SGD applies that lr directly to large gradients, and the 8x larger batch also raises the effective step, causing divergence ✓
- B. SGD is fundamentally unable to train deep networks, so any lr will diverge
- C. Larger batch size always reduces the effective learning rate, so the divergence must come from a data pipeline bug, not the optimizer change
- D. Adam and SGD are interchangeable at the same lr; the NaN must be from float16 overflow unrelated to the optimizer swap
Correct answer: A. Adam normalizes updates by gradient magnitude so a given lr behaves very differently under plain SGD, and a larger batch gives lower-variance but not smaller-magnitude gradients, so the un-normalized step is far too large and diverges.
An offline-trained churn model scores AUC 0.88 on a held-out test set but only 0.71 in production two weeks after launch. Feature distributions (PSI) are stable, and no schema changed. The training set was built by joining a snapshot that included a 'days_since_last_login' feature computed as of the label date. What is the most probable root cause?
- A. Concept drift: customer behavior changed in two weeks, so retrain more frequently
- B. Covariate shift: the PSI test is too weak to detect the shift, so use the KS test instead
- C. Temporal/target leakage: the feature was computed using information from at or after the label window, so it is unavailable or different at true serving time, inflating offline AUC ✓
- D. Sample ratio mismatch between training and serving traffic caused the degradation
Correct answer: C. Stable input distributions plus a big offline-to-online drop and a feature computed 'as of the label date' point to temporal leakage where offline features encode future information unavailable at serving time.
A team runs an A/B test, checks the p-value daily, and stops the moment it drops below 0.05 on day 6 of a planned 14-day test, declaring a win. Assuming no true effect, what is the primary statistical problem?
- A. Nothing is wrong; reaching p<0.05 earlier just means the effect is strong enough to stop early
- B. Repeated peeking inflates the Type I error rate far above 5% because each look is another chance to cross the threshold by noise; a sequential/alpha-spending procedure is required to stop early validly ✓
- C. The test is underpowered, so the p-value is unreliable and they should have used a larger sample from the start
- D. Daily checks reduce power, making a false negative (Type II error) the main risk here
Correct answer: B. Optional stopping on a fixed-horizon p-value multiplies the chances of a false positive so the real Type I rate greatly exceeds 5%; valid early stopping needs sequential testing or alpha-spending.
In scaled dot-product attention, scores are computed as softmax(QK^T / sqrt(d_k)) V. Why is the division by sqrt(d_k) present, and what breaks without it as d_k grows large?
- A. It normalizes the output to unit norm; without it the output magnitude grows and the residual connection overflows
- B. For large d_k the dot products have large variance, pushing softmax into saturated regions with vanishing gradients; scaling by sqrt(d_k) keeps score variance ~1 so gradients stay usable ✓
- C. It converts logits to probabilities; without it softmax cannot be applied at all
- D. It makes attention permutation-invariant; without it positional information leaks into the scores
Correct answer: B. With random Q,K the dot product variance scales with d_k, so unscaled logits saturate softmax and kill gradients; dividing by sqrt(d_k) keeps the variance around 1 and preserves gradient flow.
You must add a feature answering internal-policy questions over a 40,000-page compliance manual that is revised weekly, with a hard requirement that every answer cite the exact source paragraph and that removed documents stop being answerable immediately. Which approach best fits, and why?
- A. Fine-tune the base model weekly on the manual, because fine-tuning gives the best factual grounding and lowest latency
- B. RAG over a vector index of the manual, because retrieval provides citable source chunks, weekly revisions only require re-indexing (not retraining), and deleting a document removes it from answers immediately ✓
- C. A long-context prompt stuffing all 40,000 pages every call, because it guarantees the model sees everything and needs no infrastructure
- D. Fine-tune with LoRA adapters daily, because adapters are cheap and encode citations directly into the weights
Correct answer: B. RAG naturally supplies verifiable source passages for citation, handles frequent updates by re-indexing rather than retraining, and honors deletions instantly by removing chunks from the index, which fine-tuning and full-context stuffing cannot do cleanly or affordably.
An analyst finds users who enabled two-factor auth churn 30% less and recommends forcing 2FA on everyone to cut churn. What is the strongest methodological objection?
- A. The sample is too small unless it exceeds 30 users per the Central Limit Theorem
- B. Self-selection/confounding: users who opt into 2FA are already more engaged and security-conscious, so the correlation likely reflects that confounder rather than a causal effect of 2FA on churn ✓
- C. The p-value was not reported, so the 30% figure is meaningless regardless of design
- D. Churn is a lagging metric, so any observational finding about it is automatically invalid
Correct answer: B. 2FA adoption is voluntary and correlates with engagement, a classic confounder, so the observed reduction cannot be read as causal without a randomized rollout or a valid quasi-experimental design.
You deploy LoRA fine-tuning: the base weights W are frozen and you learn a low-rank update so the effective weight is W + (B A) with A (r x d) and B (d x r), rank r=8. Which statement about this setup is correct?
- A. LoRA modifies the frozen W in place during training, so you must keep a full copy of the original weights to roll back
- B. Only A and B are trained, drastically cutting trainable parameters and optimizer state; at inference B A can be merged into W so there is no added latency ✓
- C. LoRA requires 4-bit quantization of the base model to function; without QLoRA the low-rank update is mathematically undefined
- D. The rank r must equal the hidden dimension d for the approximation to be lossless, so r=8 will always underfit badly
Correct answer: B. LoRA freezes W and trains only the small A and B matrices (huge memory savings on params and optimizer state), and since BA has the same shape as W it can be folded into the weights at inference for zero added latency.
How does XGBoost handle missing feature values during training by default?
- A. It drops rows containing any missing value
- B. It learns a default split direction for missing values at each node ✓
- C. It imputes with the global mean first
- D. It raises an error requiring pre-imputation
Correct answer: B. XGBoost assigns missing values to whichever branch minimizes loss, learned per split.
Dropout regularizes a neural network primarily by:
- A. Permanently deleting neurons after training
- B. Randomly deactivating units during training to prevent co-adaptation ✓
- C. Decaying the learning rate over time
- D. Normalizing activations per mini-batch
Correct answer: B. Random deactivation forces redundant, robust representations akin to model averaging.
The 'kernel trick' in SVMs allows the algorithm to:
- A. Train without any support vectors
- B. Compute inner products in a high-dimensional space without explicitly mapping the data ✓
- C. Guarantee a linear boundary in input space
- D. Eliminate the regularization parameter C
Correct answer: B. Kernels compute dot products in feature space implicitly, avoiding explicit transformation.
As dimensionality grows, a key manifestation of the curse of dimensionality is that:
- A. Distances between points become more distinct
- B. Points become nearly equidistant, weakening distance-based methods ✓
- C. Models always underfit
- D. Fewer samples are needed to generalize
Correct answer: B. In high dimensions pairwise distances concentrate, degrading nearest-neighbor notions.
During inference, batch normalization uses:
- A. The current mini-batch statistics
- B. Running moving-average estimates of mean and variance from training ✓
- C. Freshly computed per-sample statistics
- D. No normalization at all
Correct answer: B. At test time BN relies on population statistics accumulated during training for determinism.
In PCA, the principal components correspond to:
- A. The eigenvectors of the covariance matrix ordered by eigenvalue ✓
- B. The rows with the highest variance
- C. The features with the largest raw values
- D. Randomly rotated axes
Correct answer: A. PCA directions are covariance-matrix eigenvectors, ranked by explained variance.
Why might log loss be preferred over accuracy for a probabilistic classifier?
- A. It ignores predicted probabilities
- B. It penalizes confident wrong predictions, rewarding calibrated probabilities ✓
- C. It is invariant to class imbalance
- D. It only counts the argmax class
Correct answer: B. Log loss scores probability quality, heavily punishing confident mistakes.
A subtle form of data leakage occurs when:
- A. The test set is smaller than the training set
- B. Scaling/imputation parameters are fit on the full dataset before the train/test split ✓
- C. Categorical variables are one-hot encoded
- D. The random seed is fixed
Correct answer: B. Fitting preprocessing on all data leaks test-set information into training.
A 95% frequentist confidence interval means:
- A. There is a 95% probability the true parameter lies in this specific interval
- B. 95% of such intervals from repeated samples would contain the true parameter ✓
- C. The parameter is random with 95% density in this range
- D. The sample mean is 95% accurate
Correct answer: B. The 95% refers to the long-run coverage of the procedure, not one fixed interval.
Sigmoid activations worsen the vanishing gradient problem in deep networks because:
- A. Their derivative saturates near 0 for large-magnitude inputs, shrinking backpropagated gradients ✓
- B. They output unbounded values
- C. They are non-differentiable
- D. They amplify gradients exponentially
Correct answer: A. Saturated sigmoids have near-zero slope, so gradients decay multiplicatively across layers.
In XGBoost, what does the gamma (min_split_loss) parameter control?
- A. The learning rate
- B. The minimum loss reduction required to make a split ✓
- C. The number of boosting rounds
- D. The row subsample ratio
Correct answer: B. Gamma sets the minimum gain a split must achieve, acting as a complexity/regularization control.
Why can't standard k-fold cross-validation be used directly on time-series data?
- A. It leaks future information into training folds ✓
- B. It requires labeled data
- C. It cannot handle numeric features
- D. It always overfits
Correct answer: A. Random folds let future observations train a model evaluated on the past, leaking information; time-based splits are needed.
Which OLS assumption is violated when residuals are heteroscedastic?
- A. Linearity in parameters
- B. Constant error variance (homoscedasticity) ✓
- C. Independence of features
- D. Normality of predictors
Correct answer: B. Heteroscedasticity means error variance changes across observations, violating the constant-variance assumption.
In PCA, the principal components are the eigenvectors of which matrix?
- A. The covariance matrix of the data ✓
- B. The design matrix
- C. The confusion matrix
- D. The Gram matrix of the labels
Correct answer: A. PCA diagonalizes the feature covariance matrix; its eigenvectors are the principal directions.
What does the kernel trick in SVMs enable?
- A. Reducing the number of support vectors
- B. Computing inner products in a high-dimensional space without explicit mapping ✓
- C. Balancing class weights
- D. Preventing vanishing gradients
Correct answer: B. Kernels evaluate dot products in an implicit high-dimensional space, avoiding costly explicit feature maps.
Why does bagging in a random forest reduce variance?
- A. Averaging many decorrelated trees lowers overall variance ✓
- B. Each tree uses all features
- C. It increases individual tree depth
- D. It drives bias to zero
Correct answer: A. Averaging predictions from trees trained on different bootstrap samples and feature subsets cancels their independent errors.
A very deep, unpruned decision tree typically exhibits which bias-variance profile?
- A. High bias, low variance
- B. Low bias, high variance ✓
- C. High bias, high variance
- D. Low bias, low variance
Correct answer: B. Deep trees fit training data closely (low bias) but change a lot with new data (high variance).
Why does maximum-likelihood estimation for logistic regression lack a closed-form solution?
- A. The log-likelihood is nonlinear in the parameters and must be solved iteratively ✓
- B. The data is always non-separable
- C. The sigmoid is non-differentiable
- D. The Hessian is always singular
Correct answer: A. The sigmoid makes the likelihood equations nonlinear, so methods like Newton-Raphson or gradient ascent are required.
In a soft-margin SVM, what does a small value of the C parameter do?
- A. Produces a narrow margin with no allowed misclassification
- B. Produces a wider margin that tolerates more misclassifications ✓
- C. Sets the kernel bandwidth
- D. Sets the polynomial degree
Correct answer: B. Small C weakens the misclassification penalty, favoring a wider, more regularized margin.
In A/B testing, what is a Type II error?
- A. Detecting an effect that does not actually exist
- B. Failing to detect a real effect that does exist ✓
- C. Choosing the wrong test statistic
- D. Setting alpha to 0.05
Correct answer: B. A Type II error is a false negative, failing to reject a false null when a true effect is present.