Your qualitative interviews (n=12) strongly suggest users are frustrated by a new checkout flow, but the A/B test on 50,000 users shows the new flow has a statistically significant higher conversion rate. Leadership wants to ship. What is the most defensible interpretation?
- A. The A/B test is the larger sample, so the qual data is simply noise and should be discarded
- B. Both findings can be true: conversion measures a behavioral outcome while interviews reveal experience quality and potential long-term costs, so investigate what the qual signals (e.g., post-purchase regret, support load) ✓
- C. The interviews prove the A/B test had a confound and the conversion lift is an artifact
- D. You should re-run the interviews with a larger sample to see if they match the A/B result before acting
Correct answer: B. Behavioral conversion and reported experience answer different questions, so contradictory-seeming results should be reconciled by probing what each measures rather than dismissing one.
A PM ran a survey and reports '78% of users want feature X, so we should build it.' The key question read: 'How much would you love having the time-saving feature X?' with a 5-point scale from 'Like it' to 'Love it.' What is the primary methodological flaw you should raise?
- A. The sample size was likely too small to reach significance
- B. The scale is unbalanced and the wording is leading, so the 78% reflects the instrument's bias rather than genuine demand ✓
- C. Surveys can never measure feature demand and should be replaced with interviews
- D. The 5-point scale should have been a 7-point scale for more granularity
Correct answer: B. A scale with no negative options plus loaded phrasing manufactures agreement, invalidating the result regardless of sample size.
You have two weeks and a small budget to inform a major redesign decision that will be locked for the next year. A colleague argues for a quick unmoderated usability test (n=8); another pushes for a rigorous longitudinal diary study. Which reasoning best guides the tradeoff?
- A. Always choose the more rigorous method when a decision is high-stakes and long-lived
- B. Match method to the decision's dominant risk: if the risk is usability/comprehension, the fast evaluative test is defensible; if the risk is unknown real-world behavior over time, the timeline can't support rigor and you should renegotiate scope or narrow the decision ✓
- C. The diary study is superior because it captures behavior in context, so budget should be found
- D. Run both simultaneously to triangulate within the two weeks
Correct answer: B. Method choice should follow the specific risk the decision carries, and when constraints can't cover that risk the right move is to renegotiate scope rather than run an ill-fitting study.
An experiment testing a new onboarding flow shows a p-value of 0.03 for a 0.2% increase in day-7 retention on 200,000 users. The team is celebrating. What is the most important thing to flag?
- A. p=0.03 is not below the standard 0.01 threshold, so the result is not significant
- B. With a very large sample, a trivially small effect can be statistically significant; the team must assess whether a 0.2% lift is practically meaningful against implementation cost ✓
- C. A p-value of 0.03 means there is a 3% chance the null hypothesis is true
- D. The result is invalid because retention should be measured at day 30, not day 7
Correct answer: B. Large samples make tiny effects statistically significant, so the real question is whether the effect size matters practically, not whether p cleared a threshold.
You want to compare task-completion satisfaction across 5 user segments to prioritize which segment to fix first. You size the total sample for the overall study but not per segment. What is the likely consequence?
- A. The overall sample size guarantees each segment estimate is equally precise
- B. Per-segment estimates will have wide, overlapping confidence intervals, so segment-level comparisons may be underpowered and unreliable even though the overall study looks well-powered ✓
- C. Segmenting after the fact is p-hacking and is never permissible
- D. You should pool all segments since segment differences won't affect the aggregate
Correct answer: B. Precision depends on the cell size actually being analyzed, so segmented comparisons need per-segment power even when the total sample is large.
A senior VP has already publicly committed to a strategy, and your research produces strong evidence it will harm the target users. What is the most effective senior-researcher move?
- A. Withhold the findings to preserve your relationship with the VP, and revisit after launch
- B. Frame the findings around the VP's own goals and the business risk, present the evidence with severity and confidence levels, and offer a path that partially preserves the commitment while mitigating the harm ✓
- C. Escalate above the VP immediately with the raw data to force a reversal
- D. Soften the findings so they don't contradict the decision, since the decision is already made
Correct answer: B. Influencing a committed stakeholder works best by connecting evidence to their goals and offering a viable mitigation path, not by hiding, diluting, or bypassing.
Leadership says research is 'too slow' and wants to stop funding a dedicated function in favor of PMs running their own studies. Beyond defending your value, what is the strongest structural response?
- A. Insist all research must go through trained researchers to protect quality
- B. Propose a tiered ResearchOps model: enable PMs with vetted templates, guardrails, and a repository for low-risk questions, while reserving dedicated research for high-risk, high-ambiguity decisions ✓
- C. Agree to speed up by cutting sample sizes across all studies
- D. Produce a report quantifying hours spent to justify the current headcount
Correct answer: B. The durable answer to speed-and-cost pressure is a tiered operating model that democratizes low-risk work with guardrails while protecting rigor where stakes are high.
A PM shares a study concluding 'users prefer blue buttons' based on 5 users clicking faster on blue in one session. The PM is proud. How should you salvage this without alienating them?
- A. Tell the team the study is worthless and redo it yourself
- B. Acknowledge the initiative, then coach: reframe the finding as a hypothesis, note confounds (order effects, tiny n, speed ≠ preference), and co-design a cleaner follow-up if the question matters ✓
- C. Publish the finding as-is to encourage more PM research
- D. Quietly correct the repository entry without telling the PM
Correct answer: B. Salvaging democratized research means validating the effort while coaching on rigor and reframing overreach as a testable hypothesis, preserving the person's willingness to keep contributing.
You're researching a novel AI feature where the core question is whether users appropriately trust the model's outputs. Established usability heuristics don't cover this. What is the soundest approach?
- A. Apply standard SUS scoring since it's the industry benchmark for usability
- B. Design for trust calibration specifically: measure whether user reliance tracks actual model accuracy (over- and under-reliance), using tasks with known-correct and known-wrong AI outputs ✓
- C. Ask users directly how much they trust the AI on a 1-10 scale and report the average
- D. Wait until best practices for AI UX are established before researching
Correct answer: B. Trust in AI is about calibration—reliance matching real accuracy—so the study must expose users to correct and incorrect outputs and measure appropriate reliance, not self-reported trust alone.
Your team keeps re-running similar foundational studies because past insights get lost. You're building a research repository. Which design choice most directly prevents repeat studies?
- A. Store final report PDFs organized by the requesting team
- B. Structure findings as tagged, searchable atomic insights linked to evidence, so future questions can be answered by querying existing knowledge before commissioning new work ✓
- C. Restrict repository access to researchers to maintain quality control
- D. Archive raw session recordings so anyone can re-analyze them later
Correct answer: B. Atomic, tagged, evidence-linked insights make institutional knowledge queryable at the question level, which is what actually intercepts redundant studies before they start.