AI Analyst

Job Summary

The ERDM Sr. AI Operations Analyst is responsible for ensuring that Sales & Finance AI solutions operate with the highest levels of trust, accuracy, control, and compliance.

We are seeking a detail-obsessed, data driven expert to serve as theguardian of accuracyfor our Sales & Finance AI chat assistants. You will design validation frameworks, partner with data engineering teams, and ensure every metric the AI surfaces - especially Bookings and Revenue - is accurate, consistent, and trustworthy. You will lead incident response on any data accuracy or hallucination even

The role also serves as the bridge between Finance users and technical development teams. This role is responsible for translating business requirements into AI-enabled solutions and ensuring that Finance AI capabilities align with business goals and leadership needs.

Core Responsibilities:

Requirements & Solution Definition

• Gather and documentbusiness requirementsfrom Finance users and stakeholders through interviews, workshops, observation, and feedback analysis
• Write clear, detaileduser stories with well-defined acceptance criteriathat IT can act on without ambiguity
• Ensure requirements reflectmetric integrity standards and executive-readiness expectations

AI Quality Assurance

• Develop and maintainAI-specific testing frameworks- define test scenarios, expected outputs, edge cases, and pass/fail criteria tailored to conversational AI behavior
• Establish and documentquality standardsfor AI-generated responses - accuracy, completeness, formatting, tone, and contextual relevance
• Validate AI-generated responsesagainst Snowflake source data using SQL queries to confirm metric accuracy and consistency
• Conduct thoroughUAT, regression testing, and release testingfor every new feature, enhancement, prompt update, data refresh, or model change before go-live
• Confirm metric integrity- verify that Bookings and Revenue figures surfaced by the AI match official Finance reporting
• Coordinatebusiness sign-off activities- manage the approval workflow with Finance stakeholders before releases are promoted to production

Hallucination Prevention

• Design and executevalidation test strategiesspecifically targeting hallucination risks - test for fabricated data, incorrect metric associations, misleading trends, and unsupported narratives
• Identify and cataloghigh-risk Finance use caseswhere hallucination potential is elevated (e.g., multi-step calculations, year-over-year comparisons, segment-level breakdowns, forecast vs. actuals)
• Monitor response qualityon an ongoing basis - conduct periodic spot-checks and audits of AI outputs across different query types and user scenarios
• Escalate and resolve quality concernsimmediately - flag suspected hallucinations to the Data Accuracy Specialist (IC4) and team lead coordinate with IT/Engineering to implement fixes and guardrails

Core Responsibilities

Executive Use Cases

• Support indesigning and testing executive-focused use cases- ensure AI responses meet the precision, tone, and contextual depth expected by C-suite and SVP users
• Test and validateexecutive prompt frameworks- verify that curated queries, guided flows, and personalized dashboards produce accurate, polished outputs
• Contribute to improvingusability and decision-support capabilities- identify opportunities to make the executive experience more intuitive, actionable, and insightful

Adoption & Change Management

• Develop and executeadoption strategies- create structured plans to drive Finance user engagement, from initial awareness through active daily use
• Leaduser onboarding- design and deliver onboarding experiences for new users, including guided walkthroughs, sandbox environments, and office hours
• Delivertraining programs- conduct live demos, workshops, and drop-in sessions tailored to different user groups (analysts, managers, directors)
• Manage communications - draft and distribute release notes, feature announcements, tips & tricks, known issue updates, and adoption progress reports to Finance stakeholders
• Collect and synthesize user feedback through surveys, interviews, and usage analytics to continuously improve the experience

Required Qualifications:

• Bachelor's degree Required
• 7+ years in Business Analysis / Finance Analytics / FP&A Operations
• Experience withAI-powered tools, chatbots, or conversational AI interfaces
• Understanding ofFinance metrics and reporting(Bookings, Revenue, or related areas)
• Working knowledge ofSQL and Snowflake
• Understanding of AI/LLM fundamentals - how large language models retrieve, process, and present information awareness of common failure modes (hallucination, data staleness, context window limitations)
• Prompt Engineering Certificate or related experience

Preferred Qualifications:

• Familiarity withAbacusor similar platforms
• Hands-on experience withprompt engineering- designing, testing, and refining prompts to improve AI response quality and accuracy
• Experience creatingtraining materials, user guides, or knowledge base content
• Experience with AI testing frameworks or quality assurance for data-driven applications
• Familiarity with AI governance or responsible AI principles