Staff Data Engineer

Role summary The Staff Data Engineer designs, builds, and operates the data foundation and the evaluation infrastructure behind ServiceNow CRM Agentic AI. The role has two halves that reinforce each other. The first is conventional but demanding data engineering: architecture, pipelines, and transformation across structured and unstructured sources, held to production standards of reliability and quality. The second is newer and rarer: building the measurement layer that tells product and AI teams whether an AI agent actually did its job. That second half changes the nature of the work. Agent behavior is probabilistic, so quality cannot be asserted, only measured—against metrics that have to be defined before they can be tracked, and against ground truth that someone has to establish and defend. This engineer defines those metrics with product and AI teams, sets the labeling and validation standards, builds the datasets that reflect how agents behave in the real world, and automates the pipelines and dashboards that turn agent execution logs into a signal the organization can act on. the Staff Data Engineer owns these decisions across a domain rather than within a single project. They set design standards and quality gates others adopt, mentor junior engineers, and generalize evaluation patterns so each Agentic AI team is not rebuilding measurement from scratch. What you do Design and architect data infrastructure Design and oversee deployment of the data architecture and pipelines that capture, manage, and store structured and unstructured data from internal and external sources. Establish the processes and data flows across cloud services, local databases, and other applicable storage forms, and own the contracts between data producers and consumers. Build and automate data transformation Develop technical tools using machine learning and data-engineering techniques to cleanse, organize, and transform data. Implement automated processes that maintain the integrity of data structures and hold quality standards on an ongoing basis rather than at a point in time. Define agentic evaluation metrics and ground truth Partner with product and AI teams to define evaluation metrics for agentic workflows, including task and mission completeness, instruction adherence, tool use, and end-to-end workflow success. Establish ground truth labeling standards, annotation guidelines, and validation criteria, and design evaluation datasets that reflect real-world agent execution rather than idealized paths. Build agentic evaluation pipelines Design and implement automated evaluation infrastructure that measures AI agent performance using LLMs and agent execution logs. Create the dashboards, reporting, versioning, and reproducibility that make evaluation datasets and results trustworthy over time and comparable across releases. Establish standards and continuous improvement Create design standards and quality assurance processes for data systems. Define quality gates and validation frameworks, analyze workflow performance, and recommend optimizations that keep the platform aligned with evolving CRM AI requirements. Lead cross-functional collaboration Collaborate with product, engineering, and data science teams. Mentor junior engineers on data engineering and evaluation design, and generalize evaluation patterns and metrics so they can be reused across Agentic AI products rather than rebuilt per team.

Required qualifications Data engineering depth . 8+ years in data engineering, with a record of owning production data platforms end-to-end. Depth and demonstrated judgment matter more than tenure. Domain ownership. Demonstrated ownership of a data domain or platform, including architecture decisions, migrations, and the operational consequences of both. Evaluation infrastructure experience . Hands-on experience building measurement or evaluation infrastructure for machine learning or AI systems: evaluation pipelines, benchmark harnesses, quality d