Quality Analyst II

About us
HighLevel is an AI-powered business operating system that gives agencies, entrepreneurs and SMBs the infrastructure to build, automate and scale. Today, HighLevel supports SMBs across 150+ countries, fueling community-driven growth rooted in real customer outcomes.
To date, businesses operating on HighLevel have generated over $7 billion in ecosystem value, demonstrating the impact of shared infrastructure at scale. By centralizing conversations, automation and intelligence into one system, we help businesses move faster, reduce complexity and execute efficiently.
Behind the platform, HighLevel powers more than 4 billion API hits and 2.5 billion message events daily. With 250 terabytes of distributed data, 250+ microservices and over 1 million domain names supported, our architecture is built for performance, resilience and long-term scalability.
Our people
With over 2,000 team members across 10+ countries, HighLevel operates as a global, remote-first organization built for speed and ownership. We value initiative, clarity and execution, creating space for ambitious people to build systems that support millions of businesses worldwide. Here, innovation thrives, ideas are celebrated and people come first, no matter where they call home.
Our impact
Every month, HighLevel enables more than 1.5 billion messages, 200 million leads and 20 million conversations for the more than 1 million businesses we support. Behind those numbers are real people building independence, expanding opportunity and creating measurable impact. We’re proud to be a part of that.
Learn more about us on our YouTube Channel or Blog Posts 
Role Overview
We are looking for a Quality Analyst II to join Customer Success — Programs & AI Innovation as an evaluator-first specialist focused on the quality of AI-powered workflows that support our advisors before, during and after every customer call.
You will spend most of your time evaluating AI output across ORA, our internal AI platform — reviewing pre-call briefs, in-call guidance, post-call grading and transcription-driven customer follow-ups. You will classify failure modes, document patterns and feed structured findings into the team that builds inside ORA. You will also do light hands-on work inside ORA — refining prompts, updating knowledge base entries and tweaking existing skills — closing the loop from “found an issue” to “shipped a fix.”
This is a role with a deliberate growth path. You start in an evaluator-first lane with light creation work. As you build judgment, exceed expectations and demonstrate strong ownership, your scope expands toward deeper creation work alongside the Lead Analyst on the team.
Key Responsibilities
1. Evaluate AI Quality Across the Advisor Stack
You are the quality owner for advisor-facing AI inside ORA. You review AI output across the full call lifecycle and surface what is working, what is broken and where the bar needs to be raised.
Review pre-call briefs for accuracy, completeness and signal quality — customer context, adoption signals, retention flags
Evaluate in-call guidance — next-best-action prompts, suggestions and recommendation flows — for relevance and accuracy in live conversation
Score post-call outputs: quality grading, AI-generated summaries, CRM updates, sentiment signals
Audit transcription-driven follow-ups — customer updates, recap notes, follow-up tasks — for accuracy, tone and customer impact
Classify issues by type and severity — hallucination, missing context, wrong action, tone mismatch, KB gap — using standardized rubrics
Participate in calibration sessions to keep scoring consistent across reviewers
2. Iterate Inside ORA — Light Creation Work
You close the loop. When you spot an issue you can fix without engineering involvement, you fix it inside ORA.
Refine and iterate on prompts within existing skills and sub-agents
Update knowledge base entries to close information gaps surfaced in reviews
Tweak existing skills and orchestrations to impro