Quality & Evaluation Manager – Generative AI
Job Title: Quality & Evaluation Manager – Generative AI
Location: Remote/Hybrid (Flexible)
Position Type: Full-time
iMerit is a leading data services company powering the next generation of artificial intelligence. Our platform enables AI development teams to accelerate model training, deployment, and refinement through high-quality, scalable, and secure data pipelines. We partner with organizations across industries to ensure their AI initiatives are grounded in robust, ethical, and performant data solutions.
We are looking for an experienced Quality & Evaluation Manager to take ownership of the quality management function across our portfolio of Generative AI projects.
This role is responsible for ensuring evaluation integrity, reviewer consistency, scalable audit operations, and high-quality model training data across complex GenAI workflows.
The ideal candidate combines operational rigor, analytical thinking, strong judgment, and the ability to navigate ambiguity in rapidly evolving AI environments.
You will be the key quality leader: managing audit teams, resolving complex edge cases, designing standardization processes, and acting as the primary quality liaison with clients.
What you’ll do
• Quality Leadership & Standardization:
%CB; Lead, mentor, and align reviewer, auditor, and evaluation teams across to ensure strict adherence to quality standards and client requirements.
%CB; Design and oversee robust quality audit frameworks, tracking key metrics like accuracy, redo percentage, coverage, and turnaround time.
%CB; Use quality analytics, audit insights, and operational telemetry to identify trends, predict quality risks, and drive targeted improvement initiatives.
%CB; Serve as the final escalation point for complex/ambiguous cases, creating comprehensive documentation to standardize decision-making across team members.
• Client & Stakeholder Alignment:
%CB; Act as the primary Quality Point-of-Contact (POC) for client stakeholders, delivering regular performance reports, actionable insights, and strategic quality plans.
%CB; Coordinate and lead both internal and client-facing calibration sessions to ensure absolute consistency in data interpretation and output quality.
%CB; Manage constantly evolving guidelines and requirements, keeping up to date with changes and communicating them to stakeholders.
• Evaluation Governance & Quality Intelligence
%CB; Define and continuously refine evaluation rubrics, calibration standards, and audit methodologies for GenAI workflows.
%CB; Monitor evaluator consistency through inter-rater agreement analysis and targeted calibration interventions
%CB; Identify systemic quality risks, ambiguous guideline interpretations, and edge-case trends to improve evaluation integrity.
%CB; Partner with DataOps, Training, and Product teams to improve measurable quality outcomes at scale.
• Process Improvement:
%CB; Drive continuous improvement initiatives focused on increasing overall quality, minimizing errors and enabling processes to scale effectively.
%CB; Collaborate cross-functionally with Operations, Training, and Product/Tech teams to proactively resolve recurring quality challenges and implement systemic solutions.
• Communication
%CB; Communicate complicated, technical concepts to non-technical audiences.
%CB; Coach evaluators and reviewers to improve reasoning quality, guideline interpretation, consistency, and decision-making accuracy.
What you’ll need
• Experience: 5–8 years in Quality Operations, QA, or Quality Leadership roles, including at least 2–3 years in AI/ML, Generative AI, data annotation, model evaluation, or related human-in-the-loop workflows.
• Domain Knowledge: Strong, practical knowledge of QA processes, including audits, calibrations, feedback, and reporting.
• AI Familiarity: Familiarity with the use cases and quality challenges specific to Generative AI (e.g., reasoning, classification, summarization, red-teaming, multimodal tasks).
• Skills: Strong analytical and data interpretation skills to drive quality decisions, and the ability to clearly communicate decisions and recommendations with stakeholders and clients.
• Work Environment: Proven success working remotely or in hybrid team environments.
• Platform/Tooling Exposure: Familiarity with annotation/evaluation platforms, quality dashboards, workflow tooling, and data-driven operational reporting.
• Ambiguity Navigation: Ability to operate effectively in fast-changing, ambiguity-heavy environments with evolving quality standards and workflows.
Preferred Experience:
• Bachelor’s or Master’s degree in a relevant field (Linguistics, Social Sciences, Philosophy, or related discipline).
• Proven track record of successfully leading quality operations on large-scale, global AI data annotation projects.
• Demonstrable experience in client relationship management and effective team mentorship.
What We Offer:
• A flexible hybrid working model tailored for optimal work-life balance.
• Competitive compensation and a comprehensive benefits package.
• A collaborative, innovative, and inclusive environment.
• Continuous professional growth and learning opportunities.
Join us to lead groundbreaking projects in Generative AI, making a tangible impact in the evolving field of artificial intelligence.
