Queue Lead - DP

Overview

Prolific Academic Ltd is seeking a Queue Lead for a remote freelance role on a high-priority initiative with a leading AI company. This is a hands-on operational position focused on running one or more annotation queues for a live AI training and evaluation program. The work centers on queue execution, quality calibration, contributor performance management, and stakeholder reporting.

The role is intended for someone with direct experience running queues and quality on AI data programs, and who can speak concretely about past projects, calibration workflows, and contributor performance systems they have built or managed.

What You'll Do

• Own queue operations, including intake, triage, prioritization, task assignment, and SLA delivery.
• Monitor task queues and backlogs, and identify bottlenecks such as unclear policy, tooling problems, or supply gaps.
• Track quality at both the project level and the individual contributor or annotator level.
• Carry out QA work by making corrections, giving annotator feedback, surfacing systematic confusion, and escalating issues to the team lead when needed.
• Manage contributor performance, including unassigning underperformers and retaining strong performers.
• Build and maintain dashboards that track throughput, disagreement rates, defect or rework rates, and contributor performance.
• Translate raw queue data into clear reporting that stakeholders can act on quickly.
• Use data trends to recommend process improvements.
• Support participants through dedicated Slack channels by answering questions, escalating complex issues to the Client Delivery Lead, and keeping communication flowing.
• Train new annotators or contributors on project guidelines.
• Run live or recorded webinars for new participants and host office hours when needed.

Requirements

• Proven, direct experience running queues and quality on AI data programs, ideally in a competitor environment.
• 3+ years in high-throughput operations such as data labeling, trust & safety, BPO, content operations, or CX operations, with recent and specific AI data ops experience.
• Ability to identify shallow, low-effort, AI-generated, or phoned-in responses.
• Demonstrated ability to design and run calibration processes from scratch.
• Experience building or maintaining spreadsheets and reporting systems; SQL, BI, Looker, or Tableau is a strong plus.
• Experience managing contributor performance at scale, including retention and removal decisions.
• Comfort working with ambiguity and fast-changing priorities on live programs.