Skip to content
ExpertWoka
Back to opportunities
Mercor

AI Evaluation Rubric Reviewer

Posted 2026-11-16
View application options
Work Type
Remote
Location
Global
Compensation
$40-$75 per hour

About this opportunity

Overview

Mercor is hiring an AI Evaluation Rubric Reviewer to assess AI-generated tasks and grading rubrics for realistic shopping and delivery scenarios. In this role, you will review model attempts, judge whether failures are fairly graded, and help improve rubric quality and task validity.

This is a remote hourly contract role. The work is performed independently on your own schedule, and the engagement may be extended, shortened, or ended early depending on project needs and performance.

What You'll Do

  • Review AI-drafted tasks, including the prompt and grading rubric, alongside model attempts.
  • Decide whether a failure is fair based on what information was available.
  • Check whether the rubric accurately grades what the prompt asked for.
  • Determine whether a valid answer would have been marked wrong.
  • Assess whether the rubric structure follows the provided guidelines.
  • Fix criteria that are not fair when possible.
  • Flag tasks that cannot be salvaged and write a short note explaining each call.

Requirements

  • Substantial experience writing or reviewing rubrics for LLM evaluation work.
  • Ability to read long agent transcripts and identify where the agent went wrong.
  • Precise writing and a careful habit of checking the data instead of relying on the reference answer.
  • Applicants should list relevant projects in their application.

Contract Terms

  • You will be engaged as an independent contractor.
  • The role is fully remote.
  • Work can be completed on your own schedule.
  • Projects may be extended, shortened, or concluded early based on needs and performance.
  • Work for Mercor will not involve access to confidential or proprietary information from any employer, client, or institution.
  • Payments are made weekly on Stripe or Wise based on services rendered.
  • H1-B and STEM OPT candidates are not supported for this role.

About the Role

The project focuses on testing AI agents on shopping and delivery tasks, with an emphasis on reviewing evaluation rubrics and ensuring that grading is fair, accurate, and aligned with the task instructions.

Skills & expertise

LLM Evaluation Prompt Evaluation Model Evaluation Rubric Review Transcript Analysis Technical Writing Data Verification

Related Opportunities

Trending Right Now

View all

Don't miss the next opportunity

Get a curated Friday digest of new AI and remote roles based on your location and interests — free, no spam.

Get Weekly Opportunities