Side by Side Quality Evaluator

Overview

We’re looking for English-fluent experts to take part in a project focused on evaluating the overall quality of AI-generated responses. As an independent contractor, you will compare responses from large language models side by side and judge which response better meets the user’s needs. Your written rationales and ratings will help improve how helpful, accurate, and reliable AI systems are.

What You'll Do

• Interact directly with different AI models and tools.
• Compare two or more model responses side by side.
• Evaluate responses on dimensions such as accuracy, helpfulness, instruction-following, clarity, and overall quality.
• Follow multi-step evaluation workflows and rating guidelines.
• Write clear explanations for your evaluation decisions.
• Maintain consistent judgments across a wide range of prompts and topics.

Requirements

• Fluency in English, with strong reading comprehension and writing skills.
• Hands-on experience using LLMs and evaluating their outputs.
• Strong critical-thinking skills for comparing responses across multiple quality dimensions.
• Careful attention to detail and ability to follow instructions closely.
• Ability to explain evaluation decisions clearly and convincingly.
• Comfort with multi-step workflows and switching between models and tools.

Compensation

Fees vary based on service and complexity.