AI Benchmark & Red-Teaming Specialist

Overview

ExpertWoka is sourcing candidates for a contract Task Author role focused on building high-quality web research benchmark tasks and evaluating frontier AI model behavior.

What You'll Do

• Research obscure facts and build precise, evidence-based benchmark tasks.
• Verify answers using primary or highly credible sources.
• Test tasks against AI models and review where failures occur.
• Analyze search trajectories, reasoning paths, and source selection.
• Improve and scale task patterns across new domains.

Requirements

• 3+ years of experience.
• Strong investigative research, fact-checking, red-teaming, or technical QA experience.
• Intermediate-to-advanced web search and OSINT skills.
• Ability to write clear, unambiguous benchmark prompts with reliable answer keys.
• Experience with benchmark or evaluation tasks such as GAIA, BrowseComp, SimpleQA, SWE-bench, Terminal-Bench, or Humanity’s Last Exam.
• Bachelor’s degree or higher in a relevant engineering domain.

Responsibilities

• Discover hard-to-surface facts through deep research.
• Verify facts against primary sources and document source URLs.
• Draft tasks designed to expose weaknesses in model research strategies.
• Test tasks against SOTA models and diagnose failure mechanisms.
• Review peer-authored tasks when needed.

Schedule

This is a contractor assignment with full-time or part-time commitment options. The role requires 40 hours per week or 20 hours per week, with 4 hours of PST overlap.

Project Details

Engagement length is 4 weeks. Start date is October 20.

Evaluation Process

• ICF
• Assessment
• Profile review