LLM - Engineering Expert

Overview

ExpertWoka is sourcing experienced AI Evaluation Engineers with an engineering simulation and design background to help create and validate challenging, simulation-based tasks for AI agents.

This is a short-term contractor engagement focused on engineering problem design, simulation workflows, trajectory analysis, and benchmark refinement.

Responsibilities

• Author original, self-contained engineering design tasks with competing constraints and objective grading criteria.
• Build, run, and validate problem environments using open-source simulation tools and Python test benches.
• Analyze agent outputs and execution logs to identify systematic failure modes.
• Refine task difficulty based on empirical performance data while avoiding ambiguity.
• Collaborate with researchers and domain experts to integrate benchmarks into the evaluation pipeline.

Requirements

• Master’s degree or PhD in Electrical, Mechanical, or Aerospace engineering.
• 10+ years of hands-on engineering design experience.
• Proficiency with at least one domain-relevant open-source simulation package.
• Strong Python scripting skills.
• Hands-on experience with modern LLMs or coding agents and evaluation concepts such as pass@k, failure-mode analysis, and nondeterministic behavior.
• Ability to audit trajectory logs and isolate reasoning or tool-use failures.
• Strong attention to physical plausibility, unit consistency, boundary conditions, convergence criteria, and technical documentation.
• Weekend on-call availability.
• 40 hours per week with 4 hours of overlap with PST.
• Personal desktop or laptop with stable, high-speed internet connection.

Project Details

• Type: Contractor
• Engagement length: up to 24 weeks
• Start: Immediately
• Compensation: $500 per task

Evaluation Process

• Shortlisted candidates will complete a Job Interest Form.
• Finalized talents will proceed to delivery review.