LLM - Engineering Expert

Overview

ExpertWoka is sourcing experienced AI Evaluation Engineers with strong engineering simulation and design expertise to help create and validate challenging, simulation-based tasks for AI agents.

What You'll Do

• Design original engineering tasks with competing constraints and clear optimization targets.
• Build and validate simulation-based problem environments using open-source tools and Python test benches.
• Analyze agent outputs and execution logs to identify recurring reasoning and tool-use failures.
• Refine benchmark difficulty based on model performance data while preserving clarity and objectivity.
• Collaborate with researchers and domain experts on high-rigor evaluation workflows.

Requirements

• Master's degree or PhD in Electrical, Mechanical, or Aerospace Engineering.
• 10+ years of hands-on engineering design experience.
• Proficiency with at least one relevant open-source simulation package.
• Strong Python scripting skills.
• Hands-on experience with modern LLMs or coding agents and evaluation concepts.
• Ability to audit trajectory logs and isolate core reasoning or tool-use failures.
• Strong attention to physical plausibility, unit consistency, boundary conditions, convergence criteria, and technical documentation.
• 40 hours per week availability.
• Weekend on-call availability.
• 4 hours of overlap with PST.
• Remote work requires a personal desktop or laptop and stable high-speed internet.

Project Details

• Short-term contractor engagement.
• Estimated duration: 12 weeks.
• Start: immediately.
• Total openings: 100.

Evaluation Process

Shortlisted candidates will complete a job interest form and then undergo a delivery review before proceeding further.