Software Engineer – LLM Evaluation & Repository Validation

Overview

Rex is looking for software engineers to help validate real-world repositories used to evaluate large language models. The work focuses on triaging issues, preparing reproducible environments, assessing test quality, and checking how effectively models fix real bugs. This opportunity is available in all countries.

What You'll Do

• Review and triage issues in open-source GitHub projects.
• Prepare repositories for evaluation, including containerisation and environment automation.
• Assess how thorough and well-written unit tests are.
• Edit and run code locally to observe how LLMs perform on bug-fixing tasks.
• Help select repositories and issues that are challenging for LLMs.
• Document findings clearly for researchers and reviewers.
• Follow project guidelines, participate in calibration and review sessions, and incorporate feedback to keep quality consistent.

Requirements

• 3+ years of software engineering experience.
• Strong experience in at least one of Python, C++, C#, Go, Rust, or Ruby.
• Proficiency with Git, Docker, and basic pipeline setup.
• Ability to navigate complex codebases and run real-world projects locally.
• Fluent written English.
• Reliable, detail-oriented, able to work independently, and able to meet deadlines in a remote setting.

Preferred Qualifications

• Prior LLM research or evaluation experience.
• Experience building or testing developer tools or automation agents.
• Open-source contribution or evaluation experience.

Rights, Confidentiality and Data Handling

• Submit only work you created yourself or that you have the right to share.
• Do not submit material covered by employer, client, or third-party confidentiality agreements, or material you do not have the right to share.
• Do not submit export-controlled, military, or defence-related technical data.
• Remove credentials, secrets, and personal data from anything you submit.
• Rex may ask for evidence of ownership or licence, may verify submissions, and may reject or remove any submission.

Legal and Compliance Notice

• All content you submit must be lawful, and you must have the right to share it.
• Do not submit confidential or proprietary material belonging to an employer, client, or third party, or personal data without a lawful basis.
• Rex may verify identity, credentials, ownership, and licensing; may reject or remove any submission; and may report unlawful submissions to the relevant authorities.
• Engagement and payment are subject to successful verification and to applicable sanctions, export-control, tax, and employment laws.
• The contractor agreement will set out ownership and licence terms for the work you deliver, to be confirmed by Rex Legal.