Reviewer and rate AI-generated code - Python

Overview

We are seeking experienced software engineers to review and rate AI-generated Python code in a short-term contract engagement. This role involves hands-on evaluation work across real-world codebases, including repository setup, issue triage, testing, and assessing code quality.

What You'll Do

• Analyze and triage GitHub issues across open-source libraries.
• Set up and configure code repositories, including Dockerization and environment setup.
• Evaluate unit test coverage and code quality.
• Modify and run codebases locally to assess bug-fixing performance.
• Collaborate with researchers to identify repositories and issues that are challenging for AI systems.

Requirements

• At least 2 years of overall experience.
• Strong experience with at least one programming language, especially Python.
• Proficiency with Git, Docker, and basic software pipeline setup.
• Ability to understand and navigate complex codebases.
• Comfort running, modifying, and testing real-world projects locally.

Preferred Qualifications

• Experience contributing to or evaluating open-source projects.
• Previous participation in LLM research or evaluation projects.
• Experience building or testing developer tools or automation agents.

Schedule

• Short-term contract.
• Minimum 40 hours per week.
• 4 hours of overlap with PST required.
• Immediate start.
• Expected duration: 2-3 weeks.

Project Details

• Payment: $25 per approved task.
• Role type: Contractor assignment.

Evaluation Process

• Interest Check Form (ICF)
• Assessment
• Delivery Review