About this opportunity
Overview
Welo Data is seeking an experienced Lead Prompt Engineer for a remote, part-time role in the United States. The selected candidate will guide a team through a technical migration from templates to LLM autoraters, using advanced prompt engineering methods and internal tools to improve model performance and support the ongoing development of AI systems.
What You'll Do
- Use Automatic Prompt Generation (APG) tools to create baseline prompts for complex parent-child template clusters.
- Run and supervise Automated Prompt Optimization (APO), review results, and identify when optimization reaches deadlocks or plateaus.
- Manually draft, test, and refine prompts for complex template structures, anti-patterns, and edge cases when tooling is limited or broken.
- Monitor shadowbot runs to ensure disagreements between human and LLM ratings are generated and tracked.
- Evaluate prompt versions against gold data to measure autorater quality against the human crowd baseline.
- Calculate quality metrics such as F1 score, precision, and recall.
- Prepare technical launch readiness justifications and launch certification documentation.
Requirements
- Native fluency in English.
- Must be based in the United States.
- Master’s or Doctorate degree in Computer Science, Data Science, Computational Linguistics, Human-Computer Interaction, Cognitive Science, or a related analytical field.
- At least 7 years of experience as a Prompt Engineer.
- Proven experience tuning LLMs for strict structured outputs and complex classification tasks.
- Familiarity with chain-of-thought and few-shot learning approaches.
- Strong ability to identify error patterns, analyze model performance, and use SQL or other data analytics tools.
- Ability to learn proprietary tools quickly and work with minimal supervision.
- Excellent verbal and written communication skills.
Preferred Qualifications
- Familiarity with enterprise-grade LLM interfaces such as the Goose API.
- Experience in AI model evaluation, data science, computational linguistics, or software engineering.
- Hands-on experience with Automated Prompt Optimization systems or tuning workflows.
- Linguistic expertise, including an understanding of semantics and logic.
Compliance
All hires must verify identity and eligibility to work in the United States and complete employment eligibility verification. New employees must complete live video verification with selected IDs and provide photos of those IDs within their first 3 days of employment.
The employer also notes that AI tools may be used to support parts of the hiring process, including application review, resume analysis, and response assessment, with final hiring decisions made by humans.
