AI Voice Evaluation Specialist - 3 month contract
About this opportunity
Overview
Innodata is hiring a detail-oriented Voice Specialist for a 3-month, full-time contract (40 hours per week) to evaluate AI model performance through real-time, voice-based conversations. The work involves comparing two models on the same scenario, assessing the quality of their responses, and documenting a structured rationale for the final preference.
This position is remote for candidates located in Canada.
What You’ll Do
- Review an assigned scenario and roleplay a natural conversation with two different AI models.
- Conduct comparable conversations with Model A and Model B while keeping the scenario and interaction style as consistent as possible.
- Maintain roughly the same number of conversational turns with each model to support a fair comparison.
- Record your voice during each interaction and pay close attention to the quality and behavior of the models’ responses.
- Evaluate each model across five defined evaluation dimensions.
- Identify and categorize relevant error clusters that may appear in the audio or conversation.
- Compare Model A and Model B using the evaluation criteria and your observations.
- Select the model that shows the stronger overall performance.
- Write a detailed rationale explaining your final preference, using specific examples from both conversations.
- Apply the evaluation guidelines consistently across different scenarios and model interactions.
Requirements
- Bachelor’s degree.
- Strong attention to detail and the ability to notice subtle differences in conversational quality.
- Excellent listening and comprehension skills.
- Strong written communication skills, including the ability to explain observations clearly and objectively.
- Ability to follow detailed evaluation guidelines consistently.
- Comfort speaking naturally and role-playing different conversational scenarios.
- Ability to compare two interactions fairly without letting personal preference affect the evaluation.
- Strong critical-thinking and analytical skills.
- Reliability and consistency when completing structured evaluation tasks.
- Familiarity with AI assistants, voice-based AI, or conversational systems.
Compensation
The expected hourly salary range is $20–$25 CAD per hour, based on experience, skills, and qualifications.
Additional Information
- The role is a 3-month contract.
- Applicants should be located in Canada.