AI Voice Evaluation Specialist - 3 month contract

Overview

Innodata is hiring a remote AI Voice Evaluation Specialist for a 3-month, W2, full-time contract at 40 hours per week. This role focuses on evaluating AI model performance through real-time, voice-based conversations and comparing responses across two models using a structured evaluation framework.

The position is based in the United States and is intended for candidates who can work remotely while meeting the role’s eligibility requirements, including authorization to work in the US.

What You'll Do

• Review assigned scenarios and roleplay natural conversations with two different AI models.
• Keep the scenario, conversational approach, and interaction style as consistent as possible across both models.
• Maintain a similar number of turns in each conversation to support a fair comparison.
• Record your voice during each interaction and observe the quality and behavior of the models’ responses.
• Evaluate both models across five defined dimensions.
• Identify and categorize relevant error clusters in the audio or conversation.
• Compare Model A and Model B using the stated criteria and your observations.
• Select the model with the stronger overall performance.
• Write a detailed rationale that explains your final preference with specific examples from both conversations.
• Apply the evaluation guidelines consistently across different scenarios and interactions.

Requirements

• Bachelor’s degree.
• Strong attention to detail and the ability to notice subtle differences in conversational quality.
• Excellent listening and comprehension skills.
• Strong written communication skills and the ability to explain observations clearly and objectively.
• Ability to follow detailed evaluation guidelines consistently.
• Comfort speaking naturally and role-playing different conversational scenarios.
• Ability to compare two interactions fairly without letting personal preferences influence the evaluation.
• Strong critical-thinking and analytical skills.
• Reliability and consistency in completing structured evaluation tasks.
• Familiarity with AI assistants, voice-based AI, or conversational systems.