AI Voice Evaluation Specialist

Overview

Innodata is seeking detail-oriented Voice Specialists to evaluate AI model performance through real-time, voice-based conversations. In this role, you will interact with two different AI models using the same assigned scenario, compare their responses, and provide structured evaluations based on defined quality criteria. The objective is to make a fair and consistent comparison between models and determine which delivers the stronger overall conversational experience.

What You’ll Do

• Review an assigned scenario and roleplay a natural conversation with two different AI models.
• Conduct comparable conversations with Model A and Model B while keeping the scenario, approach, and interaction as consistent as possible.
• Maintain approximately the same number of conversational turns with each model to support a fair comparison.
• Record your voice during each interaction and pay close attention to the quality and behavior of the model responses.
• Evaluate each model across five defined evaluation dimensions.
• Identify and categorize relevant error clusters that may occur in the audio or conversation.
• Compare Model A and Model B based on the evaluation criteria and your observations.
• Select the model with the stronger overall performance according to the evaluation framework.
• Write a detailed rationale for your final preference, referencing specific examples and observations from both conversations.
• Apply evaluation guidelines consistently across different scenarios and model interactions.

Requirements

• Bachelor’s degree.
• Strong attention to detail and ability to notice subtle differences in conversational quality.
• Excellent listening and comprehension skills.
• Strong written communication skills and the ability to explain observations clearly and objectively.
• Ability to follow detailed evaluation guidelines consistently.
• Comfort speaking naturally and roleplaying different conversational scenarios.
• Ability to compare two interactions fairly without allowing personal preferences to influence the evaluation.
• Strong critical-thinking and analytical skills.
• Reliability and consistency when completing structured evaluation tasks.
• Familiarity with AI assistants, voice-based AI, or conversational systems.