Agentic Evaluation Specialist (Bengali)

Overview

Innodata is looking for an experienced Agentic Evaluation Specialist to assess how an AI agent performs on real-world, multi-step tasks. This is evaluation work focused on agentic behavior rather than simple chatbot replies. The role is based in Singapore, with remote work arrangement, and is offered as a freelancing engagement for about 1 month, extendable. The hourly commitment is 4-5 hours daily with flexible scheduling.

The work requires careful judgment of agent traces, tool use, and response quality, with short written justifications for your decisions. Applicants should be comfortable reviewing structured outputs and evaluating whether an agent’s final answer is truly correct based on the underlying trace.

Responsibilities

• Review agentic traces from AgentX, including tool calls, tool results, and interactions with the data environment, to understand the full sequence of steps taken.
• Compare two candidate responses for the same task and choose the one that better follows instructions, completes the task, and behaves appropriately.
• Check each response against the trace to verify whether numbers, facts, and conclusions align with what the tools returned.
• Identify hallucinations, misreadings, skipped steps, or other issues that affect response quality.
• Write concise justifications, typically 2-3 sentences, explaining why one response is better.
• Apply consistent evaluation standards across accuracy, helpfulness, safety, and instruction-following.
• Assess language quality and cultural appropriateness for the Singapore language market.
• Flag unclear, incomplete, or ambiguous cases according to the provided guidelines.

Requirements

• Native proficiency in Bengali and strong English comprehension for task instructions.
• Familiarity with agentic AI workflows, including tool calling, multi-step task execution, and agent interaction with external systems and data.
• Ability to read structured technical output such as JSON, tables, basic SQL queries, and API responses.
• Strong attention to detail and consistent application of evaluation criteria.
• Clear written English for rating justifications.
• Based in Singapore, with relevant cultural and market familiarity.
• Master's degree is preferred; a Bachelor's degree with relevant experience may also be considered.

Preferred Qualifications

• Prior experience in AI data labeling, content rating, RLHF, or model evaluation, especially side-by-side comparison tasks.
• Background in linguistics, localization, QA, data analysis, or related fields.
• Experience working with AI assistants or agent-based tools in a professional or personal capacity.