About this opportunity
Job Description
Turing is hiring English-speaking AI Quality Analysts to evaluate a personalization feature for Gemini.
In this role, you’ll test how effectively the AI uses information from your past Gemini conversations, Gmail, Google Search, and YouTube activity to provide relevant and useful personalized responses.
The work combines creative prompt design with detailed AI evaluation, including assessing responses for Grounding, Integration, Helpfulness, naturalness, and personalization quality.
What You’ll Do
- Design multi-turn conversational prompts, typically 1–5 turns, based on personal context and experiences.
- Evaluate whether the AI appropriately uses personal information when responding.
- Identify unsupported claims, incorrect personalization, poor inferences, and hallucinations.
- Evaluate Grounding to determine whether personalized claims are supported by available information.
- Assess Integration to determine whether personal information is incorporated naturally.
- Compare two AI responses side by side and rank which provides the better overall experience.
- Write clear rationales explaining your evaluation and ranking decisions.
- Review debug information to verify how data sources and conversation summaries were used.
- Provide detailed annotations and constructive feedback.
- Delete evaluation conversations as required to maintain project data hygiene.
Requirements
- Strong English reading and writing proficiency.
- Excellent analytical and critical-thinking skills.
- Ability to evaluate nuanced and ambiguous AI responses.
- Strong written communication and ability to provide concise evaluation rationales.
- Ability to design creative prompts that test AI personalization.
- Strong attention to detail when comparing model responses.
- Ability to work independently in a remote environment.
- Desktop or laptop with a reliable internet connection.
- Full-time schedule flexibility within your local time zone.
- BS/BA degree or equivalent relevant experience.
Important Personal Account Requirement
Candidates must be willing to use their primary personal Google account, rather than a dedicated testing account, and enable personal data sources required for the evaluation.
The project may involve personalization based on your:
- Gemini conversation history
- Gmail
- Google Search activity
- YouTube activity
Preferred Experience
Experience in data annotation, AI quality evaluation, content moderation, LLM evaluation, or related analytical work is strongly preferred.
Relevant academic or professional backgrounds may include Linguistics, Journalism, Policy, Law, Ethics, Computer Science, and other analytical fields.
