Text Situated Alignment Specialist - Mid-Cost

Overview

ExpertWoka is sourcing Subject Matter Experts to support AI conversation evaluation research. This is a contractor assignment with no medical or paid leave.

Selected experts will review conversations between participants and AI models, assess whether sufficient context was gathered before recommendations were made, identify missed opportunities for clarification, and provide structured feedback to improve conversational quality.

Responsibilities

• Review AI conversations involving real-world interpersonal and everyday decision-making scenarios.
• Identify missing context and recommend relevant follow-up questions.
• Evaluate whether additional information would have changed the AI’s recommendations.
• Distinguish between critical and non-essential information.
• Conduct qualitative reviews and document findings using structured evaluation guidelines.
• Collaborate with research teams to ensure consistent, high-quality assessments.

Requirements

• 3–5 years of direct practice experience in a relevant field.
• Background in at least one of the following areas: social work, counseling, therapy, psychology, qualitative research, HCI, communication, mediation, coaching, student services, organizational consulting, or related advisory fields.
• Professional experience relevant to client assessment, case review, interviewing, counseling, qualitative research, user research, conversation analysis, or similar work.
• Must be based in one of the eligible regions listed for this opportunity.
• Full-time availability for the project duration.

Qualifications

• Social work qualification such as MSW and an applicable license or credential (for example LCSW, LICSW, or LMSW).
• Counseling or therapy qualification such as MA/MS with an applicable license or credential (for example LMFT, LPC, or LCPC).
• PhD or PsyD in clinical, counseling, social, or industrial-organizational psychology.
• Advanced degree or equivalent expertise in HCI, communication, anthropology, sociology, or qualitative research.
• Relevant professional qualification or demonstrated subject-matter expertise in mediation, coaching, student services, organizational consulting, or a related advisory field.

Preferred Qualifications

• Experience eliciting relevant context when initial information is incomplete.
• Experience working across relationships, work, education, family, planning, wellbeing, and resource constraints.
• Qualitative coding or structured case review experience.
• Ability to distinguish between load-bearing context and merely descriptive information.
• Experience reviewing interviews, consultations, coaching sessions, or client interactions to identify missed opportunities for clarification or information gathering.

Project Details

• Engagement length: 3 weeks.
• Start date: October 1st (estimated).
• Project type: pay-per-task.
• Average handling time: approximately 75 minutes.
• Commitment required: full-time, 40 hours per week.

Evaluation Process

• Profile review.
• ICF.