Evaluation Scenario Writer - AI Agent Testing Specialist

2 days ago, be among the first 25 applicants.

This opportunity is only for candidates currently residing in the specified country. Your location may affect eligibility and rates. Please submit your resume in English and indicate your level of English.

At Mindrift, innovation meets opportunity. We believe in using the power of collective human intelligence to ethically shape the future of AI.

What We Do

The Mindrift platform, launched and powered by Toloka, connects domain experts with cutting‑edge AI projects from innovative tech clients. Our mission is to unlock the potential of GenAI by tapping into real‑world expertise from across the globe.

About The Role

We’re looking for someone who can design realistic and structured evaluation scenarios for LLM‑based agents. You'll create test cases that simulate human‑performed tasks and define gold‑standard behavior to compare agent actions against. You'll work to ensure each scenario is clearly defined, well‑scored, and easy to execute and reuse. You'll need a sharp analytical mindset, attention to detail, and an interest in how AI agents make decisions.

Key Responsibilities

• Design structured test scenarios based on real‑world tasks.
• Define the golden path and acceptable agent behavior.
• Annotate task steps, expected outputs, and edge cases.
• Work with developers to test scenarios and improve clarity.
• Review agent outputs and adapt tests accordingly.

How To Get Started

Simply apply to this post, qualify, and get the chance to contribute to projects aligned with your skills, on your own schedule. From creating training prompts to refining model responses, you'll help shape the future of AI while ensuring technology benefits everyone.

Requirements

• Bachelor’s and/or Master’s degree in Computer Science, Software Engineering, Data Science, AI/ML, Computational Linguistics/NLP, Information Systems, or related fields.
• Background in QA, software testing, data analysis, or NLP annotation.
• Good understanding of test design principles (e.g., reproducibility, coverage, edge cases).
• Strong written communication skills in English.
• Comfortable with structured formats like JSON/YAML for scenario description.
• Ability to define expected agent behaviors (gold paths) and scoring logic.
• Basic experience with Python and JavaScript.
• Curiosity and openness to work with AI‑generated content, agent logs, and prompt‑based behavior.
• Prepared to learn new methods, switch between tasks and topics quickly, and tackle challenging guidelines.
• Remote freelance role—requires laptop, internet connection, available time, and enthusiasm.

Nice to Have

• Experience writing manual or automated test cases.
• Familiarity with LLM capabilities and typical failure modes.
• Understanding of scoring metrics such as precision, recall, coverage, reward functions.

Benefits

• Work on your own schedule from anywhere in the world.
• Get paid for your expertise, with rates up to $52 per hour depending on skills, experience, and project needs.
• Participate in an advanced AI project and gain valuable experience for your portfolio.
• Influence how future AI models understand and communicate in your field of expertise.