Agent Evaluation Analyst (Freelance)

Company Intro

At Toloka AI we create data that powers leading GenAI models and innovations. We work with frontier labs, big tech, renowned AI startups, enterprises and non-profit research organizations worldwide. We use a combination of Experts + Crowd + Tech Platform to teach AI models to reason and evaluate their efficacy and safety. We have experts in more than 50 different domains-from doctors and lawyers to physicists and engineers-and boast one of the most diverse global crowds, representing over 100 countries and speaking 40+ languages. We are a well-funded startup with an enviable portfolio of clients including Anthropic, Amazon, Microsoft, poolside, Recraft, and Shopify.

Recently, we secured strategic investment led by Bezos Expeditions with participation from Mikhail Parakhin, CTO of Shopify and board advisor to leading GenAI companies, who now serves as our Chairman of the Board. Our remote-first team is globally distributed around the world: USA, UK, the Netherlands, Israel, Czech Republic, Serbia, and more. We are headquartered in Amsterdam.

About the Role

We are looking for an Freelance Agent Evaluation Analyst to take ownership of quality, structure, and insight across the project. This role goes far beyond task-checking - it’s about critical thinking, systems-level analysis, and ensuring clarity, reliability, and consistency at scale.You’ll work as both a hands-on evaluator and an analyst, collaborating with domain experts, delivery managers, and engineers. Beyond reviewing outputs, you’ll be expected to understand the “why” behind the work, identify logical gaps or inconsistencies, and propose meaningful improvements.

This is a flexible, impact-driven role where you’ll have space to grow, contribute ideas, and help shape how evaluation and quality are scaled across the project.

This role is especially well-suited for:
Analysts, researchers, or consultants with strong structuring and reasoning skills
Junior product managers or strategists curious about AI and evaluation work
Smart problem-solvers (students or early-career professionals) who enjoy digging into logic, systems, and edge cases

You do not need a coding background. What matters most is curiosity, intellectual rigor, and the ability to evaluate complex setups with precision.

What you’ll be doing

• Fully own the QA pipeline for agent evaluation tasks;
• Review and validate tasks and golden paths created by scenario writers and experts;
• Spot logical inconsistencies, vague requirements, hidden risks, and unrealistic assumptions;
• Provide structured feedback and ensure quality alignment across contributors;
• Train, onboard, and mentor new QA team members;
• Collaborate with domain experts, delivery managers, and engineers to improve test clarity and coverage;
• Maintain and improve QA checklists, SOPs, and review guidelines;
• Contribute to test planning, prioritization, and quality benchmarks;
• Take initiative to suggest new approaches, tools, and processes that help scale validation and analysis.

What you should know / be able to do

• Strong analytical and critical thinking skills;
• Attention to detail and reliability - your work can be trusted without double-checking;
• Experience in manual QA, scenario validation, or similar analytical work;
• Comfortable working with structured formats (JSON/YAML);
• Clear written communication and documentation skills;
• Ability to give constructive feedback and coach others;
• Capable of working with a wide range of stakeholders: from engineers to directors/VPs.

Nice to have

• Background in scenario-based testing, test design, or annotation workflows;
• Experience with AI/LLM evaluation, prompt validation, or agent behavior testing;
• Some technical independence (e.g., Python skills);
• Familiarity with MCP / tool-based task execution;
• Experience working in cross-functional teams across product, delivery, and engineering.

Who you are

• Detail-obsessed but also able to see the bigger picture;
• Proactive, independent, and take true ownership of your work;
• Strong communicator who can turn complex findings into actionable insights;
• Flexible and motivated to contribute across a variety of tasks and projects;
• Believe quality is not just checking work, but making the whole product better.

What we can offer

• Freelance contract (B2B);
• Flexible payment based on the results of work;
• Flexibility: we offer freelance employment. You will also design with your manager a workday that works best for you;
• Friendly community.