Evaluation Scenario Writer

il y a 4 jours

Paris, Île-de-France Mindrift Temps plein

This opportunity is only for candidates currently residing in the specified country. Your location may affect eligibility and rates. Please submit your resume in English and indicate your level of English.

At Mindrift, innovation meets opportunity. We believe in using the power of collective human intelligence to ethically shape the future of AI. 

What we do

The Mindrift platform, launched and powered by Toloka, connects domain experts with cutting-edge AI projects from innovative tech clients. Our mission is to unlock the potential of GenAI by tapping into real-world expertise from across the globe.

About the Role

We're looking for someone who can design realistic and structured evaluation scenarios for LLM-based agents. You'll create test cases that simulate human-performed tasks and define gold-standard behavior to compare agent actions against. You'll work to ensure each scenario is clearly defined, well-scored, and easy to execute and reuse. You'll need a sharp analytical mindset, attention to detail, and an interest in how AI agents make decisions.

Although every project is unique, you might typically:

  • Designing structured test scenarios based on real-world tasks.
  • Defining the golden path and acceptable agent behavior.
  • Annotating task steps, expected outputs, and edge cases.
  • Working with devs to test your scenarios and improve clarity.
  • Reviewing agent outputs and adapting tests accordingly

How to get started

Simply apply to this post, qualify, and get the chance to contribute to projects aligned with your skills, on your own schedule. From creating training prompts to refining model responses, you'll help shape the future of AI while ensuring technology benefits everyone.

  • Bachelor's and/or Master's Degreein Computer Science, Software Engineering, Data Science / Data Analytics, Artificial Intelligence / Machine Learning, Computational Linguistics / Natural Language Processing (NLP), Information Systems or other related fields.
  • Background in QA, software testing, data analysis, or NLP annotation.
  • Good understanding of test design principles (e.g., reproducibility, coverage, edge cases).
  • Strong written communication skills in English.
  • Comfortable with structured formats like JSON/YAML for scenario description.
  • Can define expected agent behaviors (gold paths) and scoring logic.
  • Basic experience with Python and JS.
  • Curious and open to working with AI-generated content, agent logs, and prompt-based behavior.
  • You are ready to learn new methods, able to switch between tasks and topics quickly and sometimes work with challenging, complex guidelines.
  • Our freelance role is fully remote so, you just need a laptop, internet connection, time available and enthusiasm to take on a challenge.

Nice to Have

  • Experience in writing manual or automated test cases.
  • Familiarity with LLM capabilities and typical failure modes.
  • Understanding of scoring metrics (precision, recall, coverage, reward functions).

Contribute on your own schedule, from anywhere in the world. This opportunity allows you to:

  • Get paid for your expertise, with rates that can go up to $52/hour depending on your skills, experience, and project needs.
  • Take part in a flexible, remote, freelance project that fits around your primary professional or academic commitments.
  • Participate in an advanced AI project and gain valuable experience to enhance your portfolio.
  • Influence how future AI models understand and communicate in your field of expertise.

  • Evaluation Scenario Writer

    il y a 1 jour

    Paris, Île-de-France Mindrift Temps plein

    This opportunity is only for candidates currently residing in the specified country. Your location may affect eligibility and rates. Please submit your resume in English and indicate your level of English.At Mindrift, innovation meets opportunity. We believe in using the power of collective human intelligence to ethically shape the future of AI. What we...

  • Evaluation Scenario Writer

    il y a 22 heures

    Paris, Île-de-France Mindrift Temps plein

    This opportunity is only for candidates currently residing in the specified country. Your location may affect eligibility and rates. Please submit your resume in English and indicate your level of English.At Mindrift, innovation meets opportunity. We believe in using the power of collective human intelligence to ethically shape the future of AI. What we...

  • AI Agent Evaluation Analyst

    il y a 19 heures

    Paris, Île-de-France Mindrift Temps plein

    This opportunity is only for candidates currently residing in the specified country. Your location may affect eligibility and rates. Please submit your resume in English and indicate your level of English proficiency.At Mindrift, innovation meets opportunity. We believe in using the power of collective human intelligence to ethically shape the future of...

  • MCP & Tools Python Developer - Agent Evaluation Infrastructure

    il y a 1 jour

    Paris, Île-de-France Mindrift Temps plein

    This opportunity is only for candidates currently residing in the specified country. Your location may affect eligibility and rates. Please submit your resume in English and indicate your level of English proficiency.At Mindrift, innovation meets opportunity. We believe in using the power of collective human intelligence to ethically shape the future of...

  • Functional Integration Analyst

    il y a 2 semaines

    Paris, Île-de-France Pauwels Consulting Temps plein

    We are looking for an experiencedFunctional Integration Analystto strengthen the test and validation phase for back-office applications. You will support the delivery of new features and corrective changes by preparing test scenarios, executing functional validation and documenting user procedures.This mission sits within the Back Office Applications domain...

  • Banque - Stress tests climatiques, Risques ESG (IT) / Freelance

    il y a 1 semaine

    Paris, Île-de-France Free-Work Temps plein

    Pour notre client, une banque universelle, nous recherchons un expert métier spécialisé en ESG et stress tests climatiques dans le cadre des exercices ICAAP et de pilotage des risques.La mission vise à approfondir et fiabiliser la méthodologie d'évaluation des impacts business liés aux facteurs ESG, à renforcer la quantification des scénarios de...

  • Business Analyst Data

    il y a 6 jours

    Paris, Île-de-France KLETA Temps plein

    Objectifs de la missionAnalyser les parcours GPS sur plusieurs journées, en tenant compte des variations de contexte (charge de distribution, secteurs, conditions terrain), afin d?identifier des tendances récurrentes.Comparer les itinéraires, temps de parcours, distances parcourues et arrêts entre différents scénarios, pour comprendre les écarts de...

  • Consultant en stratégie et montage de projets immobiliers senior H/F

    il y a 4 jours

    Paris, Île-de-France Jobglober Temps plein

    Dans le cadre de son développement, l'entrepise souhaite renforcer l'équipe de son le pôle conseil et développement immobilier, avec un(e) consultant(e) pilote de mission confirmé(e), basé(e) en région lyonnaise ou en région parisienne.création d'une unité de laboratoires : cahier des charges, recherche et contractualisation du foncier, de...

  • Postdoc (H/F) sur l'évaluation des méthodes actuelles de détection de la sélection balancée à partir de

    il y a 2 semaines

    Paris 05 Panthéon, Île-de-France Choisir le Service Public Temps plein

    Informations générales Organisme de rattachement CNRS   Référence UMR7241-VIOLLA-003   Date de début de diffusion /11/2025 Date de parution /12/2025 Date de fin de diffusion /12/2025 VersantFonction Publique de l'Etat CatégorieCatégorie A (cadre) Nature de l'emploiEmploi ouvert uniquement aux contractuels Domaine / MétierRecherche -...

  • IT project Manager COUPA

    il y a 1 semaine

    Paris, Île-de-France Ethic Technology Temps plein

    Périmètre et missions principalesLa prestation s?inscrit dans le cadre d?un programme de transformation global impliquant plus de 20 pays. Elle vise à piloter et orchestrer un programme outillage impactant l?ensemble de la chaîne de valeur des achats (procurement).La mission s?exercera en étroite collaboration avec des ressources internes et externes et...