Director, Model Behavior

About Blue Yonder

Blue Yonder is the AI company for supply chain. Our platform helps the world\'s leading companies plan, fulfill, deliver, and operate more resilient supply chains across complex global networks. We are building toward the autonomous supply chain: intelligent systems that can understand operational context, reason through tradeoffs, use tools, collaborate with people, and take action across real supply chain workflows.

About Autonomy Labs

Autonomy Labs\' mission is to find the fastest possible path to an autonomous supply chain. We build LLM agents, learning systems, model training pipelines, evaluations, simulations, and decision-making systems for some of the hardest problems in global supply chain. The work spans LLMs, agentic workflows, tool use, software automation, evaluation, post-training, optimization, and production engineering. In short, we are having a lot of fun.

Your Mission

We are looking for a deeply technical Director of Model Behavior & Evaluation Systems to own the behavioural quality system for Blue Yonder\'s LLM agents. Our agents are not generic chatbots. They are being trained to operate supply chain software: querying state, calling APIs, interpreting operational context, proposing actions, handling exceptions, asking for missing information, and helping users make decisions in complex enterprise environments.

Your mission is to make model behavior a product-quality system, not a collection of dashboards. You will define what \"good\" means for agents operating supply chain workflows, establish the release gates that determine when behavior is ready to ship, and build the feedback loops that turn traces, customer feedback, SME review, telemetry, red-teaming, and eval failures into model improvements.

This is a director-level technical leadership role. You will lead through systems, standards, people, and decisions. You should be close enough to model traces, evals, post-training, tool use, and customer workflows to make strong technical calls, while operating at the level of ownership boundaries, launch authority, roadmap sequencing, and team building.

The stack is real and close to the work. You should expect to operate around Python, PyTorch, Hugging Face Transformers and Datasets, NVIDIA NeMo RL, OpenAI Agents SDK, Langfuse, LLM evaluation harnesses, tool-calling traces, model checkpoints, reward and preference data, synthetic scenarios, experiment reports, and production observability.

You do not need to be the person implementing every pipeline, but you do need the technical depth to challenge designs, read artifacts, understand failure modes, and guide senior engineers toward better systems.

What You\'ll Do

• Own the behavioural quality bar for Blue Yonder\'s LLM agents across customer-facing supply chain workflows.
• Build and lead the model behaviour and evaluation systems function across behaviour specs, eval governance, SME review, release gates, regression coverage, and launch readiness.
• Define launch criteria across operational correctness, tool-use accuracy, workflow completion, escalation quality, safe fallback behaviour, refusal quality, consistency, and customer trust.
• Establish evaluation authority so evals become release decision infrastructure, not just model-quality reporting.
• Set the technical direction for behaviour and eval infrastructure across Python eval harnesses, OpenAI Agents SDK workflows, Langfuse traces, LLM-as-judge workflows, deterministic checks, trace analysis, reward/report versioning, and model-candidate comparison.
• Convert model traces, tool-call failures, SME feedback, red-team findings, telemetry, and customer-facing failures into behaviour specs, eval requirements, training data needs, and model improvement priorities.
• Partner with the reinforcement learning and post-training organization to turn behaviour gaps into SFT data, preference data, reward criteria, curriculum, NeMo RL experiments, model-candidate decisions, and regression tests.
• Partner with workflow, data, product, and domain experts to turn supply chain workflow truth into durable scenario coverage, rubrics, synthetic scenarios, eval datasets, and training data requirements.
• Partner with agent architecture and product engineering teams to ensure prompts, tools, APIs, skills, system instructions, and product workflows express the intended model behaviour consistently.
• Review model traces, eval outputs, experiment summaries, dataset slices, reward reports, and post-training results closely enough to make informed launch and roadmap decisions.
• Own customer and user behaviour discovery for agent workflows: what users expect agents to do, explain, ask, verify, elevate, refuse, and act