AI Evaluation Engineer

As an AI Evaluation Engineer at Delta Exchange, you own the quality of three AI products used by live traders: a customer support chatbot, an API Copilot that generates and executes trading scripts, and an MCP server. You establish what correct means for probabilistic systems, build the datasets and scoring that measure it, and operate the release gate that prevents regressions from reaching customers. This is an emerging discipline with no established playbook, and the role carries a high degree of autonomy and ownership.

What you'll do

• Build and maintain golden datasets for AI products, trace-mined from production or generated, and verified against a documented source of truth
• Design layered scoring for AI outputs: deterministic rules where behaviour can be asserted, and model-graded evaluation where it cannot
• Manage evaluation runs end to end: schedule and execute them against release candidates, compare results against prior baselines, and triage failures into product defects, incorrect expectations, and platform issues
• Own and operate the release gate that determines whether a prompt or model change is approved for production
• Identify failure modes ahead of customers, including hallucination, incorrect tool selection, loss of context across conversation turns, and PII exposure
• Investigate production traces to distinguish retrieval failures from generation failures from tool failures
• Convert findings into actionable engineering evidence and into permanent regression coverage
• Define and report quality metrics for AI surfaces, and drive improvement against them

What you bring

• At least 1 year owning quality for AI products in a lead or primary-owner capacity, including chatbots, conversational assistants, code or content generation products, and agentic systems, working directly with prompts, tool and function schemas, and model behaviour
• Demonstrated hands-on experience building or operating an evaluation harness for an AI system, whether in-house or using a framework such as DeepEval, RAGAS or Promptfoo
• 4-6 years of QA or SDET experience
• Working proficiency in Python for evaluation platform, trace analysis and internal tooling
• Ability to analyse traces and tool calls using observability tooling such as Opik, Langfuse or LangSmith to determine root cause
• Strong API testing experience, as the majority of the surface under test is API-level
• Engineering ability to build your own tooling and automation
• Clear written communication to document findings in a form engineering can act on directly

Nice to have

• Experience with trading or exchange platforms, including familiarity with futures and options, crypto derivatives, margin, or settlement flows

What we offer

• Competitive compensation
• Leadership exposure and significant growth opportunities

About Delta Exchange

Delta Exchange is a cryptocurrency derivatives exchange with a daily traded volume of approximately $10 billion. Founded in 2018 by IIT, ISB, and serial entrepreneurs, the company operates the widest range of derivative products and serves traders globally. Delta is backed by top crypto funds and projects including Sino Global Capital, CoinFund, Gumi Cryptos, Aave, and Kyber Network.