Senior Agent Evaluation Engineer, AI Evaluation Platform

Apple

Location
Seattle, WA
Pay
$175,000–$263,300 a year
Level
Senior
Posted
, on Apple's careers site

Sponsorship check

What we checked before showing you this job.

  • Apple sponsors H-1B visas

    7,362 filings in FY2026, and every year since 2020.

  • E-Verify employer

    You can take this job on STEM OPT.

  • The post doesn't mention sponsorship

    Ask the recruiter early.

Always confirm with the recruiter. hire.page shows public records and what the post says, not a promise from Apple.

About the job

Join Apple Services Engineering to define how we measure whether AI agents actually work. We are seeking a senior engineer who has built agents and knows how to evaluate them rigorously. You will design evaluation suites, build and validate judges, and ship agents on our evaluation platform, so the platform is shaped by real use and not guesswork. You will partner with researchers, product teams, and platform engineers to turn evaluation methodology into tools other teams can adopt. We are looking for builders who move quickly, iterate in the open, and can tell a trustworthy evaluation from a theatrical one.

Description

We're building the evaluation platform that will serve all of Apple's generative AI and agent systems. Building agents got faster. Knowing whether they got better did not. This role closes that gap.

You'll build real agents and evaluate them: multi-turn trajectories, tool-use chains, non-deterministic outputs, and judge reliability. You'll write evaluation suites, validate judges against human labels, and measure variance across trials. Some agents you build will evaluate other systems (test-case generation, judging, failure triage). Others will be reference agents or built with partner teams. When the platform gets in your way, you'll file the gap, and often fix it yourself.

This is a hands-on role with a lot of autonomy. You'll work in Python with a team of 7-12 engineers.

Responsibilities

Build and ship: Take ownership of features and services within the evaluation platform: APIs, SDKs, orchestration components, evaluation runners. You'll have the room to make calls on your own work and the support to deliver it well.

Productionize ML research: Partner with research engineers to take their prototype code and turn it into reliable services. You'll learn their world quickly and translate research patterns into clean Python that holds up under real load.

Move fast, responsibly: You'll get scoped problems with room to figure out the how. We trust you to balance speed with care, to know when something needs a quick prototype and when it needs a design doc, tests, and a careful rollout.

Improve as you go: Notice the rough edges and pick them up. The flaky test, the slow build, the confusing API, the runbook that's out of date. We want someone who leaves the codebase a little better every week.

Developer experience: Help build the SDKs and abstractions that other Apple teams use to evaluate their models and agents. You'll feel the friction of bad ergonomics directly, which puts you in a great position to fix it.

Operational ownership: Your code runs in production. You write the tests, set up the CI, add the metrics, and stay close when something breaks. You don't need to be an SRE, but you take care of what you ship.

Minimum qualifications

* 4-8 years of software engineering experience shipping production software.

* Hands-on agent builder. You've built LLM agents beyond demos: tool use, MCP, multi-step workflows, context management, error recovery.

* Evaluation depth. You've designed evaluations for agents, not just models. You understand non-deterministic outputs, multi-trial measurement (pass@k, variance), trajectory scoring, and scoring drift. You've worked with tools like Inspect, Braintrust, LangSmith, or Promptfoo, or built your own.

Judge rigor. You know how to build an LLM judge, validate it against human agreement, and know when not to trust it.

Fast from idea to prototype. You can point to things you built quickly, then iterated on or shipped. You can also tell us about a prototype you killed, and why.

Fluent with AI coding tools. You use Claude Code or equivalents daily, including skills, slash commands, and agent-style workflows.

Python as primary language. Strong with FastAPI, Pydantic, and the ecosystem. Your code is clean, tested, and easy for the next person to pick up.

Judgment on speed vs. discipline. You know when a quick prototype is enough and when you need a design doc, tests, and a careful rollout.

Cross-team collaboration. You've worked closely with researchers, product teams, or platform engineers. You write clearly and flag blockers early.

Preferred qualifications

Experience working on developer platforms, internal tools, or SDKs

Production experience with LLM/agent systems — building, evaluating, or operating them

Familiarity with job orchestration frameworks (Temporal.io, Airflow, or similar)

Distributed compute experience (Ray, Dask, or Kubernetes-based job systems)

Experience with experiment tracking or ML lifecycle tooling (Weights & Biases, MLflow, etc.)

Startup or early-stage experience where you wore multiple hats and shipped under constraint

The base pay range for this role is between $175,000 and $263,300, and your base pay will depend on your skills, qualifications, experience, and location.\u003cbr\u003e\u003cbr\u003eApple employees also have the opportunity to become an Apple shareholder through participation in Apple’s discretionary employee stock programs. Apple employees are eligible for discretionary restricted stock unit awards, and can purchase Apple stock at a discount if voluntarily participating in Apple’s Employee Stock Purchase Plan. You’ll also receive benefits including: Comprehensive medical and dental coverage, retirement benefits, a range of discounted products and free services, and for formal education related to advancing your career at Apple, reimbursement for certain educational expenses — including tuition. Additionally, this role might be eligible for discretionary bonuses or commission payments as well as relocation. \u003ca href='https://www.apple.com/careers/us/benefits.html' target='_blank' aria-label='Learn more about Apple Benefits (Opens in new window)'\u003eLearn more about Apple Benefits\u003cspan class='icon icon-after icon-external' aria-hidden='true'\u003e\u003c/span\u003e\u003c/a\u003e\u003cbr\u003e\u003cbr\u003eNote: Apple benefit, compensation and employee stock programs are subject to eligibility requirements and other terms of the applicable plan or program.\u003cbr\u003e

Job text from Apple's careers site.