Research Scientist, Agentic Data & Benchmarking
IFM is hiring a Research Scientist, Agentic Data & Benchmarking in Sunnyvale, United States. It pays $150k-$450k a year and Level rates it ; you can apply on IFM.
Level's read on this job
- Level rates this research scientist role Builds AI ●●●●: the job is building AI systems.
- The posting mentions PyTorch, AI agents and fine tuning among its tools and skills.
- Level classes it as a mid-level role.
- IFM has 43 other open roles on Level.
AI in this role
About the Institute of Foundation Models
The Institute of Foundation Models (IFM) is a dedicated research lab for building, understanding, using, and risk-managing foundation models. Our mandate is to advance research, nurture the next generation of AI builders, and drive transformative contributions to a knowledge-driven economy.
As part of our team, you'll work at the core of cutting-edge foundation model training, alongside world-class researchers, data scientists, and engineers, tackling the most fundamental and impactful challenges in AI development. You'll help build groundbreaking AI systems with the potential to reshape entire industries, and contribute to establishing MBZUAI as a global hub for high-performance computing and deep learning.
About the role
The Agents team trains advanced agentic language models that use reasoning and tool use to complete real tasks on a computer. This is a specialist role at the center of the loop that drives those models: the data we train on and the benchmarks we measure against.
You'll own the agentic data pipeline end-to-end — sourcing and generating high-quality trajectories, tool-use data, and RL environments — and the evaluation suite that tells us, rigorously and reproducibly, what our agents can actually do. These two halves are inseparable: benchmarks expose where models fail, and targeted data closes the gap. The agents are only as good as the data they learn from and the evals that keep us honest, and this role owns both.
This is a research scientist position for someone who wants depth in data and measurement rather than breadth across the whole stack. You should be the kind of person who reads through datasets line by line, distrusts a metric until it's been validated, and gets satisfaction from making an eval suite that nobody questions.
Key responsibilities
-
Design and run evaluations of agentic capabilities — multi-step reasoning, tool use, long-horizon planning, computer use, and safety properties — turning ambiguous notions of "intelligence" into defensible, reproducible metrics.
-
Build and harden evaluation harnesses so benchmarks run reliably at scale against training checkpoints, with clear signal on regressions and model health.
-
Run experiments characterizing how prompting, sampling, scaffolding, and environment design affect agentic performance on internal and public benchmarks.
-
Diagnose anomalous eval results mid-training-run — determine whether the cause is the model, the data, the harness, or the infrastructure — and communicate the answer clearly.
-
Source, generate, and curate high-quality agentic training data: trajectories, tool-use traces, and task datasets for new capabilities.
-
Design and scale RL environments and reward signals, and measure their impact on model performance.
-
Manage technical relationships with external data vendors and domain experts, evaluating data quality and iterating quickly on feedback.
-
Develop QA frameworks that catch reward hacking, label noise, and contamination, keeping data and benchmark quality high.
-
Contribute to technical reports, research publications, and open-source benchmarks and tooling.
-
Partner with research and product teams to translate capability goals into measurable data and evaluation artifacts.
Benchmarking & evaluation
Agentic data
Across both
Qualifications
-
BS, MS, or PhD (or equivalent experience) in Computer Science, Machine Learning, or a related field.
-
2+ years of experience with a clear emphasis on evaluations and/or training-data curation for ML systems (related areas: LLM training/fine-tuning, RL, or distributed ML systems).
-
Strong Python and PyTorch development experience.
-
Demonstrated experience designing and deep-diving into evaluations, or curating and generating training datasets — ideally both.
-
Hands-on experience using LLM agents in your personal or professional work.
-
A habit of reading through raw data and trajectories to understand them and spot issues, and an instinct to distrust a metric until it's validated.
-
Experience with reinforcement learning, reward design, or RL environment construction for LLMs.
-
Background in statistics and experimental design — a feel for signal-to-noise, statistical power, and contamination in evaluations.
-
Experience with large-scale dataset sourcing, curation, and processing, including working with external vendors or domain experts.
-
Strong knowledge of the literature on agent evaluation, RL, LLM reasoning, and tool use.
-
Experience building or operating data pipelines and evaluation infrastructure reliable at scale (e.g., PyTorch, Ray).
-
Experience evaluating or generating data for software-engineering or computer-use agents.
-
Contributions to published research, public benchmarks, and/or open-source ML software.
Academic qualifications
Minimum qualifications
Preferred qualifications
Representative projects
-
Stand up a new agentic benchmark from scratch — define the task, build the dataset and scoring, validate against known signals, and ship a view that makes the result legible to researchers and leadership.
-
Build an RL environment for a new high-value capability: design the reward, generate and QA the trajectory data, and measure the lift on model performance.
-
Diagnose a mid-training regression: an eval suite returns anomalous numbers and you determine whether it's the model, the harness, the data, or the infrastructure.
-
Partner with an external data vendor or domain expert to source high-quality trajectories, then build the QA framework that keeps reward hacking and contamination out.
-
Take a flaky distributed eval pipeline and make it reliable — better retries, better observability, faster feedback to researchers.
Compensation
$150,000 – $450,000/year + discretionary performance bonus + benefits
The posted salary range represents the company’s good faith estimate of the base compensation for this position upon hire. The actual base compensation offered may vary within this range depending on individual qualifications, including but not limited to relevant skills, experience, education, certifications, and specific business needs. In addition to base salary, total compensation includes an annual discretionary performance bonus and comprehensive benefits.
Benefits Include
- Comprehensive medical, dental, and vision benefits
- Bonus
- 401K Plan
- Generous paid time off, sick leave and holidays
- Paid Parental Leave
- Employee Assistance Program
- Life insurance and disability
How we rate this
Research Scientist, Agentic Data & Benchmarking at IFM rates 100 out of 100 for how much of the daily work is AI. That makes it Builds AI (AI Level 4 of 4). The level is about AI in the job, not seniority.
Builds AI. The job is building AI systems.
- ●●●● Builds AI80 to 100
- ●●●○ Works on AI60 to 79
- ●●○○ Uses AI40 to 59
- ●○○○ Little AI0 to 39
Levels come from how often the tools, models and workflows of the role are named in the posting itself. Read the description and count the AI tools, models and workflows it names.
Prepare for this job
Built from this posting and from similar jobs on Level.
Skills and AI tools this role asks for
Also common in research scientist jobs at this level
Pay
- This job starts at
- $150k
- Typical research scientist job
- $192k
This posting starts 22% below the typical research scientist job. See research scientist jobs
Questions you could be asked
- How do you decide when an AI agent can act on its own versus asking for approval first?
- Walk me through fine-tuning a model: what data did you use, and how did you check the result?
- What are the limits of PyTorch that you've run into, and how did you work around them?
- How would you decide a model or AI system is ready to ship?
- Tell me about a time a model underperformed in production. How did you find out, and what did you change?
Adapt your resume
- List these exact terms on your resume: AI agents, Fine-tuning, and PyTorch. An applicant tracking system matches the wording, not the idea.
- Attach one line of real, concrete experience to at least one of them. A tool named with nothing behind it rarely survives a human read.
- Lead with what you built, trained or shipped. This role is judged on the AI system itself, not the tools around it.
Get your CV reviewed before you apply
Send your CV and the role you want next. Your review arrives by email within 3 days.
Get new research scientist jobs (Builds AI ●●●●) by email
One email a week, each job rated for how much AI is in the work. No recruiter spam, unsubscribe in one click.
Free.
Similar roles
Research roles that build AI, at other companies.
What kind of AI work fits you?
Answer 12 practical questions in about three minutes. Get a simple profile, the work it points to, and live roles to explore next.
Find my next step