Senior Software Engineer (Agent Training and Evals)

AI in this role

ai-agentsfine-tuningml-opsai-research

Senior Software Engineer - (Agent Training and Evals)

Engineering

Prolific

Prolific is not just another player in the AI space – we are the architects of the human data infrastructure that's reshaping the landscape of AI development. In a world where foundational AI technologies are increasingly commoditized, it's the quality and diversity of human-generated data that truly differentiates products and models.

The role

As a Senior Software Engineer, you'll be one of the first engineers on our Agent Training & Evals team.
You'll build infrastructure to evaluate AI agents at scale against real human expertise, in safe and secure environments. The better we get at this, the more useful, reliable and safe agents become.
This is a genuine 0→1 role. There isn't a well-established playbook or existing architecture for you to simply maintain. You’ll shape the team’s technical direction and the products we build.
You'll work closely with engineering, product and AI research to turn ambiguous problems into scalable systems, and you'll see the impact of what you build directly.

What you’ll be doing in the role

Build a platform to support evaluating and fine-tuning agents
Take prototypes to production; turn experiments and research ideas into reliable, well-tested software
Own what we ship as it scales in production
Solve hard problems around data, compute, orchestration, scaling, security and reliability
Work with emerging agentic AI technologies and rapidly evolving models
Make effective product and technical decisions in a highly experimental environment
Partner closely with customers, product engineering and AI research to turn problems and opportunities into working products
Drive direct impact in a fast-moving, high-ownership, cross-functional team

What you’ll bring to the role

A background in Software Engineering, Data Engineering or MLOps
Strong experience building scalable infrastructure or distributed systems in the cloud
Solid understanding of how systems are deployed, scaled and secured
Experience applying good engineering practice and raising the bar for those around you
Hands-on experience with integrating and evaluating AI models or agents
Interest in how AI agents are trained, evaluated and improved
A product mindset with a focus on delivering business and customer value
Comfort with ambiguity, and the ability to move quickly from idea to working system
Python experience is useful but not essential

Key Technologies

Cloud Platforms: Google Cloud Platform
Programming Languages: Python, JavaScript, and TypeScript
Frameworks: FastAPI, Vue.js, Container-based and Serverless architectures
Databases: PostGres SQL
DevOps and Monitoring: GitHub Actions, Kubernetes, Datadog, Langfuse

Why Prolific is a great place to work

We've built a unique platform that connects researchers and companies with a global pool of participants, enabling the collection of high-quality, ethically sourced human behavioral data and feedback. This data is the cornerstone of developing more accurate, nuanced, and aligned AI systems.

We believe that the next leap in AI capabilities won't come solely from scaling existing models, but from integrating diverse human perspectives and behaviors into AI development. By providing this crucial human data infrastructure, Prolific is positioning itself at the forefront of the next wave of AI innovation – one that reflects the breadth and the best of humanity.

Working for us will place you at the forefront of AI innovation, providing access to our unique human data platform and opportunities for groundbreaking research. Join us to enjoy a competitive salary, benefits, and remote working within our impactful, mission-driven culture.

Links to more information on Prolific

Privacy Statement

By submitting your application, you agree that Prolific may collect your personal data for recruiting and global organisation planning. Prolific's Candidate Privacy Notice explains what personal information Prolific may process, where Prolific may process your personal information, its purposes for processing your personal information, and the rights you can exercise over Prolific use of your personal information.

How we rate this

Senior Software Engineer (Agent Training and Evals) at Prolific rates 66 out of 100 for how much of the daily work is AI. That makes it Works on AI (AI Level 3 of 4). The level is about AI in the job, not seniority.

Classification

Works on AI. The daily work is on AI products, without building the model.

●●●● Builds AI80 to 100
●●●○ Works on AI60 to 79
●●○○ Uses AI40 to 59
●○○○ Little AI0 to 39

Levels come from how often the tools, models and workflows of the role are named in the posting itself. Open the description and count.

Prepare for this job

A free preview built only from this posting: what it asks for, what you could be asked in an interview, and how to adjust your resume.

Skills and AI tools this role asks for

AI AgentsFine TuningML OpsAI Research

Questions you could be asked

How do you decide when an AI agent can act on its own versus asking for approval first?
Walk me through fine-tuning a model: what data did you use, and how did you check the result?
How do you monitor a model once it's live, and how do you know it needs retraining?
Tell me about a research question you investigated. What did you find?
Describe a typical day in a role like this one: which parts run through AI directly?

Adapt your resume

List these exact terms on your resume: AI Agents, Fine Tuning, ML Ops, and AI Research. An applicant tracking system matches the wording, not the idea.
Attach one line of real, concrete experience to at least one of them — a tool named with nothing behind it rarely survives a human read.
Show where AI is part of your daily process, not a one-off project — this role expects it to be a running habit.

Want an expert to read your CV for this job?

Free. Send your CV and the role you want next. We reply by email within 2 to 4 business days.

Get a free CV review

Get new remote software engineer jobs (Works on AI ●●●○ or higher) by email

One email a week with the new remote software engineer jobs (Works on AI ●●●○ or higher), each rated for how much AI is in the work. No recruiter spam, unsubscribe in one click.

Also alert me for new jobs at Prolific

Free. One email a week. Unsubscribe in one click.