AI Quality Engineer

AI in this role

prompt-engineeringai-agentsai-evaluation

About Rootly

At Rootly, we are on a mission to be the go-to way companies respond when things go wrong, helping every organization be more reliable. We do this by building an industry-leading incident management platform that allows companies around the world to consistently and quickly resolve incidents. We are not simply transforming an industry, we are carving an entirely new +$B segment ourselves and need incredible talent to achieve this ambitious goal together.
Customers love Rootly. Some of the fastest growing companies around the world such as NVIDIA, Figma, Canva, Tripadvisor, Squarespace and more rely on Rootly to power their critical incident management process. They obsess over our delightful enterprise-ready platform and unique partnership model. See why our customers have reviewed us 5 stars on G2.
Investors love Rootly. We are backed by some of the most respected funds in the world from Y Combinator to operators like the CTO of Dropbox and GitHub. We'd be happy to disclose our entire funding and profitability picture live during the interview. As a culture we relentlessly put transparency first. We conduct monthly financial reviews as a team so everyone has a pulse on the health of the business and publish what we are building in our weekly changelog.
Rootly is building the AI-native future of incident management, and we need someone who can push our AI to its limits before our customers do. As our AI Quality Engineer, you'll own the evaluation and optimization of Rootly's agentic AI features -- designing test scenarios, running adversarial prompts, interpreting outputs, and working directly with engineering and product to close the loop on performance.
This isn't traditional QA. You'll spend your days thinking like an attacker, a confused user, and a power user all at once -- probing how our AI agents reason, make decisions, and handle edge cases across complex incident workflows.
What you'll do

Design and execute prompt-based test scenarios that cover happy paths, edge cases, and adversarial inputs across Rootly's agentic AI features
Evaluate AI outputs for accuracy, relevance, consistency, and alignment with expected workflow behaviour
Build and maintain an evaluation framework; structured test libraries, scoring rubrics, and regression suites to track AI performance over time
Identify failure modes, hallucinations, reasoning gaps, and unexpected agent behaviours; document findings and work with engineers to resolve them
Partner with Product and Engineering on new AI feature releases, contributing to acceptance criteria and quality gates before launch
Define and track quality metrics (accuracy rates, failure frequency, regression trends) and report findings to stakeholders
Stay current on LLM evaluation techniques, prompt engineering best practices, and agentic testing methodologies

What we're looking for

+5 years in QA, product operations, AI/ML evaluation, or a closely related role
Hands-on experience testing or evaluating LLM-powered or agentic AI products
Strong prompt engineering instincts -- you understand how wording, context, and structure affect model behaviour
Comfortable writing scripts or working with evaluation tools (Python a plus; not required to be a full-stack engineer)
Sharp analytical thinking; you can spot a subtle reasoning failure and articulate exactly why it's a problem
Clear written communicator; able to translate AI behaviour findings for both technical and non-technical audiences

Nice to have

Familiarity with incident management, DevOps, or IT operations workflows is a strong asset
Experience with evaluation frameworks (e.g. LangSmith, PromptFlow, Braintrust, or similar)
Exposure to red-teaming or adversarial testing of AI systems
Comfortable writing E2E tests with Playwright
Background working at a B2B SaaS or developer-tools company
Familiar with mobile app testing (iOS/Android)

Why Rootly?

We’re not just another startup. We’re building something category-defining and want teammates who crave ownership, love solving hard problems, and thrive in a high-bar, high-impact environment.
Here’s what you can expect when you join Rootly:

Competitive compensation and early equity in a fast-growing, venture-backed company.
Comprehensive medical, dental, and vision coverage.
3 weeks of vacation, plus unlimited sick and mental health days, and a company-wide end-of-year shutdown to recharge.
$500 stipend for home office setup.
Unlimited token usage and access to AI tools
A fast-moving, high-impact environment where your leadership and ideas directly shape the future of the company.

If this sounds like the kind of challenge and opportunity you’re looking for, apply now and let’s build something great together.
Rootly is an equal opportunity employer. We aim to create an environment where every team member at Rootly feels like they belong so they can have a greater impact on our business and customers. We do not discriminate on the basis of race, religion, colour, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.

How we score this

AI Quality Engineer at Rootly scores 65 out of 100 for how much of the daily work is AI. That makes it AI Level 3 of 4 (Works on AI). The level is about AI in the job, not seniority.

Classification

AI Level 3. The daily work is on or around AI systems, without necessarily building the model: remove AI and the job is hollow.

AI Level 480 to 100
AI Level 360 to 79
AI Level 240 to 59
AI Level 10 to 39

Bands come from how often the tools, models and workflows of the role are named in the posting itself. Open the description and count.

Prepare for this job

A free preview built only from this posting: what it asks for, what you could be asked in an interview, and how to adjust your resume.

Skills and AI tools this role asks for

Prompt EngineeringAI AgentsAI Evaluation

Questions you could be asked

How do you structure and test a prompt to get consistent output from a language model?
How do you decide when an AI agent can act on its own versus asking for approval first?
How do you decide that one model's output is better than another's for a given task?
Describe a typical day in a role like this one: which parts run through AI directly?
If you removed AI from this role, what would be left, and how do you decide what still needs a human?

Adapt your resume

List these exact terms on your resume: Prompt Engineering, AI Agents, and AI Evaluation. An applicant tracking system matches the wording, not the idea.
Attach one line of real, concrete experience to at least one of them — a tool named with nothing behind it rarely survives a human read.
Show where AI is part of your daily process, not a one-off project — this role expects it to be a running habit.

Want your resume actually rewritten for this job?

The free preview above is everything we have today. A full resume rewrite is not live yet and has no price set. Join the waitlist and we will email you if we open it.

Get new AI jobs at AI Level 3+ by email

One email a week with the new AI jobs at AI Level 3+, each rated AI Level 1 to 4 for how much AI is in the work. No recruiter spam, unsubscribe in one click.

Also alert me for new jobs at Rootly

Free. One email a week. Unsubscribe in one click.