Applied Science Manager, Artificial General Intelligence , Frontier AI Assets: Assessments
AI in this role
Key job responsibilities
As part of the AGI team, an Applied Science Manager will lead and mentor a team of Applied Scientists who develop comprehensive quality strategies and auditing frameworks that safeguard the integrity of data collection workflows. The manager will guide the team in designing auditing strategies with detailed SOPs, quality metrics, and sampling methodologies that align with core scientist team developing Amazon Nova models. The Applied Science Manager will oversee expert-level manual audits, meta-audits to evaluate auditor performance, and provide coaching to uplift overall quality capabilities across the team. A critical aspect of this role involves managing the development and maintenance of LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment. The Applied Science Manager will also oversee the configuration of data collection workflows and ensure effective communication of quality feedback to stakeholders. The manager will have a direct impact on enhancing customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services.
The Applied Science Manager will be responsible for recruiting, hiring, and developing team members, conducting performance reviews, setting clear expectations and growth plans, and fostering a culture of scientific excellence and innovation. The manager will communicate with senior leadership, cross-functional technical teams, and customers to collect requirements, describe product features and technical designs, and articulate product strategy.
A day in the life
An Applied Science Manager with the AGI team will lead quality solution design, guide root cause analysis on data quality issues, drive research into new auditing methodologies, and find innovative ways of optimizing data quality while setting examples for the team on quality assurance best practices and standards. The manager will work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice. The manager will also conduct regular 1:1s with team members, provide mentorship and coaching, and ensure the team delivers high-impact results aligned with organizational goals.
Basic qualifications
- 3+ years of scientists or machine learning engineers management experience
- Knowledge of ML, NLP, Information Retrieval and Analytics
Preferred qualifications
- Experience building machine learning models or developing algorithms for business application
- Experience building complex software systems, especially involving deep learning, machine learning and computer vision, that have been successfully delivered to customers
Amazon is an equal opportunity employer and does not discriminate on the basis of protected veteran status, disability, or other legally protected status.
Our inclusive culture empowers Amazonians to deliver the best results for our customers. If you have a disability and need a workplace accommodation or adjustment during the application and hiring process, including support for the interview or onboarding process, please visit https://amazon.jobs/content/en/how-we-hire/accommodations for more information. If the country/region you’re applying in isn’t listed, please contact your Recruiting Partner.
The base salary range for this position is listed below. Your Amazon package will include sign-on payments and restricted stock units (RSUs). Final compensation will be determined based on factors including experience, qualifications, and location. Amazon also offers comprehensive benefits including health insurance (medical, dental, vision, prescription, Basic Life & AD&D insurance and option for Supplemental life plans, EAP, Mental Health Support, Medical Advice Line, Flexible Spending Accounts, Adoption and Surrogacy Reimbursement coverage), 401(k) matching, paid time off, and parental leave. Learn more about our benefits at https://amazon.jobs/en/benefits.
USA, MA, Boston - 183,800.00 - 248,700.00 USD annually
USA, WA, BELLEVUE - 183,800.00 - 248,700.00 USD annually
How we score this
Applied Science Manager, Artificial General Intelligence , Frontier AI Assets: Assessments at Amazon scores 84 out of 100 on AI centrality, which makes it AI Level 4 of 4 (Builds AI) on this board. The level measures how much of the work is AI, not seniority.
AI Level 4. Building AI systems is the job itself: without AI, the role would not exist.
- AI Level 480 to 100
- AI Level 360 to 79
- AI Level 240 to 59
- AI Level 10 to 39
Bands come from how often the tools, models and workflows of the role are named in the posting itself. Open the description and count.
Prepare for this job
A free preview built only from this posting: what it asks for, what you could be asked in an interview, and how to adjust your resume.
Skills and AI tools this role asks for
Questions you could be asked
- Walk me through a computer vision problem you solved, from raw data to a deployed model.
- What NLP problem have you worked on, and how did you measure whether it actually worked?
- How would you decide a model or AI system is ready to ship?
- Tell me about a time a model underperformed in production. How did you find out, and what did you change?
Adapt your resume
- List these exact terms on your resume: Computer Vision and Nlp. An applicant tracking system matches the wording, not the idea.
- Attach one line of real, concrete experience to at least one of them — a tool named with nothing behind it rarely survives a human read.
- Lead with what you built, trained or shipped — this role is judged on the AI system itself, not the tools around it.
Want your resume actually rewritten for this job?
The free preview above is everything we have today. A full resume rewrite is not live yet and has no price set. Join the waitlist and we will email you if we open it.
Get new AI jobs at AI Level 4+ by email
One email a week with the new AI jobs at AI Level 4+, each rated AI Level 1 to 4 for how much AI is in the work. No recruiter spam, unsubscribe in one click.
Free. One email a week. Unsubscribe in one click.
Similar roles
Other roles rated AI Level 4 at other companies.
What kind of AI work fits you?
Answer 12 practical questions in about three minutes. Get a simple profile, the work it points to, and live roles to explore next.
Find my next step