Director, Clinical AI Research & Evaluation

Job Title: Director, Clinical AI Research & Evaluation

Introduction to role:

About AISI AI Science & Innovation (AISI) sits at the centre of AstraZeneca’s R&D AI transformation within Enterprise AI Unit (EAI). Our remit is to build, buy, and deliver the AI models and agents that change pipeline outcomes across discovery, translational science, biomarkers, and clinical development, ultimately improve patients’ lives.

The opportunity

We are building an end-to-end Enterprise AI engine that unites data foundations, AI models, platforms, and business-facing applications to accelerate results across the value chain. Success comes from reusing what already works, sharing ideas across teams, and scaling impact rather than building in isolation.

Role overview

AstraZeneca is building an AI capability for Clinical Development that will change how trials are designed, conducted, monitored, and analyzed. We are hiring a Director, Clinical AI Research & Evaluation to lead the scientific foundation that makes clinical AI systems trustworthy, measurable, and ready to scale into regulated practice. The role sits in the Clinical AI Applied Science & Translation team, based in Boston, MA. You will set the research agenda for how clinical AI systems are evaluated, how human professionals interact with AI in clinical workflows, and what evidence standards must exist before a system can be trusted at scale. This is a research leadership role with delivery accountability. You will define the methodology as we practice it, while remaining close enough to the work to credibly challenge it. You will direct a team of scientists and engineers, partner with product owners and Engineering, and establish evaluation and human-factors practices that extend across the Clinical Operations and Development AI portfolio. The initial focus for this role is an agentic system being built to reduce the administrative burden for research associates conducting trials at sites but with the expectation that patterns discovered and built here will scale to additional clinical AI system we develop along the way. The strongest candidates combine deep research expertise in AI evaluation or human computer interaction with the practical grounding to close the distance between principled methodology and working with and building regulated software.

Accountabilities:

Research Leadership

• Define and own the scientific methodology for evaluating clinical AI systems: evaluation harnesses, benchmark construction, failure mode taxonomy, red-team protocols, and monitoring strategies across the system lifecycle. Build a structured catalog of ways the system can fail clinically and run structured testing before any deployment.
• Advance methods for evaluation in clinically complex conditions such as noisy or delayed ground truth, expert disagreement, multi-step agentic workflows, human in-the-loop performance, and automation bias or where simple metrics fail.
• Develop human factors and workflow-integration methods beyond usability testing: how AI output shapes human judgment, what oversight mechanisms are effective, how clinical professionals calibrate trust in AI-assisted work, and what human review effectively ensure patient safety.
• Publish and communicate findings externally where appropriate to build AstraZeneca’s scientific credibility and attract research talent.

Applied Delivery

• Design and implement evaluation frameworks that determine when a system is ready to move from prototype to controlled pilot, from pilot to deployment, and from non-GxP to regulated use.
• Participating in defining human review requirements for user facing AI outputs: what requires sign-off before action, what can be surfaced directly, and how oversight should be structured in the validation plan.
• Build evaluation assets such as test sets, simulation environments, scoring rubrics, regression suites, and analysis pipelines, some or all of which are reusable across
• Lead hands-on failure analysis: fabrications/falsifications, unsupported claims, data leakage, subgroup performance, and unsafe workflow behavior.
• Be part of a team of scientists and engineers focused on evaluation, human factors, and responsible deployment.

Cross-Functional Governance

• Work with Quality, Regulatory, Privacy, Security, and R&D IT to ensure evaluation of evidence meets GxP, data integrity, and inspection-readiness requirements from the outset.
• Advise product owners and leadership on evidence-based go or no-go decisions; communicate residual risk clearly.
• Create reusable evaluation standards and clinical AI governance practices that can be adopted across AISI’s Clinical Development portfolio and contribute to broader industry standards.
• Represent the team’s scientific position in external forums, standards bodies, and regulatory engagements where relevant.

Essential Skills/Experience:

• PhD in Machine Learning, Computer Science, Human-Computer Interaction, Statistics, Biomedical Informatics, or a related quantitative discipline; or an MD or equivalent experience with a research record that demonstrates comparable rigor.
• 8 or more years of post-PhD (or equivalent) experience in AI/ML research, evaluation science, or human factors in healthcare, life sciences, clinical research, or another safety-critical domain.
• Deep expertise in evaluation methodology for AI/ML systems: experimental design, statistical inference, uncertainty quantification, measurement reliability, and evaluation of LLM-based or multi-step agentic workflows.
• Demonstrated research leadership: ability to set a scientific agenda, direct a team, produce research outputs, and translate principled methodology into practical, shipping systems.
• Current, hands-on coding ability in Python and SQL, including evaluation pipelines, data analysis, statistical testing, and version-controlled collaborative development.
• Strong grounding in human-computer interaction or human factors and its application to AI-assisted decision-making in high-stakes environments.
• Excellent communication skills across technical, clinical, quality, regulatory, and executive audiences; ability to represent uncertainty and residual risk without overstating or obscuring.
• Demonstrated ability to lead cross-functional work through influence, including go or no-go recommendations under ambiguity.
• Experience with LLM evaluation platforms, LLMOps/MLOps & observability for deployed AI systems.

Desirable Skills/Experience:

• Direct experience with clinical trial conduct, site monitoring, risk-based monitoring, medical monitoring, or clinical data management in a pharmaceutical, biotech, or CRO environment.
• Experience producing audit-ready evidence under GxP requirements, including model risk management, audit trails, traceability, and inspection readiness for regulated AI systems.
• Experience designing or conducting prospective validations, implementation studies and/or human-computer interaction studies in live clinical or healthcare workflows.
• Peer-reviewed publications, standards body contributions, open source benchmarks, or regulatory and industry consortium engagement in AI evaluation, human factors for AI, or clinical AI governance.
• Familiarity with causal inference, calibration, subgroup analysis, weak supervision, active learning, or evaluation methods for learning with imperfect labels.
• Experience working with Quality or regulatory functions on computerized system validation, validation plans, or submission-relevant AI evidence.

What success looks like

• The team establishes an evaluation standard for clinical AI that is recognized internally as the bar for production readiness and externally as a principled contribution to the field.
• Clinical experts, Quality partners, and regulatory functions trust that evaluation evidence reflects real clinical risk rather than optimistic benchmarks.
• Every agentic release by the team is traceable from intended use through test evidence to deployment decisions, with human oversight requirements defined, enforced, and captured in the validation plan.
• Demonstrated scientific leadership via publications, standards of contributions, or methodological advances.
• Evaluation of assets and human factors methods are reused across the Clinical Development AI portfolio without reinvention.

Office working requirements

When we put unexpected teams in the same room, we unleash bold thinking with the power to inspire life-changing medicines. In-person working gives us the platform we need to connect, work at pace, and challenge perceptions. That’s why we work, on average, a minimum of three days per week from the office. We balance this expectation with individual flexibility. Join us in our unique and ambitious world.

The annual base pay for this position ranges from $186,014 to $279,021 USD. Our positions may be eligible for various incentives, including short-term incentive bonuses, equity-based awards for salaried roles, and commissions for sales roles. Benefits may include qualified retirement programs, paid time off—including vacation, holidays, and leave—as well as health, dental, and vision coverage, in accordance with the terms of the applicable plans.

AstraZeneca is an equal opportunity employer committed to diversity and inclusion and to providing a workplace free from discrimination. AstraZeneca is committed to accommodating persons with disabilities. Accommodation is available upon request for all aspects of the recruitment, assessment, and selection process. Requests may be made by emailing AZCHumanResources@astrazeneca.com.

Why AstraZeneca:

Here your science meets scale. You will partner with diverse technologists and clinicians who bring cutting-edge data and engineering into the same room to unleash bold thinking and deliver real change for patients. We back ideas with investment, move fast by reusing what works, and build ecosystems with startups and leading academics to accelerate impact across the enterprise. You will be trusted to take ownership, experiment in hackathons, wrangle large and complex data, and turn principled methods into production systems that raise the bar for the industry. We value kindness alongside ambition, and we provide the coaching, feedback, and community you need to grow while making a visible difference.

Call to Action:

Lead the science that makes clinical AI trusted at scale—bring your agenda, your rigor, and your voice to shape how trials run and patients benefit.

#LI-Hybrid

Date Posted

Closing Date

Our mission is to build an inclusive environment where equal employment opportunities are available to all applicants and employees. In furtherance of that mission, we welcome and consider applications from all qualified candidates, regardless of their protected characteristics. If you have a disability or special need that requires accommodation, please complete the corresponding section in the application form.