Skip to content

Black Forest Labs

Research Lead, Evaluations

Black Forest Labs is hiring a Research Lead, Evaluations. Level rates it ; you can apply on Black Forest Labs.

Level's read on this job

  • On Level's scale, this role is Builds AI ●●●●: the job is building AI systems.
  • The posting mentions Stable Diffusion and AI research among its tools and skills.
  • Level classes it as a lead role.
  • Black Forest Labs has 15 other open roles on Level.
  • Posted in the last few days.

AI in this role

Stable Diffusion
ai-research

About Black Forest Labs

We’re the team building the foundational layer for visual intelligence: multimodal models trained jointly on images, video, audio, and action to see, generate, and understand the world.

Our founding team pioneered latent diffusion and Stable Diffusion, the breakthroughs that made generative AI accessible to millions. With FLUX, we're pushing into generative media at scale, machines that act in the physical world, and emerging applications in computer use.

Our FLUX models top public benchmarks and run in production in some of the world's biggest platforms. We're a small, fast growing research lab competing with the largest companies in the world. From our labs in Freiburg to San Francisco, we're building what comes next.

Why This Role

BFL’s research organization is split into different focus areas, such as pre-training, post-training, reinforcement learning, data, representation learning, evaluations and more. During the day to day work, researchers from these different focus areas work together to solve exciting and hard problems towards solving visual intelligence.

Having the right means to measure what matters is the central starting point for every research project. Researchers make decisions based on the outcomes of evaluations. The right decision can only be made if an evaluation is precise and accurate. As the Evaluations Lead you’re the core driver to ensure that. Central responsibilities include building our eval roadmap in collaboration with the research and product teams, managing and extending our evaluation team, and cross-functional work with project leads to make sure each research project has the best evaluation toolbox to help the people working on it to make the right decisions. Finally, you’re responsible for growing the people in your team and maintain an eye on their long term development during the daily project work.

What You’ll Work On

  • Defining what we evaluate: You are the core driver to create, maintain and extend our evaluations roadmap by collaborating with the product and other research leads who provide the required input.

  • Defining how we evaluate: You’ll work with the team to implement our evals roadmap. This includes building new evals. Both benchmarks and the necessary objective functions, involving both automated, AI-based, and human evaluations.

  • Minimizing noise in our eval pipeline: Developing frontier AI system requires the most accurate means to measure what matters! You will ensure that results are reproducible and trustworthy, that we understand the error bars behind them and that we measure what we want to measure.

  • Understanding what we cannot measure: You’re responsible for highlighting and clearly communicate limitations of each individual evaluation and you will support our project teams to ensure that their chosen set of evaluations has no blind spots in when combined. ****

  • Observing training runs and visualizing important findings: You will help with monitoring our most critical training runs from an independent standpoint to ensure that regressions and emerging capabilities are caught early and are clearly communicated.

  • Managing our evals team: Your role is a hands-on management role - you’re responsible for growing the team and the people in it. You will define our hiring roadmap and make sure that each team member can grow within BFL, both in terms of their technical skillset and their career.

What We’re Looking For

  • Proven eval experience: You’ve previously designed, run and maintained large and complex evaluation suites for frontier AI systems, involving various kinds of evaluation methods involving both human and automated (both with or without AI) approaches.

  • Strong research background: You should have proven experience of designing and implementing evaluations for AI research. Strong understanding of Python is required. A strong understanding of agentic coding is a must.

  • Experience as a people manager: One of your core responsibilities will be to grow the team and its members. Understanding what it means to lead a team and how it’s different from an IC role is a must. You should have actively managed a team for at least one year.

  • Strong understanding in statistics and good judgement about numbers: You should be familiar with variance, confidence intervals and significance testing. You should be able to identify and resolve situations when blindly trusting numbers leads to wrong conclusions.

  • Excellent communication skills: You will have to communicate results with project leads and research leadership. Understanding how to communicate these results and what they don’t tell us is important since they’re the foundation for important decisions.

  • First principles thinking: Crafting evaluations for new capabilities is not straightforward. You need to be able to derive new evals which requires proficiency in thinking from first principles

How We Work Together

We’re a distributed team with real offices that people actually use. This role is based in Freiburg, and you’ll join us in the office at least 2 days a week, or one full week every other week.

Everything we do is grounded in four values:

  • Obsessed. We are a frontier research lab. The science has to be right, the understanding deep, the product beautiful.

  • Low Ego. The work speaks. The best idea wins, no matter who said it. Credit is shared. Nobody is above any task.

  • Bold. We take the ambitious bet. We ship, we do not wait for conditions to be perfect.

  • Kind. People over politics. We treat each other with genuine warmth. Agency without empathy creates chaos.

If this sounds like work you’d enjoy, we’d love to hear from you.

Compensation

EUR255K- 295K + equity, depending on profile

Note: We may use AI-assisted tools to help organize applications. All hiring decisions are always made by our team.

How we rate this

Research Lead, Evaluations at Black Forest Labs rates 85 out of 100 for how much of the daily work is AI. That makes it Builds AI (AI Level 4 of 4). The level is about AI in the job, not seniority.

Classification

Builds AI. The job is building AI systems.

  1. ●●●● Builds AI80 to 100
  2. ●●●○ Works on AI60 to 79
  3. ●●○○ Uses AI40 to 59
  4. ●○○○ Little AI0 to 39

Levels come from how often the tools, models and workflows of the role are named in the posting itself. Read the description and count the AI tools, models and workflows it names.

Prepare for this job

Built from this posting and from similar jobs on Level.

Skills and AI tools this role asks for

AI ResearchStable Diffusion

Questions you could be asked

  1. Tell me about a research question you investigated. What did you find?
  2. Walk me through how you've used Stable Diffusion in your day-to-day work.
  3. How would you decide a model or AI system is ready to ship?
  4. Tell me about a time a model underperformed in production. How did you find out, and what did you change?

Adapt your resume

  • List these exact terms on your resume: AI Research and Stable Diffusion. An applicant tracking system matches the wording, not the idea.
  • Attach one line of real, concrete experience to at least one of them. A tool named with nothing behind it rarely survives a human read.
  • Lead with what you built, trained or shipped. This role is judged on the AI system itself, not the tools around it.

Get your CV reviewed before you apply

Send your CV and the role you want next. Your review arrives by email within 3 days.

Get new research jobs (Builds AI ●●●●) by email

One email a week, each job rated for how much AI is in the work. No recruiter spam, unsubscribe in one click.

Free.

Similar roles

Research roles that build AI, at other companies.

What kind of AI work fits you?

Answer 12 practical questions in about three minutes. Get a simple profile, the work it points to, and live roles to explore next.

Find my next step

More jobs at Black Forest Labs

Related searches

Same AI level