# Research Engineer, Infrastructure, Training Systems at Thinking Machines Lab

Thinking Machines Lab is hiring a Research Engineer, Infrastructure, Training Systems in San Francisco, United States. It pays $400k-$500k a year and Level rates it Builds AI ●●●●; you can [apply on Thinking Machines Lab](https://jobsbylevel.com/go/ed280f12-8f65-4e0f-bb81-32d1d081b883?utm_medium=ai-agent).

AI Level 4, AI centrality 98 out of 100. San Francisco.

## Details

- Company: [Thinking Machines Lab](https://jobsbylevel.com/companies/thinking-machines-lab)
- AI level: AI Level 4 (score 98 out of 100)
- Location: San Francisco
- Salary: $400k-$500k
- Posted: August 4, 2026
- Apply: https://jobsbylevel.com/go/ed280f12-8f65-4e0f-bb81-32d1d081b883?utm_medium=ai-agent
- Free CV review for this job: https://jobsbylevel.com/cv-review?job=research-engineer-infrastructure-training-systems-at-thinking-machines-lab-579533

## Prepare for this job

The job page lists 4 likely interview questions for this offer, with a CV checklist: https://jobsbylevel.com/jobs/research-engineer-infrastructure-training-systems-at-thinking-machines-lab-579533

## Description

About Thinking Machines The mission of Thinking Machines is to build AI that extends human will and judgment. We are training frontier models with Inkling, developing Tinker to let people make models their own, and crafting interfaces that broaden human-AI communication. We believe the future worth building is human, and we're hiring people who want to build it. About the Role Frontier model training depends on systems that can keep large accelerator clusters working efficiently and reliably through runs lasting days or weeks. We’re looking for engineers to build and improve those systems, working closely with researchers on the infrastructure behind large-scale pre-training and mid-training. The work spans the training stack: dividing models across accelerators, improving numerical and hardware efficiency, coordinating distributed workloads, and recovering from failures without losing valuable training time. You might optimize FP4/FP8 training, implement a parallelism strategy, remove a communication bottleneck, or make a large training workload more reliable. We’re especially interested in two complementary areas: Distributed Training: Build and optimize the frameworks, runtimes and workflows that make large-scale model training possible. Relevant experience includes FSDP, sharded model and optimizer states, expert or context parallelism, distributed checkpointing, training-job scheduling, and improvements to model FLOPs utilization. Communication / Collectives: Build and optimize the communication systems that connect accelerators during training. Relevant experience includes NCCL or equivalent runtimes, collective algorithms, topology-aware communication, communication/computation overlap, and fault-tolerant training. You do not need to specialize in both areas. We value depth in a difficult subsystem, the ability to reason across the training stack, and a track record of turning that understanding into working systems. Our training systems stack is written in Rust. Prior Rust experience is helpful, but strong systems engineers who are interested in learning it are encouraged to apply. What You’ll Do Design, implement and operate distributed training infrastructure that scales across thousands of GPUs and nodes. Improve training throughput, accelerator utilization and time to train by diagnosing bottlenecks across compute, memory, networking and storage. Develop parallelism strategies, runtime components, communication primitives or workload orchestration that support evolving model architectures. Make long-running training jobs more reliable through better observability, checkpointing, failure diagnosis and recovery. Work closely with researchers to understand new workloads, test systems changes and measure their effect on training performance and correctness. Build reusable, maintainable frameworks and libraries that make experimentation faster and training more reproducible. Share useful findings through documentation, open-source contributions or technical reports. Your focus will depend on your strengths and the problems the team needs to solve. What We’re Looking For Strong engineering skills: you write performant, maintainable code and can debug complex systems. Hands-on experience building, optimizing or operating distributed model-training infrastructure, with a clear understanding of your personal contributions. Depth in at least one relevant area, such as distributed training frameworks, parallelism, GPU communication, low-precision training, runtime systems or training-workload orchestration. Understanding of deep learning frameworks such as PyTorch or JAX and the systems beneath them. The ability to investigate problems across subsystem boundaries and independently carry a solution through implementation and validation. A collaborative approach to working with researchers and engineers, including clear communication about evidence, uncertainty and tradeoffs. A bachelor’s degree in a relevant technical field

The description is cut here. Read the full offer: https://jobsbylevel.com/jobs/research-engineer-infrastructure-training-systems-at-thinking-machines-lab-579533

Source: https://jobsbylevel.com/jobs/research-engineer-infrastructure-training-systems-at-thinking-machines-lab-579533

## Cite this page

Level. https://jobsbylevel.com/jobs/research-engineer-infrastructure-training-systems-at-thinking-machines-lab-579533.

Get job alerts: https://jobsbylevel.com/newsletter