Senior Distributed Systems Engineer

AI in this role

pytorch

About the Institute of Foundation Models The Institute of Foundation Models (IFM) designs and operates ultra-scale GPU supercomputing systems to train next-generation foundation models. We believe performance, fault tolerance, and scalability are co-designed across model architecture, communication systems, runtime, and hardware topology. This role sits at the core of that effort — driving communication performance, distributed reliability, and cross-layer optimization for large-scale training workloads. The Mission We are looking for a deeply technical engineer to co-design and optimize the communication stack for large-scale distributed training, including hybrid parallelism and Mixture-of-Experts (MoE) workloads. This is not a network operations role. This is a systems-level engineering position focused on performance engineering, distributed debugging, and communication-runtime co-design. · Design and optimize expert-parallel and hybrid-parallel communication patterns · Drive high-performance hierarchical collectives for MoE workloads · Co-design runtime orchestration with communication topology awareness · Reduce tail latency and improve determinism across thousands of GPUs · Architect fault-tolerant distributed execution under real-world cluster failures Core Technical Scope · Communication-compute overlap and topology-aware collective optimization · Deep debugging of NCCL, RDMA, and custom communication layers · Hybrid expert parallel strategies in modern large-scale MoE systems · Elastic and resilient distributed job orchestration concepts · Congestion analysis and routing optimization across InfiniBand/RoCE fabrics · Microbenchmarking and performance modeling for communication-heavy workloads Expected Technical Depth · Hybrid expert parallel communication for Mixture-of-Experts training · Scaling behavior under network pressure · Distributed orchestration for elastic, large-scale training · Fault detection and recovery in distributed GPU workloads · Cross-layer bottlenecks: GPU ↔ NIC ↔ PCIe ↔ NVSwitch ↔ Fabric ↔ Scheduler Required Background · Experience optimizing distributed training at 1,000+ GPU scale (or equivalent depth) · Hands-on expertise with RDMA, InfiniBand, RoCE, and GPUDirect RDMA · Deep familiarity with NCCL and/or UCX internals · Strong systems programming ability (C/C++, Rust, or Go) · Strong familiarity with modern model training frameworks such as PyTorch · Ability to troubleshoot and profile training performance issues related to communication bottlenecks · Ability to translate research ideas into production-grade optimizations · Experience debugging distributed hangs, desynchronization, and performance regressions What We Mean by "Hardcore" · You can explain why an communication degrades at scale and how to fix it · You have improved real cluster throughput via communication redesign · You can trace a distributed hang across ranks and identify the root cause · You are comfortable working at the boundary between hardware and runtime Application Requirements · Include a link to your GitHub (required) · Provide links to relevant distributed systems, HPC, or large-scale training projects · Include a list of publications and/or public technical reports (if applicable) · Describe the hardest distributed debugging problem you solved · Include measurable performance improvements you have delivered Academic Qualifications Master’s, or Bachelor’s + 1 year of relevant experience.

How we rate this

Senior Distributed Systems Engineer at IFM rates 94 out of 100 for how much of the daily work is AI. That makes it Builds AI (AI Level 4 of 4). The level is about AI in the job, not seniority.

Classification

Builds AI. The job is building AI systems.

●●●● Builds AI80 to 100
●●●○ Works on AI60 to 79
●●○○ Uses AI40 to 59
●○○○ Little AI0 to 39

Levels come from how often the tools, models and workflows of the role are named in the posting itself. Open the description and count.

Prepare for this job

A free preview built only from this posting: what it asks for, what you could be asked in an interview, and how to adjust your resume.

Skills and AI tools this role asks for

PyTorch

Questions you could be asked

What's a project where you used PyTorch hands-on?
How would you decide a model or AI system is ready to ship?
Tell me about a time a model underperformed in production. How did you find out, and what did you change?

Adapt your resume

List these exact terms on your resume: PyTorch. An applicant tracking system matches the wording, not the idea.
Attach one line of real, concrete experience to at least one of them — a tool named with nothing behind it rarely survives a human read.
Lead with what you built, trained or shipped — this role is judged on the AI system itself, not the tools around it.

Want an expert to read your CV for this job?

Free. Send your CV and the role you want next. We reply by email within 2 to 4 business days.

Get a free CV review