Senior Systems Software Engineer - Data Center Performance

NVIDIA has been transforming computer graphics, PC gaming, and accelerated computing for more than 25 years. It’s a unique legacy of innovation that’s fueled by great technology—and amazing people. Today, we’re tapping into the unlimited potential of AI to define the next era of computing. An era in which our GPU acts as the brains of computers, robots, and self-driving cars that can understand the world. Doing what’s never been done before takes vision, innovation, and the world’s best talent. As an NVIDIAN, you’ll be immersed in a diverse, supportive environment where everyone is inspired to do their best work. Come join the team and see how you can make a lasting impact on the world.

We are hiring a proficient engineer to join our Applied Systems Engineering Performance team. This role integrates comprehensive experimentation and software engineering to explore how emerging platforms respond under demanding HPC and AI workloads. You will reveal performance restrictions, establish trustworthy measurements, and provide collaborating teams with the insights they need to improve new platforms. Your work will connect innovations in individual technologies with the challenge of harmonizing their performance at datacenter scale.

What you'll be doing:

• Characterize node, rack, and cluster performance using microbenchmarks and representative applications. Measure throughput, latency, and scaling efficiency to identify compute, memory, communication, and storage bottlenecks.
• Identify Linux performance problems related to CPU scheduling, NUMA locality, CPU and memory affinity, interrupt handling, and resource contention. Assess their impact on GPU usage and application efficiency.
• Examine distributed workload scaling, considering collective communication, synchronization delays, slow nodes, and the interplay between computation and communication. Assess how GPU and NIC positioning, interconnect structure, and network traffic impact performance.
• Combine application profiles, system traces, hardware counters, and cluster telemetry to isolate root causes. Develop focused reproductions and collaborate with hardware, firmware, driver, library, and application teams to implement and verify fixes.
• Develop and maintain Python tools for automated benchmark execution, data collection, and performance analysis. Set up repeatable baselines, assess run-to-run variability, and spot regressions across hardware configurations and software releases.
• Define and complete performance validation for new platforms and cluster deployments. Identify underperforming nodes, links, or storage paths, and verify that corrective actions restore encouraged workload performance.

What we need to see:

• Extensive practical experience in Linux performance analysis, including knowledge of processes and threads, CPU and memory architecture, NUMA, I/O, and networking.
• Experience investigating performance in multi-node HPC or GPU-accelerated systems, distinguishing local resource bottlenecks from communication overhead, load imbalance, and shared-resource contention.
• Deep expertise in at least one relevant area: GPU application performance, Linux systems performance, high-speed networking and collective communication, parallel storage, or HPC performance tooling.
• Proficiency in Python to build maintainable automation and analysis tools. Experience in developing and debugging performance-sensitive code in C, C++ and CUDA.
• Ability to build controlled experiments, interpret profiling and counter data, and demonstrate reproducible improvements in application performance.
• Independently driving technical efforts across hardware and software boundaries, communicating findings clearly, and working with component owners to resolve issues.
• BS or MS in Computer Science, Engineering, Mathematics, Physics (or equivalent experience), with 8+ years of applicable experience.

Ways to stand out from the crowd:

• Experience working with profiling and tracing tools like Nsight Systems, Nsight Compute, Linux perf, or eBPF-based tools.
• Practical experience with RDMA and MPI or NCCL, and analyzing performance across PCIe, NVLink, InfiniBand, or Ethernet interconnects.
• Experience with Slurm, container runtimes, and cgroups, including understanding how workload placement, resource isolation, and CPU/GPU/NIC affinity impact performance.
• Experience improving distributed training, inference, or scientific applications by conducting strong- and weak-scaling studies, or building automated performance regression frameworks using tools such as ReFrame.