Principal Model Optimization Engineer

Every day, tens of millions of people come to Roblox to explore, create, play, learn, and connect with friends in 3D immersive digital experiences– all created by our global community of developers and creators.

At Roblox, we’re building the tools and platform that empower our community to bring any experience that they can imagine to life. Our vision is to reimagine the way people come together, from anywhere in the world, and on any device. We’re on a mission to connect a billion people with optimism and civility, and looking for amazing talent to help us get there.

A career at Roblox means you’ll be working to shape the future of human interaction, solving unique technical challenges at scale, and helping to create safer, more civil shared experiences for everyone.

ML Platform at Roblox supports hundreds of ML use cases and billions of inferences per day across Discovery, Safety, Engine and much more. We build what ML engineers use to go from idea to production: data and training pipelines, the SDKs and tools they build with, the GPU fleet and the scheduling that runs their jobs.

As a Principal Engineer on ML Platform, you will set the technical direction for these systems. You will find what slows ML engineers down or wastes compute and design durable fixes. You will measure the results and keep improving them until they show up as faster iteration, more reliable jobs and better use of every GPU.

You Will:

• Design platform capabilities that many teams adopt, so that one investment improves hundreds of use cases instead of one.
• Build GPU and compute management that keeps a scarce, mixed fleet busy. This covers quotas and priority tiers, preemption, scheduling across clusters, and capacity planning with Infrastructure.
• Define the platform’s success metrics, instrument them, and improve them quarter over quarter. Examples include time from code to first training run, job success rate, GPU utilization, and cost per training run or per inference.
• Lead deep investigations when jobs are slow or failing, from pipeline and scheduler behavior down to GPU profiles and hardware errors. Turn the findings into fixes that stay fixed.
• Work closely with ML engineers and data scientists to learn how they work, and make the paved path the easiest one to take.
• Partner with Infrastructure, Data Platform and Security on shared foundations such as storage, networking, data access and compliance.
• Raise the technical bar through design reviews, written proposals and code, and mentor senior engineers.

You Have:

• 6+ years of professional experience and a tool chest of system design experience upon which to draw to build performant systems for all of Roblox.
• A track record of building ML infrastructure end to end. Examples include training pipelines, workflow orchestration (e.g. Kubeflow, Airflow, Ray), data or feature pipelines, experiment tracking and model registries, or model serving.
• Experience designing SDKs, APIs or developer tools that were widely adopted, and a strong sense of what makes a platform easy to use.
• Hands-on experience running GPU workloads at scale on Kubernetes. That includes scheduling, quotas, multi-cluster dispatch, autoscaling, and debugging GPU and node failures (e.g. Xid errors, NCCL hangs).
• Enough depth in training and inference performance (distributed training, mixed precision, batching, quantization) to know where the platform should do the work for its users.
• A habit of measuring: you set clear success metrics, ship in steps, and keep iterating until the numbers move.
• Experience leading cross-team technical work without direct authority, and writing clearly for both engineers and leaders.
• A Bachelor’s degree in Computer Science, Computer Engineering or a related field, or equivalent experience.

For roles that are based at our headquarters in San Mateo, CA: The starting base pay for this position is as shown below. The actual base pay is dependent upon a variety of job-related factors such as professional background, training, work experience, location, business needs and market demand. Therefore, in some circumstances, the actual salary could fall outside of this expected range. This pay range is subject to change and may be modified in the future. All full-time employees are also eligible for equity compensation and for benefits as described on this page.

Roles that are based in an office are onsite Tuesday, Wednesday, and Thursday, with optional presence on Monday and Friday (unless otherwise noted).

Roblox provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws. Roblox also provides reasonable accommodations to candidates with qualifying disabilities or religious beliefs during the recruiting process.

For US based roles only, please note the Company may not be able to employ candidates for this role who have United States work authorization related to certain U.S. visa categories, or support future H-1B sponsorship at this time.