Distributed LLM Inference Engineer

Anyscale is hiring a Distributed LLM Inference Engineer for a remote role open to applicants in United States. It pays $170k-$245k a year and Level rates it ; you can apply on Anyscale.

Level's read on this job

Level rates this role Builds AI ●●●●: the job is building AI systems.
The posting lists this role as remote.
Tools and skills the posting cites include OpenAI, vLLM and PyTorch.
Level classifies it as a senior role.
Level lists 20 other open roles at Anyscale.

AI in this role

Distributed LLM Inference Engineer to build and optimize large-scale machine learning inference infrastructure using Ray and vLLM.

openaivllmpytorchtensorflowtensorrt-llmtritonraycuda

distributed-systemsml-inferencedeep-learninggpu-optimization

About Anyscale

At Anyscale, we're on a mission to democratize distributed computing and make it accessible to software developers of all skill levels. We’re commercializing Ray, a popular open-source project that's creating an ecosystem of libraries for scalable machine learning. Companies like OpenAI, Uber, Spotify, Instacart, Cruise, and many more, have Ray in their tech stacks to accelerate the progress of AI applications out into the real world.

With Anyscale, we’re building the best place to run Ray, so that any developer or data scientist can scale an ML application from their laptop to the cluster without needing to be a distributed systems expert.

Proud to be backed by Andreessen Horowitz, NEA, and Addition with $250+ million raised to date.

About the role

As a Distributed LLM Inference Engineer, you will help systems and optimizations that push the boundaries of performance for inference at large scale. This is an incredibly critical role to Anyscale as it allows us to achieve a market leading position for AI infrastructure.

As part of this role, you will

Iterate very quickly with product teams to ship the end to end solutions for Batch and Online inference at high scale which will be used by open-source Ray users and customers of Anyscale

Work across the stack integrating Ray Data and LLM engine providing optimizations achieving low cost solutions for large scale ML inference
Integrate with Open source software like vLLM, work closely with the community to adopt these techniques in Anyscale solutions, and also contribute improvements to open source
Follow the latest state-of-the-art in the open source and the research community, implementing and extending best practices

We'd love to hear from you if you have

Familiarity with running ML inference at large scale with high throughput and low latency
Familiarity with deep learning and deep learning frameworks (e.g. PyTorch)
Solid understanding of distributed systems, ML inference challenges

Bonus points!

ML Systems knowledge

Experience using Ray
Work closely with community on LLM engines like vLLM, TensorRT-LLM
Contributions to deep learning frameworks (PyTorch, TensorFlow)
Contributions to deep learning compilers (Triton, TVM, MLIR)
Prior experience working on GPUs / CUDA

Compensation

At Anyscale, we take a market-based approach to compensation. We are data-driven, transparent, and consistent. As the market data changes over time, the target salary for this role may be adjusted.

This role is also eligible to participate in Anyscale's Equity and Benefits offerings, including the following:

Stock Options
Healthcare plans, with premiums covered by Anyscale at 99% for both employees and dependents
401k Retirement Plan
Education & Wellbeing Stipend
Paid Parental Leave
Fertility Benefits
Paid Time Off
Commute reimbursement
100% of in-office meals covered

Anyscale Inc. is an Equal Opportunity Employer. Candidates are evaluated without regard to age, race, color, religion, sex, disability, national origin, sexual orientation, veteran status, or any other characteristic protected by federal or state law.

Anyscale Inc. is an E-Verify company and you may review the Notice of E-Verify Participation and the Right to Work posters in English and Spanish

How we rate this

Distributed LLM Inference Engineer at Anyscale rates 95 out of 100 for how much of the daily work is AI. That makes it Builds AI (AI Level 4 of 4). The level is about AI in the job, not seniority.

Classification

Builds AI. The job is building AI systems.

●●●● Builds AI80 to 100
●●●○ Works on AI60 to 79
●●○○ Uses AI40 to 59
●○○○ Little AI0 to 39

Levels come from how often the tools, models and workflows of the role are named in the posting itself. Read the description and count the AI tools, models and workflows it names.

Key facts

Distributed LLM Inference Engineer is an open job at Anyscale.
Level rates this role AI Level 4 of 4: Builds AI.
The listed salary is $170k-$245k a year.
The role is based in San Francisco and is marked remote.
The job was posted on May 27, 2026.
Seniority: senior.
Applications are handled by Anyscale. Apply on Anyscale

Prepare for this job

Built from this posting and from similar jobs on Level.

Skills and AI tools this role asks for

Distributed SystemsML InferenceDeep learningGPU OptimizationOpenAIvLLMPyTorchTensorFlow

Questions you could be asked

Tell me about a project where distributed systems was part of your work. What did you do?
Tell me about a project where ml inference was part of your work. What did you do?
Tell me about a project where deep learning was part of your work. What did you do?
Tell me about a project where gpu optimization was part of your work. What did you do?
Walk me through how you've used OpenAI in your day-to-day work.

Adapt your resume

List these exact terms on your resume: Distributed Systems, ML Inference, Deep learning, GPU Optimization, and OpenAI. An applicant tracking system matches the wording, not the idea.
Attach one line of real, concrete experience to at least one of them. A tool named with nothing behind it rarely survives a human read.
Lead with what you built, trained or shipped. This role is judged on the AI system itself, not the tools around it.

Want an expert to read your CV for this job?

Free. Send your CV and the role you want next. We reply by email within 2 to 4 business days.

Get a free CV review Get an express CV review Check your resume with the ATS resume checker

Get new remote software engineering jobs (Builds AI ●●●●) by email

One email a week with the new remote software engineering jobs (Builds AI ●●●●), each rated for how much AI is in the work. No recruiter spam, unsubscribe in one click.

Free. One email a week. Unsubscribe in one click.