Senior or Staff Software Engineer, Inference / Compute Infrastructure Engineering

Together AI is hiring a Senior or Staff Software Engineer, Inference / Compute Infrastructure Engineering. Level rates it ; you can apply on Together AI.

Level's read on this job

Builds AI ●●●● is how Level rates this software engineer role: the job is building AI systems.
The posting mentions Cursor, Elevenlabs and Decagon among its tools and skills.
Level classes it as a lead role.
Together AI has 83 other open roles on Level.

AI in this role

cursorelevenlabsdecagon

fine-tuning

About the Role

REMOTE IN INDIA

We're looking for a software engineer to build the Kubernetes-native control plane that provisions and runs our GPU inference fleet. You'll design a manifest-driven API where the inference team declares what they need, whether that's a cluster, a model deployment, or a capacity change, and our controllers handle the reconciliation, provider/runtime selection, and lifecycle management underneath, so the inference team never has to know or care which specific serving stack, scheduler, or hardware pool is doing the work. You'll also build the systems that keep the fleet efficient, not just running, including defragmentation and rebalancing logic that consolidates scattered workloads back into contiguous capacity, and scheduling/bin-packing improvements that push GPU utilization up without hurting latency. The core value we're after is decoupling the people building on top of the platform from the operational and runtime complexity underneath, while squeezing more usable capacity out of the same hardware. You'll build the controllers, reconciliation loops, and self-service surface (API/CLI, not tickets) that make that decoupling real, plus the event-driven health, remediation, and utilization systems that keep it running and efficient without a human in the loop. Strong candidates have hands-on experience with Kubernetes controller/CRD patterns, have built or operated a platform API that abstracts multiple backends behind one interface, understand GPU scheduling and capacity efficiency (fragmentation, bin-packing, right-sizing), and think about GPU infrastructure as software to be engineered.

A product mindset - you've built internal platforms or APIs consumed by other engineering teams and care about the developer experience of what you ship.You build it, you own it. You are not only responsible for delivering the software but also for operating and supporting it in production.

Responsibilities

Build the provisioning state machine: design and implement the software that models the full lifecycle of a physical host from discovery, inference bring-up to GPU driver/CUDA stack, health validation, and decommission/RMA — as explicit, versioned states and transitions.
Build the self-service API: design declarative APIs and a control plane so the inference team can request, scale, and tear down inference clusters with one API call — no ticket, no human in the loop.
Automate self-healing: detect degraded or failed nodes, drain them safely, trigger repair or replacement, and reintroduce healthy capacity into the pool automatically.
Own reliability of the pipeline: idempotency, retries, rollback, and drift detection so the provisioning system is as dependable as any other production service.
Partner with the inference/ML platform team: understand the cluster shapes they need — topology, interconnect, scheduling constraints — and encode them as first-class abstractions in the platform.
Engineer it like software: strong typing, automated tests, code review, versioning, and CI/CD for infrastructure code — this is a product, not a collection of Ansible playbooks.

Requirements

Core requirements (all levels):

Strong software engineering background in Go, Python, Rust, or similar — you write and test real software for a living.
Experience with durable workflow orchestration tools such as Temporal, Cadence, or equivalent to run long-lived, manifest-driven workflows that survive failures and resume mid-execution.
Experience building software control planes or orchestration systems that model state and reconcile it over time (e.g., Kubernetes controllers/operators, custom reconciliation loops, workflow engines).
Experience with event-driven systems — designing and building software around message queues, event streams, or pub/sub (e.g., Kafka, NATS, SQS) rather than polling or cron-driven scripts.
A product mindset. You’ve built internal platforms or APIs consumed by other engineering teams and care about the developer experience of what you ship.

Nice to have:

Exposure to bare-metal provisioning (PXE/iPXE, Redfish/IPMI, BMC) and/or networking fundamentals (VLANs, BGP, fabric design), or GPU/accelerator infrastructure.
Experience with GPU cluster software stacks (NCCL, CUDA, InfiniBand/RoCE).
Prior work at a hyperscaler, GPU cloud, or datacenter-scale infrastructure organization.
Systems programming in Rust or Go.

About Together AI

Together AI, the AI Native Cloud, is purpose-built for AI engineers. AI application developers get high-performance inference that scales reliably, fine-tuning and reinforcement learning for creating frontier-level specialized models, and pre-training at massive scale for fully custom intelligence, all around a marketplace of leading open models that teams can run, adapt, and own. Trusted by Cursor, Decagon, ElevenLabs, Salesforce, and Zoom, Together serves 400+ trillion tokens a month.

Equal Opportunity

Together AI is an Equal Opportunity Employer and is proud to offer equal employment opportunity to everyone regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity, veteran status, and more.

Please see our privacy policy at https://www.together.ai/privacy.

How we rate this

Senior or Staff Software Engineer, Inference / Compute Infrastructure Engineering at Together AI rates 98 out of 100 for how much of the daily work is AI. That makes it Builds AI (AI Level 4 of 4). The level is about AI in the job, not seniority.

Classification

Builds AI. The job is building AI systems.

●●●● Builds AI80 to 100
●●●○ Works on AI60 to 79
●●○○ Uses AI40 to 59
●○○○ Little AI0 to 39

Levels come from how often the tools, models and workflows of the role are named in the posting itself. Read the description and count the AI tools, models and workflows it names.

Key facts

Senior or Staff Software Engineer, Inference / Compute Infrastructure Engineering is an open job at Together AI.
Level rates this role AI Level 4 of 4: Builds AI.
The role is based in India.
The job was posted on August 19, 2026.
Seniority: lead.
Applications are handled by Together AI. Apply on Together AI

Prepare for this job

Built from this posting and from similar jobs on Level.

Skills and AI tools this role asks for

Fine-tuningCursorElevenlabsDecagon

Also common in software engineer jobs at this level

LLM AWS Machine learning C++AI agents Java

Pay

Typical software engineer job: $168k

This posting doesn't list pay. Use the typical software engineer figure when pay comes up. See software engineer jobs

Questions you could be asked

Walk me through fine-tuning a model: what data did you use, and how did you check the result?
Walk me through how you've used Cursor in your day-to-day work.
What are the limits of Elevenlabs that you've run into, and how did you work around them?
What's a project where you used Decagon hands-on?
How would you decide a model or AI system is ready to ship?

Adapt your resume

List these exact terms on your resume: Fine-tuning, Cursor, Elevenlabs, and Decagon. An applicant tracking system matches the wording, not the idea.
Attach one line of real, concrete experience to at least one of them. A tool named with nothing behind it rarely survives a human read.
Lead with what you built, trained or shipped. This role is judged on the AI system itself, not the tools around it.

Want an expert to read your CV for this job?

Free. Send your CV and the role you want next. We reply by email within 2 to 4 business days.

Get a free CV review Get an express CV review Check your resume with the ATS resume checker

Get new software engineer jobs (Builds AI ●●●●) by email

One email a week with the new software engineer jobs (Builds AI ●●●●), each rated for how much AI is in the work. No recruiter spam, unsubscribe in one click.

Free. One email a week. Unsubscribe in one click.