Principal, Software Engineer (L5)
AI in this role
Design and scale carrier test platforms and production LLM agents for automated telecommunications troubleshooting.
Who we are
At Twilio, we’re shaping the future of communications, all from the comfort of our homes. We deliver innovative solutions to hundreds of thousands of businesses and empower millions of developers worldwide to craft personalized customer experiences.
Our dedication to remote-first work, and strong culture of connection and global inclusion means that no matter your location, you’re part of a vibrant team with diverse experiences making a global impact each day. As we continue to revolutionize how the world interacts, we’re acquiring new skills and experiences that make work feel truly rewarding. Your career at Twilio is in your hands.
.
Hiring and how we work
We use Artificial Intelligence (AI) to help make our hiring process efficient. That said, every hiring decision is made by real Twilions!
Also, while we are a remote-first company, you may be asked to report in person on an ad-hoc basis for team gatherings, functional off-sites or customer meetings.
.
See yourself at Twilio
Join the team as Twilio’s next Principal Software Engineer.
About the job
This position is needed to raise the technical depth of the systems that continuously test, measure, and assure the quality of Twilio's global carrier connections, and the operational platforms that Super Network's operations depend on.
Every call and message Twilio makes crosses a carrier network we don't control. This team is how we know those connections are healthy. We monitor and test them continuously, worldwide, for both voice and messaging reliability, and we build what our operations teams use to act on what we find. Increasingly that means LLM agents troubleshooting carrier problems without a human involved — and that's the part we most want to push further.
Responsibilities
In this role, you’ll:
- Design and deliver major components of Twilio's carrier test and observability platform, covering active voice and messaging testing, quality scoring, and global test coverage.
- Build high-throughput data and alerting systems that turn the data into signals operations teams can act on without drowning in noise.
- Extend and harden our production LLM systems for automated carrier troubleshooting — expanding the classes of issue they resolve without human intervention, and owning the evaluation, observability, and guardrails that make autonomous action safe.
- Diagnose cross-boundary failures where test results, carrier behavior, and platform telemetry disagree, and establish the observability needed to tell those cases apart.
- Contribute to technical design and review across the team, mentor engineers, and help shape written standards that outlast any single project.
Qualifications
Twilio values diverse experiences from all kinds of industries, and we encourage everyone who meets the required qualifications to apply. If your career is just starting or hasn't followed a traditional path, don't let that stop you from considering Twilio. We are always looking for people who will bring something new to the table!
*Required:
- 12+ years of software engineering experience, including significant time at staff or principal level working on systems at large scale.
- Deep expertise designing, operating, and debugging distributed systems that ingest and process high-volume time-series or event data in production, including alerting, anomaly detection, and failure isolation.
- Hands-on experience taking LLM-based systems to production and keeping them there: context and tool design, evaluation against real outcomes, cost and latency management, and the guardrails required when model output triggers real actions rather than just producing text.
- Strong proficiency in at least one backend language used at scale (Java, Go, Scala, or similar) and fluency with event streaming and cloud infrastructure (Kafka, AWS, Kubernetes).
- Ability to communicate complex technical trade-offs clearly in writing and to influence peers through design review rather than authority.
Desired:
- Experience with telecommunications, messaging, or voice systems
- Experience with active network testing, monitoring, or quality-of-service measurement.
- Experience with agentic systems that take action on live infrastructure, and with the evaluation practices that make that trustworthy.
Location
This role will be remote, and based in Ireland.
Travel
We prioritize connection and opportunities to build relationships with our customers and each other. For this role, you may be required to travel occasionally to participate in project or team in-person meetings.
What We Offer
Working at Twilio offers many benefits, including competitive pay, generous time off, ample parental and wellness leave, healthcare, a retirement savings program, and much more. Offerings vary by location.
Twilio thinks big. Do you?
We like to solve problems, take initiative, pitch in when needed, and are always up for trying new things. That's why we seek out colleagues who embody our values — something we call Twilio Magic. Additionally, we empower employees to build positive change in their communities by supporting their volunteering and donation efforts.
So, if you're ready to unleash your full potential, do your best work, and be the best version of yourself, apply now! If this role isn't what you're looking for, please consider other open positions.
.
Stay alert to recruitment fraud
We care about your safety. Scammers sometimes impersonate Twilio recruiters through fake job postings, emails, websites, or messages. Please ensure you are engaging with an official @twilio.com email address. We will never ask for payment, gift cards, cryptocurrency, or banking information during the recruiting process. We do not make job offers without a formal interview process or conduct interviews exclusively through text-based messaging apps.
.
Twilio is proud to be an equal opportunity employer. We do not discriminate based upon race, religion, color, national origin, sex (including pregnancy, childbirth, reproductive health decisions, or related medical conditions), sexual orientation, gender identity, gender expression, age, status as a protected veteran, status as an individual with a disability, genetic information, political views or activity, or other applicable legally protected characteristics. We also consider qualified applicants with criminal histories, consistent with applicable federal, state and local law. Qualified applicants with arrest or conviction records will be considered for employment in accordance with the Los Angeles County Fair Chance Ordinance for Employers and the California Fair Chance Act. Additionally, Twilio participates in the E-Verify program in certain locations, as required by law.
How we rate this
Principal, Software Engineer (L5) at Twilio rates 85 out of 100 for how much of the daily work is AI. That makes it Builds AI (AI Level 4 of 4). The level is about AI in the job, not seniority.
Builds AI. The job is building AI systems.
- ●●●● Builds AI80 to 100
- ●●●○ Works on AI60 to 79
- ●●○○ Uses AI40 to 59
- ●○○○ Little AI0 to 39
Levels come from how often the tools, models and workflows of the role are named in the posting itself. Open the description and count.
Prepare for this job
A free preview built only from this posting: what it asks for, what you could be asked in an interview, and how to adjust your resume.
Skills and AI tools this role asks for
Questions you could be asked
- How do you decide when an AI agent can act on its own versus asking for approval first?
- Tell me about a project where software engineering was part of your work. What did you do?
- Tell me about a project where distributed systems was part of your work. What did you do?
- Tell me about a project where observability was part of your work. What did you do?
- Tell me about a project where llm was part of your work. What did you do?
Adapt your resume
- List these exact terms on your resume: AI Agents, Software Engineering, Distributed Systems, Observability, and LLM. An applicant tracking system matches the wording, not the idea.
- Attach one line of real, concrete experience to at least one of them — a tool named with nothing behind it rarely survives a human read.
- Lead with what you built, trained or shipped — this role is judged on the AI system itself, not the tools around it.
Want your resume actually rewritten for this job?
The free preview above is everything we have today. A full resume rewrite is not live yet and has no price set. Join the waitlist and we will email you if we open it.
Get new remote AI jobs (Builds AI ●●●●) by email
One email a week with the new remote AI jobs (Builds AI ●●●●), each rated for how much AI is in the work. No recruiter spam, unsubscribe in one click.
Free. One email a week. Unsubscribe in one click.
Similar roles
Software Engineering roles that build AI, at other companies.
What kind of AI work fits you?
Answer 12 practical questions in about three minutes. Get a simple profile, the work it points to, and live roles to explore next.
Find my next step