Member of Technical Staff (Index Engineer, Search Crawler)
Perplexity is hiring a Member of Technical Staff (Index Engineer, Search Crawler) in London, United Kingdom. Level rates it ; you can apply on Perplexity.
Level's read on this job
- Builds AI ●●●● is how Level rates this role: the job is building AI systems.
- Level classes it as a lead role.
- Perplexity has 143 other open roles on Level.
- Posted in the last few days.
AI in this role
AI answers are only as trustworthy as the information behind them. Perplexity grounds its answers in fresh, real-world sources, and the index is what makes that possible. We're looking for an engineer to work on the indexing and data processing layer of Perplexity's Search Crawler. You'll build systems that ingest, process, and organize large volumes of crawled content into efficient, queryable structures, working primarily on offline and nearline pipelines that keep data fresh, complete, and accessible to downstream retrieval. Throughput, scalability, and correctness are critical across datasets that change continuously.
Responsibilities
Design, implement, and operate large-scale indexing systems and data pipelines at the core of Perplexity's search infrastructure.
Develop and optimize indexing strategies that balance performance, freshness, and resource efficiency.
Work on storage formats, compaction, and update mechanisms that keep data accessible and current.
Keep pipelines reliable and predictable under high-throughput conditions.
Build well-tested components with clear responsibilities and interface contracts that stay flexible as the system evolves.
Define and implement observability across offline and near line pipelines, including structured logs, metrics, and data quality signals.
Monitor throughput, resource usage, and cost, and drive optimisations as business needs change.
Partner with retrieval, ranking, and ML teams to make sure indexing outputs meet their requirements.
Enable safe experimentation on indexing strategies and processing logic through controlled rollouts and clearly defined quality signals.
Qualifications
5+ years of experience building production backend or data infrastructure systems.
Strong programming experience in Go, C++, or Rust.
Experience with high-throughput data processing systems and very large datasets.
Experience building or operating databases, storage systems, data planes, or indexing pipelines.
Strong understanding of distributed systems, fault tolerance, consistency, and scalability.
Experience running production systems and handling operational incidents.
A systems-thinking mindset and the ability to reason about end-to-end data flows.
How we rate this
Member of Technical Staff (Index Engineer, Search Crawler) at Perplexity rates 87 out of 100 for how much of the daily work is AI. That makes it Builds AI (AI Level 4 of 4). The level is about AI in the job, not seniority.
Builds AI. The job is building AI systems.
- ●●●● Builds AI80 to 100
- ●●●○ Works on AI60 to 79
- ●●○○ Uses AI40 to 59
- ●○○○ Little AI0 to 39
Levels come from how often the tools, models and workflows of the role are named in the posting itself. Read the description and count the AI tools, models and workflows it names.
Get new software engineering jobs (Builds AI ●●●●) by email
One email a week, each job rated for how much AI is in the work. No recruiter spam, unsubscribe in one click.
Free.
Similar roles
Software Engineering roles that build AI, at other companies.
What kind of AI work fits you?
Answer 12 practical questions in about three minutes. Get a simple profile, the work it points to, and live roles to explore next.
Find my next step