Backend Software Engineer (Member of Technical Staff, Search Crawer)

Requirements

• Minimum of 5 years of software development experience, with strong knowledge of data structures and algorithms in at least one of the following languages: Python, C++, Rust, or Go
• Experience with large-scale web crawlers is highly desirable
• Proven experience building, deploying, and optimizing high-load, distributed, and hardware‑adjacent services
• Deep understanding of cloud infrastructure, with hands‑on experience in Kubernetes (K8s) and AWS
• Demonstrated passion for writing clean, efficient, and scalable systems

What the job involves

• We are seeking an experienced Backend Software Engineer to join our Crawler team
• In this role, you will design, develop, and operate systems that ingest, process, and manage web‑scale data in support of our next generation of advanced search technologies
• This is a critical, high‑impact engineering position, requiring expertise across both backend and frontend components of our data acquisition stack
• Take ownership of and lead projects focused on developing large‑scale web crawlers, ingestion pipelines, and data processing systems
• Build, maintain, and optimize core backend and frontend components for crawler services, including storage, retrieval, and UI dashboards for data management
• Collaborate closely with Search and Infrastructure teams to ensure the reliable, high‑quality ingestion and processing of billions of web pages
• Architect and implement full‑stack features and scalable distributed systems that handle high‑load and real‑time data operations
• Rapidly iterate, experiment with novel approaches, and continuously enhance system performance, usability, and reliability