Staff/Principal ML Engineer

We are looking for an exceptional Staff/Principal ML Engineer to join our client, a cutting-edge real-time AI research lab and inference provider backed by top-tier venture funds. Powering hundreds of millions of daily active users and serving over 10 trillion LLM tokens monthly , the platform operates at massive scale behind low-latency voice and conversational AI APIs . In this role, you will take true end-to-end technical ownership of inference optimization, containerizing foundational models and scaling sub-second, high-concurrency workloads on high-performance GPU infrastructure.

What You’ll Be Doing:

• Taking complete end-to-end technical ownership of the model serving and inference optimization lifecycle.
• Taking cutting-edge speech, voice, and LLM architectures directly from the AI research team and containerizing them for production deployment.
• Optimizing serving runtimes and kernels for sub-second latency and ultra-high-concurrency workloads on distributed NVIDIA GPU clusters .
• Architecting and scaling distributed inference platforms using Ray, Kubernetes, and Docker to ensure 99.99% operational reliability under heavy global traffic.
• Diagnosing low-level system bottlenecks, memory allocations, and GPU compute performance across custom model serving pipelines.
• Driving architectural standards, mentoring senior team members, and collaborating cross-functionally between research and core platform engineering.

What We’re Looking For:

• Staff/Principal-level software engineering and ML systems background (equivalent to Google L5/L6 or Meta E5/E6).
• Deep systems-level experience profiling, benchmarking, and optimizing high-throughput, low-latency machine learning pipelines in production.
• Consultant mindset: exceptional technical leadership, high ownership, ability to make sound architectural trade-offs, and clear communication across engineering squads.
• Strong problem-solving abilities within low-level hardware-software interfaces and GPU architectures.

Must-Haves:

• Strong programming mastery in C++, Python, and/or Rust.
• Hands-on expertise with high-performance model serving frameworks: vLLM and TensorRT-LLM (TRT-LLM).
• Deep knowledge of CUDA programming, GPU kernel optimization, and NVIDIA GPU architecture.
• Proven production experience with containerization and distributed orchestration using Kubernetes, Ray, and Docker.
• Demonstrable track record deploying and running large-scale LLM/GenAI inference systems in production.
• Fluent English for daily international technical collaboration.