Senior Distributed ML Engineer

Senior Distributed ML Engineer - AI Safety Research Non-Profit - Berlin (Hybrid)

We are seeking a Machine Learning (ML) Research Scientist specialising in probabilistic inference to join a highly technical research team working at the forefront of machine learning and probabilistic modelling. In this role, you will develop and evaluate advanced probabilistic inference methods, with a particular focus on amortised inference, translating theoretical insights into practical, scalable implementations.

• Collaborate with ML researchers and engineers to accelerate model development, training, and inference across distributed computing environments.
• Build, scale, and improve the reliability and efficiency of distributed training and inference workflows.

System Optimization & Bottleneck Diagnosis:

• Investigate performance bottlenecks across distributed ML workloads.
• Profile code and optimise the utilisation of GPU, CPU, memory, and networking resources.

Tooling & Best Practices:

• Develop tools, libraries, and orchestration infrastructure to make distributed computing resources accessible.
• Establish, document, and maintain engineering best practices across technical teams while evaluating emerging technologies.

Requirements:

• Degree in Computer Science, Computer Engineering, Software Engineering, or a related field (Master's or PhD preferred).
• 3+ years of experience designing and implementing distributed machine learning training systems or frameworks.
• Hands-on experience with technologies like Megatron, DeepSpeed, Hugging Face Accelerate, FSDP, vLLM, or verl.
• Experience with cloud platforms (AWS, GCP, Azure), workload managers (Ray, SLURM), containerization (Docker, Kubernetes), gRPC, and vector databases.
• Proven track record in GPU performance profiling and optimisation using tools like PyTorch Profiler, PyProf, or NVIDIA Nsight.
• Strong problem-solving, debugging, communication, and cross-team collaboration skills.