Senior Data Software Engineer with Apache Airflow, Databricks

Responsibilities

• Develop new data pipelines and maintain existing ones
• Migrate existing data pipelines from BDP to Databricks
• Maintain ML ETL pipelines and apply necessary changes to ML code
• Monitor daily pipelines and resolve issues as they arise
• Support ML model development, training, and deployment
• Perform data integration and transformation through ETL processes
• Contribute to database design, optimization, and data quality assurance
• Document processes and workflows in clear and unambiguous language
• Collaborate with the team using BitBucket, Jira, Confluence, and Teams

Requirements

• 3+ years of experience in data engineering with advanced Proficiency in Python, Spark, and PySpark
• Hands-on experience with Databricks and AWS
• Background in ML model development, training, and deployment
• Skills in data pipeline development, maintenance, and ETL processes
• Understanding of database design, optimization, and data quality assurance
• Fluent in English with strong documentation skills

Nice to have

• Familiarity with pandas
• Knowledge of Databricks with MLFlow or AWS Sagemaker