Lead Data Software Engineer with Databricks, Apache Kafka, Apache Spark, Kubernetes

We are seeking a Lead Data Software Engineer to drive the migration of an existing data analytics platform, originally built on Azure resources such as DataFactory, Databricks, EventHub, and Cassandra, to a cloud-agnostic platform capable of on-premises deployment.

Responsibilities

• Implement streaming and batch Spark pipelines running on K8S
• Develop objects of the data generator framework
• Deploy and test solutions on local and development environments
• Enhance and refine the current actively evolving solution
• Investigate and resolve bugs
• Conduct unit testing to ensure code quality
• Participate in refinement, planning, and demo sessions
• Guide and mentor team members while syncing with a team lead

Requirements

• 5+ years of experience with Apache Spark, Databricks, and Kubernetes (K8S)
• Proficiency in Python
• Experience with Confluent/Apache Kafka
• Completed EPAM Data Engineering course or equivalent qualification
• Capability to quickly adapt and dive into the active implementation phase of a project
• Ability to work independently while coordinating with a team lead
• Strong communication skills and a proactive, hands-on, team-player attitude
• English proficiency at B2 level or higher
• Nice to have Knowledge of Java
• Familiarity with Kafka Connect, KSQL, and Kafka Streams
• Background in Ansible and Argo CD
• Understanding of TDD