Data Engineer

Freelance Databricks Migration Specialist (Energy Sector) – Contract Role

Location: Germany remote
Contract length: 6 months rolling
Project Length: Up to two years
Capacity: Full time

An energy-sector client is undertaking a major transformation of its data platform and is seeking a Freelance Databricks Migration Specialist to lead the migration to Azure Databricks. This is a hands-on role focused on architecture, data engineering, and end-to-end migration execution.

Key Responsibilities

• Design target architecture on Azure + Databricks (Delta Lake, medallion layers, governance).
• Assess the existing data estate and define a clear migration roadmap.
• Rebuild legacy ETL/ELT processes using PySpark and Spark SQL.
• Implement ingestion pipelines via Databricks Workflows, Autoloader, ADF, or Event Hub.
• Configure Delta Lake tables, optimisation patterns, data quality checks, and schema evolution.
• Set up platform governance (Unity Catalog, permissions, cluster policies).
• Integrate Databricks with Azure services including ADLS, Key Vault, Purview, and Azure DevOps.
• Establish CI/CD pipelines and best practices for code versioning and releases.
• Collaborate with internal teams and provide documentation and knowledge transfer.

Required Skills & Experience

• Strong experience delivering Databricks migration projects end-to-end.
• Expertise in Azure Databricks, PySpark, Spark optimisation, and distributed data processing.
• Solid background in Azure services: ADLS Gen2, Data Factory, Key Vault, DevOps, Event Hub.
• Hands-on experience implementing Delta Lake, Unity Catalog, and medallion architecture.
• Proven ability to migrate from SQL Server, Oracle, Hadoop, Teradata, or similar legacy platforms.
• Strong knowledge of data governance, RBAC, secrets management, and monitoring.
• Experience designing scalable ingestion pipelines (batch and streaming).
• Excellent problem-solving skills and ability to work autonomously within agile teams.

Nice-to-Have

• Sector experience in energy, utilities, metering, grid, or IoT/SCADA data.
• Familiarity with MLflow and Databricks SQL Warehouses.
• Data modelling experience (Star Schema, Data Vault, Lakehouse patterns).

Tech Stack

Azure Databricks, Delta Lake, Unity Catalog, PySpark, Spark SQL, Databricks Workflows, ADLS Gen2, Azure Data Factory, Azure DevOps, Key Vault, Event Hub/Kafka, Azure Monitor, Log Analytics.