Senior Databricks Engineer – Platform, Data Engineering

Responsibilities

• Manage workspaces, clusters, jobs, security configurations, libraries, and overall platform health.
• Build and optimize data pipelines and transformations using PySpark and T‑SQL.
• Create automation and tooling using Python, Bash, and PowerShell.
• Implement and automate data governance, access controls, auditability, and compliance best practices.
• Translate conversational ideas into clear solution architectures; break work into tasks in Planner or Azure DevOps for team execution.
• Estimate level of effort and forecast delivery dates with reasonable accuracy.
• Produce high-quality documentation including workflows, gap analyses, design specs, build documents, test plans, and transition documentation.
• Oversee unit, scenario‑based, integrated, capacity, and parallel testing.
• Participate in a 24×7×365 on‑call rotation to support Databricks, SQL Server, and other relevant applications.
• Support system upgrades, patches, performance tuning, observability, and proactive monitoring.
• Maintain accountability for data integrity across applications and pipelines.
• Complete system build and documentation in adherence to departmental guidelines and change management policy.
• Review teammates’ work and provide constructive feedback.
• Communicate effectively with peers, stakeholders, and leadership; provide regular status updates and escalations.
• Drive discussions, mentor accountable leaders, and coordinate resources across initiatives.
• Provide feedback on training materials; conduct training sessions; develop and mentor other team members.

Qualifications

• Solid experience with Databricks administration, including workspace management, cluster configuration, security, jobs, and libraries.
• Strong proficiency in Python and scripting with Bash and PowerShell.
• Experience with PySpark and/or T‑SQL for data processing and transformation.
• Excellent communication skills with the ability to articulate technical concepts clearly to varied audiences.
• Demonstrated critical thinking, problem‑solving, and troubleshooting skills.
• Ability to estimate effort and deliver on time.
• Strong understanding of data governance principles and best practices.
• Commitment to information security best practices.