Databricks Engineer

We are seeking an experienced Data Engineer to design, build, and maintain our cloud-based data infrastructure on Azure and Databricks. This role is pivotal in ensuring our data platform is scalable, secure, and optimised for advanced analytics and reporting.

Key Responsibilities

Platform Development & Maintenance

1. Design and implement robust data pipelines using Azure Data Factory, Databricks, and related Azure services.
2. Build ETL/ELT processes to transform raw data into structured, analytics-ready formats.
3. Optimise pipeline performance, ensure data quality, and maintain high availability of data services.

Infrastructure & Architecture

1. Architect and deploy scalable data lake solutions using Azure Data Lake Storage.
2. Implement data governance and security measures aligned with compliance standards.
3. Use Infrastructure-as-Code tools (e.g., Terraform) for reproducible deployments.
4. Define best practices for data organisation, partitioning, and storage optimisation.

Databricks Development

1. Develop and optimise jobs within Databricks, leveraging Delta Lake for reliable storage and ACID transactions.
2. Implement medallion architecture (bronze, silver, gold layers) and reusable notebooks.
3. Optimise cluster configurations for cost and performance.
4. Establish CI/CD pipelines for Databricks deployments.

Monitoring & Operations

1. Implement monitoring and alerting solutions using Azure Monitor, Log Analytics, and Databricks tools.
2. Troubleshoot performance issues, optimise resource utilisation, and ensure SLAs are met.
3. Define disaster recovery procedures and backup strategies.

Collaboration & Documentation

1. Partner with data scientists, analysts, and business stakeholders to deliver tailored data solutions.
2. Document technical architectures, data flows, and operational procedures for knowledge sharing.

Essential Skills & Experience

1. 5+ years of experience with Azure Cloud services (Data Factory, Data Lake Storage, SQL Database, Synapse Analytics).
2. Strong hands-on experience with Databricks, Delta Lake, and cluster management.
3. Proficiency in SQL and Python for data manipulation and pipeline development.
4. Experience with Git/GitHub and CI/CD practices.
5. Solid understanding of data modelling concepts and data lake architectures.
6. Knowledge of data governance, security, and compliance requirements.

Desirable Skills

1. Experience with Terraform or other Infrastructure-as-Code tools.
2. Familiarity with Azure DevOps or similar CI/CD platforms.
3. Exposure to data quality frameworks and testing approaches.
4. Azure certification (e.g., Azure Data Engineer Associate).
5. Databricks certification.