Lead Databricks Data Engineer

Location: Remote (preferred- On-Site)

The project focuses on building and deploying a Network Investment Decision Engine, including data ingestion, optimisation modelling, Databricks deployment, automated testing and a decision-support dashboard.

About the Role

We are looking for a strong Lead Databricks Data Engineer to join a strategic data and optimisation project for a major South African retail client.

This is a remote role, but strong stakeholder management is essential. The person will work directly with senior client stakeholders, including the CEO and other C-level executives, and must be confident in communicating complex technical topics at an executive level.

Key Responsibilities

• Lead the technical delivery of the Databricks-based solution.

• Design and develop scalable data pipelines, models and ingestion processes.

• Work closely with Data Scientists, Business Analysts and client stakeholders.

• Translate complex business requirements into practical technical solutions.

• Implement data quality checks, testing, monitoring and automated data refreshes.

• Support the deployment and optimisation of the solution in Databricks.

• Provide technical leadership and challenge assumptions when required.

• Present progress, risks, technical decisions and recommendations to senior/C-level stakeholders.

• Ensure strong alignment between technical delivery and business objectives.

• Contribute to technical documentation and best practices.

Requirements

• Strong commercial experience as a Lead Data Engineer / Senior Data Engineer.

• Strong hands-on experience with Databricks.

• Excellent Python and SQL skills.

• Experience building data ingestion and ETL/ELT pipelines.

• Strong understanding of cloud data platforms and data architecture.

• Proven experience working with senior stakeholders and C-level executives.

• Excellent communication, presentation and stakeholder management skills.

• Ability to explain complex technical concepts clearly to non-technical audiences.

• Strong ownership, leadership and problem-solving mindset.

• Experience in retail, optimisation, financial modelling or decision-support solutions is a plus.

Key Success Factor

This role requires someone who is technically strong but equally strong in stakeholder management. The ability to confidently work with the CEO and other C-level stakeholders is critical.