Data Platform Engineer

Job Overview

We are seeking a Senior Data Platform Engineer with 15+ years of experience to lead the administration and optimization of large-scale data infrastructure within the banking sector. This role requires deep expertise in Kubernetes (on-premises & Azure Kubernetes), Spark Cluster management, and CI/CD automation with Azure DevOps . The ideal candidate will be responsible for ensuring high availability, scalability, security, and performance of data platforms while working with Apache Ranger, Kafka, Trino, Iceberg, Datahub, and Airflow .

Key Responsibilities

• Administer and manage on-premises Kubernetes (in-house package) and Azure Kubernetes.
• Optimize Spark Clusters for high-performance data processing in banking applications.
• Implement and maintain CI/CD pipelines using Azure DevOps (ADO) .
• Ensure security and governance through Apache Ranger and role-based access control.
• Deploy and manage streaming data solutions using Kafka .
• Support data orchestration via Airflow to ensure efficient workflow execution.
• Maintain scalable data storage solutions using Iceberg and Trino .
• Integrate and track data lineage through Datahub to enhance transparency and data governance.
• Develop robust monitoring solutions to track system health and performance metrics.
• Collaborate with data engineers and business teams to drive innovation and automation.

Required Qualifications

• 15+ years of experience in data platform engineering, infrastructure management, or DevOps .
• Strong hands-on experience with Kubernetes administration (on-premises & Azure).
• Extensive experience managing Spark Clusters for large-scale data processing.
• Proficiency in CI/CD implementation using Azure DevOps .
• In-depth knowledge of Apache Ranger, Kafka, Airflow, Trino, Iceberg, and Datahub .
• Expertise in data security, governance, and access control mechanisms .
• Understanding of financial data compliance in banking environments.
• Solid problem-solving skills and ability to troubleshoot complex data infrastructure issues.