Spark Cluster Deployment: Deploy, configure, and maintain Apache Spark clusters on Kubernetes, ensuring scalability, reliability, and performance.
Application Deployment: Collaborate with data engineers and data scientists to deploy Spark applications and workloads, ensuring they run efficiently.
Monitoring and Optimization: Implement monitoring solutions to track cluster performance, resource utilization, and application health. Proactively identify and resolve performance bottlenecks.
Resource Management: Manage cluster resources, including CPU, memory, and storage allocation, to ensure optimal utilization and cost efficiency.
Security: Implement and maintain security measures, including authentication, authorization, and encryption, to protect sensitive data and Spark clusters.
Backup and Recovery: Develop and maintain backup and recovery strategies to ensure data integrity and availability in case of failures.
Documentation: Maintain clear and comprehensive documentation of Spark cluster configurations, deployment procedures, and best practices.
Troubleshooting: Quickly diagnose and resolve issues related to Spark clusters, applications, and Kubernetes infrastructure.
Collaboration: Work closely with cross-functional teams, including data engineers, data scientists, and DevOps, to understand application requirements and optimize Spark clusters accordingly.
Requirements
Proven experience deploying and managing Apache Spark on Kubernetes in a production environment.
Proficiency in containerization technologies, particularly Docker and Kubernetes.
Strong knowledge of Spark architecture, including cluster, driver, and worker nodes.
Familiarity with Spark tuning, optimization, and performance monitoring.
Experience with resource management tools like Kubernetes Resource Quotas and LimitRanges.
Understanding of data processing and analytics workflows.
Excellent problem-solving and troubleshooting skills.
Strong communication and collaboration skills.
Experience with Spark cluster orchestration tools like Helm.
Knowledge of Spark ecosystem components such as Spark SQL, Spark Streaming, and MLlib.
Familiarity with cloud-based solution (Azure).
Certification in Kubernetes (e.g., Certified Kubernetes Administrator – CKA).
Knowledge of CI/CD pipelines and infrastructure as code (IaC) tools (e.g., Terraform).
Scripting skills in languages like Python, Bash, or Shell.
Understanding of DevOps practices and automation.