We're looking for a seasoned Big Data Engineering Lead with expertise in Scala, Python, and PySpark to lead our client data engineering team. You'll be responsible for designing and implementing scalable, efficient, and fault-tolerant data pipelines, as well as mentoring team members and driving technical innovation.
Key Responsibilities:
- Design and develop large-scale data pipelines using Scala, Python, and PySpark
- Lead and mentor a team of data engineers to build and maintain data architectures
- Collaborate with cross-functional teams to identify data requirements and implement data-driven solutions
- Ensure data quality, integrity, and security across all data pipelines
- Develop and maintain technical documentation for data pipelines and architectures
- Stay up-to-date with industry trends and emerging technologies in big data, cloud computing, and data engineering
- Drive technical innovation and recommend new tools and technologies to improve data engineering capabilities
Requirements:
- 5+ years of experience in big data engineering, with expertise in Scala, Python, and PySpark
- Strong experience with big data technologies such as Apache Spark, Hadoop, and Kafka
- Experience with cloud-based data platforms such as AWS, GCP, or Azure
- Strong understanding of data architecture, data governance, and data security
- Excellent leadership and mentoring skills, with experience leading high-performing teams
- Strong communication and collaboration skills, with ability to work with cross-functional teams
- Bachelor's or Master's degree in Computer Science, Engineering, or related field