Data Engineer

Key Responsibilities

• Design, develop, and maintain data pipelines, ETL/ELT processes, and data integration workflows .
• Architect and optimize data lakes, data warehouses, and streaming platforms .
• Work with structured, semi-structured, and unstructured data at scale.
• Implement real-time and batch data processing solutions .
• Collaborate with Data Scientists, Analysts, and Business stakeholders to deliver high-quality data solutions.
• Ensure data security, lineage, governance, and compliance across platforms.
• Optimize queries, data models, and storage for performance and cost efficiency .
• Automate processes and adopt DevOps/DataOps practices for CI/CD in data engineering.
• Troubleshoot complex data-related issues and resolve production incidents.
• Mentor junior engineers and contribute to technical strategy and best practices.

Technical Skills (Must-Have Tough Requirements)

Programming & Scripting

• Proficiency in Python, Scala, or Java for data engineering.
• Strong SQL skills (query optimization, tuning, advanced joins, window functions).

Big Data & Distributed Systems

• Expertise with Apache Spark, Hadoop, Hive, HBase, Flink, Kafka .
• Hands-on with streaming frameworks (Kafka Streams, Spark Streaming, Flink) .

Cloud & Data Platforms

• Deep knowledge of AWS (Redshift, Glue, EMR, Athena, S3, Kinesis) ,
  or Azure (Synapse, Data Factory, Databricks, ADLS) ,
  or GCP (BigQuery, Dataflow, Pub/Sub, Dataproc) .
• Experience with Snowflake, Databricks, or Teradata .

ETL/ELT & Orchestration

• Strong experience with Airflow, Luigi, Azkaban, Prefect .
• ETL tools like Informatica, Talend, SSIS .

Data Modeling & Storage

• Experience with Data Lakes, Data Warehouses, and Lakehouse architectures .
• Knowledge of Star Schema, Snowflake Schema, Normalization/Denormalization .

DevOps & Automation

• Proficiency in CI/CD (Jenkins, GitLab, Azure DevOps) for data pipelines.
• Experience with Docker, Kubernetes, Terraform, Ansible for infrastructure automation.

Other Tough Skills

• Strong knowledge of Data Governance, MDM, Data Quality, Metadata Management .
• Familiarity with Graph Databases (Neo4j), Time-Series Databases (InfluxDB, TimescaleDB) .
• Understanding of machine learning data pipelines (feature engineering, model serving).

Qualifications

• Bachelor’s/Master’s degree in Computer Science, Data Engineering, or related field.
• 7–10 years of experience in data engineering or big data development .
• At least 2–3 large-scale end-to-end data platform implementations .
• Preferred Certifications:
  AWS Certified Data Analytics – Specialty
  Google Professional Data Engineer
  Databricks Certified Data Engineer