Kafka Developer (Data Streaming & CDC)

Role Overview

Seeking a highly skilled Kafka Developer with deep expertise in Kafka KSQL Streams-based transformations and hands-on experience working with Oracle, MS SQL, and PostgreSQL databases—especially in environments with non-standard schemas (e.g., tables without primary keys, with/without constraints, and foreign keys). The ideal candidate will be responsible for designing and implementing scalable, real-time data pipelines and ensuring robust CDC (Change Data Capture) mechanisms for incremental data replication.

Key Responsibilities

• Design and implement Kafka Streams and KSQL-based transformations for real-time data processing.
• Develop and maintain CDC pipelines using Kafka Connect, Debezium, or custom connectors.
• Handle initial data loads from large relational datasets via manual exports and ensure seamless transition to incremental CDC.

Work with complex relational schemas, including:

• Tables without primary keys
• Tables with/without constraints
• Foreign key relationships
• Optimize data ingestion and transformation pipelines for performance, reliability, and scalability.
• Collaborate with data architects, DBAs, and application teams to ensure data integrity and consistency.
• Document technical designs, data flow diagrams, and operational procedures.
• Communicate effectively with cross-functional teams and stakeholders.

Required Skills and Experience

• Strong hands-on experience with Apache Kafka, Kafka Streams, and KSQL.
• Proficiency in Kafka Connect and CDC tools (Debezium, Confluent).
• Deep understanding of Oracle, MS SQL Server, and PostgreSQL internals and schema design.
• Experience handling non-standard table structures and resolving challenges in CDC replication.
• Familiarity with manual data export/import strategies and their integration into streaming pipelines.
• Strong knowledge of data serialization formats (Avro, JSON, Protobuf).
• Proficient in Java or Scala for custom Kafka development.
• Excellent communication skills—both written and verbal.

Preferred Qualifications

• Experience with schema registry, data governance, and data quality frameworks.
• Familiarity with CI/CD pipelines, GitOps, and containerized deployments (Docker, Kubernetes).
• Prior experience in data architecture or data platform engineering roles.