Role Overview
Seeking a highly skilled Kafka Developer with deep expertise in Kafka KSQL Streams-based transformations and hands-on experience working with Oracle, MS SQL, and PostgreSQL databases—especially in environments with non-standard schemas (e.g., tables without primary keys, with/without constraints, and foreign keys). The ideal candidate will be responsible for designing and implementing scalable, real-time data pipelines and ensuring robust CDC (Change Data Capture) mechanisms for incremental data replication.
Key Responsibilities
- Design and implement Kafka Streams and KSQL-based transformations for real-time data processing.
- Develop and maintain CDC pipelines using Kafka Connect, Debezium, or custom connectors.
- Handle initial data loads from large relational datasets via manual exports and ensure seamless transition to incremental CDC.
Work with complex relational schemas, including:
- Tables without primary keys
- Tables with/without constraints
- Foreign key relationships
- Optimize data ingestion and transformation pipelines for performance, reliability, and scalability.
- Collaborate with data architects, DBAs, and application teams to ensure data integrity and consistency.
- Document technical designs, data flow diagrams, and operational procedures.
- Communicate effectively with cross-functional teams and stakeholders.
Required Skills and Experience
- Strong hands-on experience with Apache Kafka, Kafka Streams, and KSQL.
- Proficiency in Kafka Connect and CDC tools (Debezium, Confluent).
- Deep understanding of Oracle, MS SQL Server, and PostgreSQL internals and schema design.
- Experience handling non-standard table structures and resolving challenges in CDC replication.
- Familiarity with manual data export/import strategies and their integration into streaming pipelines.
- Strong knowledge of data serialization formats (Avro, JSON, Protobuf).
- Proficient in Java or Scala for custom Kafka development.
- Excellent communication skills—both written and verbal.
Preferred Qualifications
- Experience with schema registry, data governance, and data quality frameworks.
- Familiarity with CI/CD pipelines, GitOps, and containerized deployments (Docker, Kubernetes).
- Prior experience in data architecture or data platform engineering roles.