Senior Data Engineer

Job Description

Key Responsibilities:

• Data Pipeline Development & Management: Design, implement, and maintain scalable and reliable data pipelines to ingest, transform, and load structured, unstructured, and real-time data feeds from diverse sources.
• Data Processing & Optimization: Build efficient, high-performance systems by leveraging techniques like data denormalization, partitioning, caching, and parallel processing.
• Database Engineering: Write and optimize complex SQL queries on relational (Redshift, PostgreSQL) and NoSQL (MongoDB) databases.
• Documentation & Continuous Improvement: o Maintain thorough documentation including technical specifications, data flow diagrams, and operational procedures.

Qualifications:

• Bachelors or Masters degree in Computer Science, Engineering, Data Science, or a related field.
• At least 8 years of experience in data engineering within a production environment.
• Advanced knowledge of Python and Linux shell scripting for data manipulation and automation.
• Strong expertise in SQL/NoSQL databases such as PostgreSQL and MongoDB.
• Experience building stream processing systems using Apache Kafka.
• Proficiency with Docker and Kubernetes in deploying containerized data workflows.
• Good understanding of cloud services (AWS or Azure).
• Hands-on experience with ELK stack (Elasticsearch, Logstash, Kibana) for scalable search and logging.
• Experience working with Lakehouse systems, data denormalization, and data labeling practices.
• Experience in data correlation, enrichment from external sources, and managing data integrity at scale.
• Familiarity with AI models supporting data management.