Senior Data Engineer
Job Description
Role - Senior Data Engineer
Contract - 3 months Renewable
Client Industry - Entertainment and Development
Work location - Onsite - Abu Dhabi
Notice period acceptable - Immediately available or maximum notice of 30 days allowed
Job description:
Data Engineering (ETL/ELT & Data Processing):
Understand, analyse, and document business requirements to design data-driven solutions aligned with organizational goals.
Deep expertise in ETL methodologies, Big Data, and Data Warehousing principles, including architecture design, data modelling, and performance optimization.
Design, develop, test, optimize, and deploy ETL pipelines using Azure Data Factory (ADF), Azure Databricks, and Azure Data Lake Storage.
Good to have Hands-on experience with leading ETL tools such as Talend, Informatica, and Penta ho for building scalable ETL/ELT workflows.
Translate source-to-target mapping documents into efficient ETL processes to support data integration and transformation.
Write, implement, and maintain robust ETL code and stored procedures using SQL, Python, and Spark.
Cloud & Data Warehousing:
Responsible for designing, implementing, and managing ETL processes on Microsoft Azure Cloud, with strong proficiency in ADF, Azure Databricks, Azure Synapse Analytics, and Azure Data Lake.
Administer and maintain cloud-based data platforms, including Azure Cloud and Snowflake Enterprise Data Warehouse, ensuring data security, performance, and availability.
Develop and manage Spark jobs within Azure Databricks for processing large scale datasets.
Cl/CD & DevOps:
Working an Agile development environment, applying best practices for Cl/CD pipelines using Azure DevOps, GitHub, and automation tools.
Automate deployment workflows, manage version control, and ensure continuous integration and delivery of data solutions.
Implement DevOps practices for monitoring, testing, and optimizing data pipelines, ensuring high system reliability and performance.
Monitoring & Maintenance:
Proactively monitor, troubleshoot, and maintain all ETL components to ensure data quality, system stability, and timely delivery of data.
Optimize data workflows for performance, scalability, and cost-efficiency in cloud environments.
Machine Learning & API Integration:
Deploy Machine Learning (ML) models in the cloud using Azure Machine Learning and integrate with data pipelines.
Design and integrate APls to facilitate seamless data flow between systems and applications.
Leadership & Collaboration:
Lead, mentor, and support team members engaged in data engineering activities, promoting knowledge sharing and best practices.
Collaborate closely with business stakeholders, data scientists, and ITteams to align technical solutions with business needs.
Qualifications
Essential:
5+ years of experience within area of Data Engineering expertise
5+ years&apos experience of ETL development using Talend / Azure Data Services
5+ years&apos experience of developing SQL queries, stored procedures, and views
3+ years&apos experience developing/coding in Python
3+ years&apos experience of Azure or any cloud solutions
Desirable:
3+ years&apos experience of Azure Databricks or any cloud data warehouse such as (Snowflake, redshift)
2+ years&apos experience of Azure DF
Education
Essential:
Bachelor&aposs Degree in Computer Science, Engineering, or a related field
Desirable:
Advanced certification in Azure Data Engineering (Azure DF, Azure ML)
Certification in Azure Databricks or any cloud warehouse
Any certification in data science is a big plus.