Senior Data Engineer(Databricks and Azure)

2 days ago

Kuala Lumpur, Kuala Lumpur, Malaysia Unison Group Full time

Overview

We are looking for a Senior Data Engineer with strong expertise in Databricks, PySpark, Delta Lake, and cloud-based data pipelines. The ideal candidate will design and build scalable ETL/ELT solutions, implement Lakehouse/Medallion architectures, and integrate data from multiple internal and external systems. This role requires strong technical leadership and hands-on architecture experience.

Key Responsibilities

  • Design, build, and optimize data ingestion and transformation pipelines using Databricks, PySpark, and Python.
  • Implement Delta Lake and Medallion architecture for scalable enterprise data platforms.
  • Develop ingestion frameworks for data from SFTP, REST APIs, SharePoint/Graph API, AWS, and Azure sources.
  • Automate workflows using Databricks Workflows, ADF, Azure Functions, and CI/CD pipelines.
  • Optimize Spark jobs for performance, reliability, and cost efficiency.
  • Implement data validation, quality checks, and monitoring with automated alerts and retries.
  • Design secure and governed datasets using Unity Catalog and cloud security best practices.
  • Collaborate with analysts, business users, and cross-functional teams to deliver curated datasets for reporting and analytics.
  • Provide technical leadership and guidance to junior team members.

Required Skills

  • 5–8+ years of experience in Data Engineering.
  • Strong hands-on experience with Databricks, PySpark, Delta Lake, SQL, Python.
  • Experience with Azure Data Lake, ADF, Azure Functions, or AWS equivalents (S3, Lambda).
  • Experience integrating data from APIs, SFTP servers, vendor data providers, and cloud storage.
  • Knowledge of ETL/ELT concepts, Lakehouse/Meddalion architecture, and distributed processing.
  • Strong experience with Git, Azure DevOps CI/CD, and YAML pipelines.
  • Ability to optimize Spark workloads (partitioning, caching, Z-ordering, performance tuning).

Good to Have

  • Exposure to Oil & Gas or trading analytics (SPARTA, KPLER, IIR, OPEC).
  • Knowledge of Power BI or data visualization concepts.
  • Familiarity with Terraform, Scala, or PostgreSQL.
  • Experience with SharePoint development or .NET (optional).

  • Senior Data Engineer(Databricks and Azure)

    2 days ago

    Kuala Lumpur, Kuala Lumpur, Malaysia UnisonTech Consulting Sdn Bhd Full time

    OverviewWe are looking for aSenior Data Engineerwith strong expertise inDatabricks, PySpark, Delta Lake, and cloud-based data pipelines. The ideal candidate will design and build scalable ETL/ELT solutions, implement Lakehouse/Medallion architectures, and integrate data from multiple internal and external systems. This role requires strong technical...

  • Databricks Data Engineer

    2 hours ago

    Kuala Lumpur, Kuala Lumpur, Malaysia AVANADE ASIA PTE LTD Full time

    The Databricks Data Engineer will be responsible for the design, development, and maintenance of scalable and high-performance data pipelines within the Databricks Lakehouse Platform. This role involves using Apache Spark, Delta Lake, and various Databricks services to process large volumes of batch and streaming data, ensuring data quality, reliability, and...

  • Databricks Data Engineer

    2 days ago

    Kuala Lumpur, Kuala Lumpur, Malaysia Avanade Full time

    Come join usThe Databricks Data Engineer will be responsible for the design, development, and maintenance of scalable and high-performance data pipelines within the Databricks Lakehouse Platform. This role involves using Apache Spark, Delta Lake, and various Databricks services to process large volumes of batch and streaming data, ensuring data quality,...

  • Databricks Architect

    3 hours ago

    Kuala Lumpur, Kuala Lumpur, Malaysia AVANADE ASIA PTE LTD Full time

    The Databricks Solution Architect is a senior technical leadership role responsible for defining, designing, and overseeing the implementation of scalable, secure, and high-performance data platforms using the Databricks Lakehouse Platform. This individual will translate business strategy into technical architecture, guide development teams, and ensure all...

  • Azure Devops Engineer

    2 days ago

    Kuala Lumpur, Kuala Lumpur, Malaysia Thakral One Full time

    Design, implement, and maintain CI/CD pipelines for data and AI workloads using Azure DevOpsSupport infrastructure-as-code deployment using TerraformManage environment segregation (development, staging, production) and enforce access controls and deployment governanceAutomate build, testing, and release processes for Airflow, Databricks, and related...

  • Azure DevOps Engineer

    1 week ago

    Kuala Lumpur, Kuala Lumpur, Malaysia Adecco Full time 120,000 - 180,000 per year

    Location: KL | On-siteJob type: 6 months contract (renewable up to 5 years)Responsibilities:Design, implement, and maintain CI/CD pipelines for data and AI workloads using Azure DevOpsSupport infrastructure-as-code deployment using TerraformManage environment segregation (development, staging, production) and enforce access controls and deployment...

  • Azure Devops Engineer

    1 week ago

    Kuala Lumpur, Kuala Lumpur, Malaysia IITMatrix Full time 100,000 - 120,000 per year

    Job description• Design, implement, and maintain CI/CD pipelines for data and AI workloads using Azure DevOps• Support infrastructure-as-code deployment using Terraform• Manage environment segregation (development, staging, production) and enforce access controls and deployment governance• Automate build, testing, and release processes for...

  • Azure DevOps Engineer

    1 week ago

    Kuala Lumpur, Kuala Lumpur, Malaysia Thakral One Full time 80,000 - 120,000 per year

    The Opportunity.Thakral One is looking for anAzure DevOps Engineerwith 5–7 years of experience incloud infrastructure and CI/CD automation. The role involves designing and managing pipelines fordata and AI workloads, supportingTerraform-based IaC, and optimizing deployments acrossAzure,Databricks, andAirflowenvironments.The Role.Design, implement, and...

  • Data Engineer

    1 week ago

    Kuala Lumpur, Kuala Lumpur, Malaysia Oxydata Software Full time 80,000 - 120,000 per year

    Job Title:Senior Data Engineer Location: MalaysiaOverview:ROCKWOOL is seeking a (Senior) Data Engineer to join our global Data Science & Engineering team. Based in Malaysia, you will play a critical role in supporting and developing our global data platform, which collects and processes IoT factory data across our international operations. The current...

  • Azure DevOps Engineer

    1 week ago

    Kuala Lumpur, Kuala Lumpur, Malaysia Thakral One Full time 80,000 - 120,000 per year

    The Opportunity.We are seeking aSenior Azure DevOps Engineerto design, implement, and manageCI/CD pipelinesand infrastructure automation for enterprise-grade data and AI workloads. The role will be part of a large-scale digital transformation initiative with a leading global consulting firm, supporting the build-out of robust and secure cloud environments...