Databricks Data Engineer

2 weeks ago

Kuala Lumpur, Kuala Lumpur, Malaysia AVANADE ASIA PTE LTD Full time 120,000 - 240,000 per year

The Databricks Data Engineer will be responsible for the design, development, and maintenance of scalable and high-performance data pipelines within the Databricks Lakehouse Platform. This role involves using Apache Spark, Delta Lake, and various Databricks services to process large volumes of batch and streaming data, ensuring data quality, reliability, and accessibility for data consumers.

Key Responsibilities

  • Data Pipeline Development: Design, build, and maintain robust and scalable ETL/ELT pipelines using Databricks, PySpark/Scala, and SQL to ingest, transform, and load data from diverse sources (e.g., databases, APIs, streaming services) into Delta Lake.
  • Databricks Ecosystem Utilization: Utilize core Databricks features such as Delta Lake, Databricks Workflows (or Jobs), Databricks SQL, and Unity Catalog for pipeline orchestration, data management, and governance.
  • Performance Optimization: Tune and optimize Spark jobs and Databricks clusters for maximum efficiency, performance, and cost-effectiveness.
  • Data Quality and Governance: Implement data quality checks, validation rules, and observability frameworks. Adhere to data governance policies and leverage Unity Catalog for fine-grained access control.
  • Collaboration: Work closely with Data Scientists, Data Analysts, and business stakeholders to translate data requirements into technical solutions and ensure data is structured to support analytics and machine learning use cases.
  • Automation & DevOps: Implement CI/CD and DataOps principles for automated deployment, testing, and monitoring of data solutions.
  • Documentation: Create and maintain technical documentation for data pipelines, data models, and processes.
  • Troubleshooting: Monitor production pipelines, troubleshoot complex issues, and perform root cause analysis to ensure system reliability and stability.

Qualifications

Required Skills & Experience:

  • 5+ years of hands-on experience in Data Engineering.
  • 3+ years of dedicated experience building solutions on the Databricks Lakehouse Platform.
  • Expert proficiency in Python (PySpark) and SQL for data manipulation and transformation.
  • In-depth knowledge of Apache Spark and distributed computing principles.
  • Experience with Delta Lake and Lakehouse architecture.
  • Strong understanding of ETL/ELT processes, data warehousing, and data modeling concepts.
  • Familiarity with at least one major cloud platform (AWS, Azure, or GCP) and its relevant data services.

Preferred Skills & Certifications:

  • Experience with Databricks features like Delta Live Tables (DLT), Databricks Workflows, and Unity Catalog.
  • Experience with streaming technologies (e.g., Kafka, Spark Streaming).
  • Familiarity with CI/CD tools and Infrastructure-as-Code (e.g., Terraform, Databricks Asset Bundles).
  • Databricks Certified Data Engineer Associate or Professional certification.

  • Databricks Architect

    2 weeks ago

    Kuala Lumpur, Kuala Lumpur, Malaysia AVANADE ASIA PTE LTD Full time $120,000 - $200,000 per year

    The Databricks Solution Architect is a senior technical leadership role responsible for defining, designing, and overseeing the implementation of scalable, secure, and high-performance data platforms using the Databricks Lakehouse Platform. This individual will translate business strategy into technical architecture, guide development teams, and ensure all...

  • Databricks Machine Learning Lead

    2 weeks ago

    Kuala Lumpur, Kuala Lumpur, Malaysia AVANADE ASIA PTE LTD Full time 120,000 - 180,000 per year

    The Databricks M/L Technical Lead is a senior, hands-on role responsible for the design, development, and delivery of highly scalable, secure, and performant data solutions on the Databricks Lakehouse Platform. It is expected to provide technical leadership to a team of engineers, defining coding standards, implementing architectural patterns, and ensuring...

  • lead / Senior data engineer: Python / AWS / Databricks

    2 days ago

    Kuala Lumpur, Kuala Lumpur, Malaysia Capcon Asia Full time 120,000 - 200,000 per year

    Why are they awesome?Data driven technology spin off from a leading aviation strategy, logistics and air cargo consultancyBuilding data-driven SaaS tools that help airlines make informed commercial decisions from; routes, maintenance, demand planning, operations and forecast modelsProducts / Projects2 core SaaS products that are mature - actively used by...

  • Data Engineer

    2 days ago

    Kuala Lumpur, Kuala Lumpur, Malaysia Oxydata Software Full time 80,000 - 120,000 per year

    Job Title:Senior Data Engineer Location: MalaysiaOverview:ROCKWOOL is seeking a (Senior) Data Engineer to join our global Data Science & Engineering team. Based in Malaysia, you will play a critical role in supporting and developing our global data platform, which collects and processes IoT factory data across our international operations. The current...

  • Data Engineer Data bricks

    2 days ago

    Kuala Lumpur, Kuala Lumpur, Malaysia Tata Consultancy Services (TCS) Full time 80,000 - 120,000 per year

    Job Description:Data processing Proficiency.Database Technologies (Microsoft SQL & Cloud-native) 2 SQL Level 2 - 1 Oracle & LinuxDevelop and optimize ETL processes using Databricks.Containerization: Utilize Docker for packaging Databricks.Problem-Solving and debugging.DevOps Practices: Implement CI/CD, automation, and version control for Databricks...

  • Data Engineer

    2 days ago

    Kuala Lumpur, Kuala Lumpur, Malaysia Axiata Digital Labs Full time 120,000 - 240,000 per year

    Key ResponsibilitiesDevelop and maintain data models (customer 360, product, network, usage)Work with data lakes, warehouses, and lakehouses (e.g., Hadoop, Snowflake, BigQuery, Databricks)Implement data quality checks (deduplication, validation, reconciliation)Enforce data privacy and regulatory compliance (GDPR, PDPA, CCPA, telco-specific...

  • Analytics Engineer/Data Engineer

    5 minutes ago

    Kuala Lumpur, Kuala Lumpur, Malaysia TWO95 International, Inc Full time 480,000 - 960,000 per year

    Analytics Engineer/Data EngineerKey Responsibilities· Data Transformation & Modeling: Build analytics-ready datasets using a layered approach (Medallion Architecture – Bronze, Silver, Gold) in Databricks.· Delta Live Tables (DLT): Design, manage, and optimize DLT pipelines to deliver reliable and automated transformations at scale.· Enable Self-Service:...

  • Data Engineer

    2 days ago

    Kuala Lumpur, Kuala Lumpur, Malaysia Getronics Solutions (Malaysia) Sdn. Bhd. Full time 900,000 - 1,200,000 per year

    Job Description - Data EngineerSummaryResponsible for building and maintaining our data infrastructure including pipelines and data storage.Roles & responsibilitiesDesign, develop, and maintain data pipelines to ingest data from SAP ECC, SAP S4 Hana, and SAPCollaborate with data architects and other stakeholders to build efficient data integration...

  • Data Engineer

    2 days ago

    Kuala Lumpur, Kuala Lumpur, Malaysia Getronics Full time 80,000 - 160,000 per year

    Job Description - Data EngineerSummaryResponsible for building and maintaining our data infrastructure including pipelines and data storage.Roles & responsibilitiesDesign, develop, and maintain data pipelines to ingest data from SAP ECC, SAP S4 Hana, and SAP.Collaborate with data architects and other stakeholders to build efficient data integration...

  • Senior Data Engineer

    2 days ago

    Kuala Lumpur, Kuala Lumpur, Malaysia Talentbook Solutions Full time 60,000 - 120,000 per year

    Urgent opening for Senior Data Engineer-Quality who's focused on testing ETL pipelines, data structure etc ... must be able to work EMEA hours 4pm to 1am (hybrid) ... salary up to RM15kPerm employment with top notch US Delivery Center in Malaysia (4 Positions)Role SummaryWe are looking for a detail-oriented and passionate Quality Engineer to ensure the...