Real Time Data

Working Hour

Regular Hours
Monday - Friday

Business Area

Strategy & Analytics

Location

Malaysia - Kuala Lumpur

Description

**Primary Objective**:

- Subject matter expert on Data Lake / Hadoop platform management and Data Ingestion for Analytics use case implementation
- Utilizes big and small data to generate features, derive impactful insight, visualise information, and support impactful decision-making
- Design, maintain, and improve real-time data pipeline for customization of integration & visualization tools, databases, data warehouse, and analytical systems
- Real-time Data Management for structured and unstructured data for Analytics Department
- Scaling the Capabilities & Data Build Squad team that works on providing internal teams with secure, reliable, performative, and user-friendly access to data to support analytics & other downstream critical processes.
- This role requires technical expertise and willingness to learn a wide variety of technologies to develop real-time data pipelines, data product APIs, and modern & scalable data infrastructure
- Big Data Engineer will be a key part of our team and will help bring structure to vast amounts of data, making it digestible and build scalable data platforms that enable data products, business analytics, and data science.

**Key Responsibilities**:

- Gather and process raw data in real-time at scale.
- Design and implement real-time data pipelines
- Simplify access to real-time data for internal stakeholders using real time
- Read, extract, transform, stage and load data to selected tools and frameworks as required and requested.
- Collaborate with data stakeholders and stewards on the verification and the accuracy of the information collected.
- Provide technical lead to Data Owners and Stewards on data definition, data lineage changes by supporting intake process, performing impact analysis, and conducting domain specific profiling.
- Monitoring data performance and modifying infrastructure as needed.
- Custom integration with various banking systems, data warehouses, and analytics systems.
- Setup of data-access or visualization tools for data scientists such as data science workbench
- Develop and implement analytics use case to yield business value from data and insight
- Extracts and transforms structured and unstructured big and small data to generate features, derive impactful insight, visualise information, and support impactful decision-making
- Design of data pipeline and implement data ingestion from end-to-end to Big Data repositories.
- Human-centred design (together with digital team)
- Data Management & Data Governance for Big Data Platform
- Stay abreast of emerging technologies and projects in the modern Data Engineering/Data Lake/Big Data space

**Requirements**:
**Requirements**:Bachelor Degree - IT, Computer Science or related discipline through certification
- 5+ years of relevant industry experience.
- Excellent knowledge of Python, Scala or Java and SQL.
- Demonstrated ability to build high-volume data ingestion and streaming pipelines (e.g. Kafka, AWS Kinesis, Spark Streaming)
- Working experience with big data technologies (e.g. Spark, Kafka, Hive).
- Experience designing, building, and scaling a production-ready event streaming system.
- Experience in optimizing SQL queries (e.g. data partitioning, bucketing, indexing)
- Experience in building data lake/warehouse solutions consisting of structured and unstructured data.
- Big Data Technology and Programming skill and knowledge such as Python, Hive, Hue, Spark, Pyspark, Yarn, HBase, Cloudera, Nifi, Airflow, Kafka, Linux)
- Knowledge in advanced data technologies like ElasticSearch (Filebeat, LogStash and etc would be an added advantage

**Benefits**:
Dental, Education support, Miscellaneous allowance, Medical, Loans, Sports (e.g. Gym), Parking, Vision, Regular hours, Mondays - Fridays, Casual Business Wear, Performance Based Rewards