Senior Data Engineer

Duties and Responsibilities

• Improve the existing layer of data quality modules into a separate data quality engine which identifies, communicates and tracks data quality issues in an automated fashion.
• Develop the back end and front end of data quality dashboards which provides insights into the availability and quality of incoming data.
• Develop connectors for the data pipeline to enable connection with data source and data sink of different technologies to handle batch & streaming data and output batched data into data lake.
• Develop automated DAG based workflows which process data streams on trigger/scheduled intervals.
• Working closely with the data engineering lead to research and identify new technologies to improve the existing pipeline.
• Integrate new data management technologies and software engineering tools into existing structures to improve the performance of data storage, retrieval and security.
• Ensure systems meet business requirements and industry practices.
• Preparation of the big data infrastructure to be analysed by Data Scientists.
• Design, build and integrate data from various resources, and big data management.
• Expertise in Designing and Architecting Hadoop Applications, Data Design, Data construct, install, test and maintain highly scalable data management systems and recommending the right solutions and technologies for the applications.
• Designing of Relational databases and managing of various staging environments.
• Writing complex SQL queries on the data to make sure it is easily accessible, works smoothly.
• Ensure the best possible performance, quality and responsiveness of the application.
• Familiar with Agile software development cycle.
• Ability to deep dive into data or technical issues to come up with effective solutions., excellent in technical communication and working directly with stakeholders.

Qualifications Requirements

Working Experience

• 3+ year as Data Engineer or Bigdata Data Engineer

Qualifications

• 3+ years of working experience in Python programming with data frames / pandas.
• Hands-on experience with Spark, RDDs, large Datasets, Dataframes, Spark SQL, NoSQL.
• Experience on ETL jobs from various databases using custom programming and tools. Knowledge on cloud based ETL jobs.
• Experience with different Hadoop components like MapReduce, HDFS, YARN, HiveQL, PIG, Sqoop, Oozie and Spark (Scala/PySpark), Elasticsearch, Solr, performance tuning.
• Experience working with workflow managers like Airflow, Oozie.
• Working experience using scripting languages like shell / bash.
• Experience with Database Design, management and Administration (MySQL, PostgreSQL, Oracle, Spark SQL, HiveQL).
• Familiar with NoSQL.
• Knowledge on Data Governance tools like Apache Atlas, Ranger.
• Experience in handling streaming platforms like Spark Streaming, Kafka, Flink, Storm / any bigdata steaming.
• Strong knowledge of SQL and must be able to write large complex queries.
• Experience handling different file formats both structure and unstructured (Parquet, avro, xml, json, YAML, csv, txt..etc).
• Experience with offline storage, threading, and performance tuning.
• Ability to understand business requirements and translate them into technical requirements.
• Knowledge in deploying bigdata pipelines in cloud providers like aws, azure, GCP.
• Proficient understanding of code versioning tools, such as Git, familiarity with continuous integration, Continuous delivery (CI/CD) for application software integration and deployment using Jenkins.
• Good understanding of cloud services and scalable infrastructure.
• Exposure in virtualization/ dockerization will be added advantage.
• University degree in Computers or relevant diploma
• Professional certification will be added advantage

Working Condition

Regular working hour (10am – 7pm) at Dattel Sdn Bhd office in Malaysia. Respond to reasonable requests for additional hours as and when it is required in performing the responsibilities.