Senior Data Engineer

Key PointsDetails
Position Title Senior Data Engineer
Division Data Management
Position Type Contract / Permanent
LocationPetaling Jaya, Malaysia

Duties and Responsibilities

  • Improve the existing layer of data quality modules into a separate data quality engine which identifies, communicates and tracks data quality issues in an automated fashion.
  • Develop the back end and front end of data quality dashboards which provides insights into the availability and quality of incoming data.
  • Develop connectors for the data pipeline to enable connection with data source and data sink of different technologies to handle batch & streaming data and output batched data into data lake.
  • Develop automated DAG based workflows which process data streams on trigger/scheduled intervals.
  • Working closely with the data engineering lead to research and identify new technologies to improve the existing pipeline.
  • Integrate new data management technologies and software engineering tools into existing structures to improve the performance of data storage, retrieval and security.
  • Ensure systems meet business requirements and industry practices.
  • Preparation of the big data infrastructure to be analysed by Data Scientists.
  • Design, build and integrate data from various resources, and big data management.
  • Expertise in Designing and Architecting Hadoop Applications, Data Design, Data construct, install, test and maintain highly scalable data management systems and recommending the right solutions and technologies for the applications.
  • Designing of Relational databases and managing of various staging environments.
  • Writing complex SQL queries on the data to make sure it is easily accessible, works smoothly.
  • Ensure the best possible performance, quality and responsiveness of the application.
  • Familiar with Agile software development cycle.
  • Ability to deep dive into data or technical issues to come up with effective solutions., excellent in technical communication and working directly with stakeholders.

Qualifications Requirements

Working Experience

  • 3+ year as Data Engineer or Bigdata Data Engineer

Qualifications

  • 3+ years of working experience in Python programming with data frames / pandas.
  • Hands-on experience with Spark, RDDs, large Datasets, Dataframes, Spark SQL, NoSQL.
  • Experience on ETL jobs from various databases using custom programming and tools. Knowledge on cloud based ETL jobs.
  • Experience with different Hadoop components like MapReduce, HDFS, YARN, HiveQL, PIG, Sqoop, Oozie and Spark (Scala/PySpark), Elasticsearch, Solr, performance tuning.
  • Experience working with workflow managers like Airflow, Oozie.
  • Working experience using scripting languages like shell / bash.
  • Experience with Database Design, management and Administration (MySQL, PostgreSQL, Oracle, Spark SQL, HiveQL).
  • Familiar with NoSQL.
  • Knowledge on Data Governance tools like Apache Atlas, Ranger.
  • Experience in handling streaming platforms like Spark Streaming, Kafka, Flink, Storm / any bigdata steaming.
  • Strong knowledge of SQL and must be able to write large complex queries.
  • Experience handling different file formats both structure and unstructured (Parquet, avro, xml, json, YAML, csv, txt..etc).
  • Experience with offline storage, threading, and performance tuning.
  • Ability to understand business requirements and translate them into technical requirements.
  • Knowledge in deploying bigdata pipelines in cloud providers like aws, azure, GCP.
  • Proficient understanding of code versioning tools, such as Git, familiarity with continuous integration, Continuous delivery (CI/CD) for application software integration and deployment using Jenkins.
  • Good understanding of cloud services and scalable infrastructure.
  • Exposure in virtualization/ dockerization will be added advantage.
  • University degree in Computers or relevant diploma
  • Professional certification will be added advantage

Working Condition

Regular working hour (10am – 7pm) at Dattel Sdn Bhd office in Malaysia. Respond to reasonable requests for additional hours as and when it is required in performing the responsibilities.

Career Form