• 1Search for courses by Study Area, Level and Location
  • 2We deliver you all the matched results
  • 3Choose one or more course providers to contact you
Industry

Distance from location (kms)

Exact 5 10 25 50

3

June

Big Data Engineer- Hadoop/spark

Jobs on jobactive - Melbourne, VIC

IT
Source: uWorkin

JOB DESCRIPTION

Primary responsibility for this team is to undertake the development and maintainence of a bespoke application.

  • Build and support Hadoop data sets for a team of highly skilled Data Scientists.
  • Coding and development in whatever way you choose... Scala, Python, R, Java, Haskell, Go etc
  • Optimising Spark jobs using Presto / Impala.

Execute platform design and build activities, including construction of new CI/CD pipelines. As needed, review and optimize the performance of infrastructure, operating system, and installation/configuration. Address platform related issues, working with the vendors as required. Maintain data flows to ingest data from data sources. Upgrade of OpenShift to version 4.5. Produce automation scripts to automate operational routine tasks and Integrating new data sources into existing Big Data Ecosystems.

Specific Requirements For Big Data Engineer
  • Any experience with functional programming languages such as Scala, Haskell etc is highly regarded.
  • Technical delivery in an Agile development environment.
  • Knowledge of at least one of the following is also preferred: Scala, Python, R,Pig.
  • Deliver best practice real time data streaming in Apache Spark.
  • Solid understanding of unstructured data sets and HDFS platforms, Hadoop, HBase, HIVE, Impala etc.
  • Experience working with API gateways such as APIGEE or WSO2 is highly advantageous
  • Must have experience working with Big Data technologies such as Spark and Elastic Stack
  • Good working knowledge with relational and non-relational database backends such as MySQL, Cassandra/HBase etc.
  • Excellent knowledge of working with DevOps tools such as – Bamboo, Jira, BitBucket/GitLab, SonarQube, Coverity etc.
  • Work in a DevOps model, across multiple projects simultaneously.


Requirements
  • Minimum of 4 years of experience in a structured development environment.
  • Educations Qualifications graduation / higher graduation certification.
  • Passion for implementing continuous integration and deployment concepts
  • Experience practicing infrastructure as code using automation scripts
  • Understanding of enterprise architectures and Service Oriented Architectures
  • Experience with scripting and deployment automation
  • Strong experience with both on premise and cloud environments
  • Familiarity administering both Linux and Windows operating systems
  • Experience in software development
  • As part of your role, you will be working closely with the IT Operations team and Vendors. It is therefore very important that you have strong communication and excellent stakeholder management experience.
  • A lateral thinker and talented problem solver
  • Passionate about providing commercial support and business solutions
  • Practical and tactical while keeping the big picture and organizational impact front of mind
  • Organized, love working at pace, managing multiple projects, processes, and people
  • Committed to collaborating and working effectively across internal functional areas, as well as with external parties