Lead Engineer - Spark Streaming

RecruiterDNA - Pittsburgh, PA

We are looking for an mid level to Sr. level Engineer with experience in Real-time Big Data Pipelines with Spark and Kafka running MapReduce.


This full-time position with RecruiterDNA's Client, is classified as at the :Software Engineer, IV,  Level, and will not have any direct reports but will act a lead and mentor.

Industry:  Pharma Tech, Healthcare technology

Location:  Mountain View, CA  or  Suburb outside of  Pittsburgh, PA.   Not  C2C, remote or virtual work arrangements. 

Is relocation assistance available:  Yes

General Responsibilities:

Develop custom batch-oriented and real-time streaming data pipelines working within the MapReduce ecosystem, migrating flows from ELT to ETL Ensure proper data governance policies are followed by implementing or validating data lineage, quality checks, classification, etc.

Act in a technical leadership capacity: Mentor junior engineers and new team members, and apply technical expertise to challenging programming and design problems Resolve defects/bugs during QA testing, pre-production, production, and post-release patches Possess a quality mindset, squash bugs with a passion, and work hard to prevent them in the first place through unit testing, test-driven development, version control, continuous integration, and deployment.

Conduct design and code reviews Analyze and improve efficiency, scalability, and stability of various system resources Contribute to the design and architecture of the project

Operate within Agile Development environment and apply the methodologies Required Knowledge and Skills: Proficient understanding of distributed computing principles Good knowledge of Big Data querying tools, such as Pig or Hive Good understanding of Lambda Architecture, along with its advantages and drawbacks Proficiency with MapReduce, HDFS Ability to solve any ongoing issues with operating the cluster Ability to lead change, be bold, and have the ability to innovate and challenge the status quo Passionate about solving customer problems and develop solutions that result in a passionate customer/community following Basic

Basic Qualifications:

Bachelor’s degree in Engineering/IT/Computer Science 6+ years’ experience in software engineering

2+ years experience:  developing ETL processing flows using MapReduce technologies like Spark and Hadoop In Software Design

1+ years’ experience:  developing with ingestion and clustering frameworks such as Kafka, Zookeeper, YARN building stream using Spark-Streaming with various messaging systems, such as Kafka or RabbitMQ

Experience with integration of data from multiple data sources Experience leading projects or teams

Preferred Qualifications:

Master Degree in Engineering/IT/Computer Science 8+ years’ experience in software engineering

1+ years’ experience with: DataBricks and Spark NoSQL databases, such as HBase, Cassandra, MongoDB Big Data ML toolkits, such as Mahout, SparkML, or H2O

2+ years’ experience with Scala or Java Language as it relates to product development. Management of Spark or Hadoop clusters, with all included services Experience Service Oriented Architecture ( SOA) /microservices Demonstrable advanced knowledge of data architectures, data pipelines, real-time processing, streaming, networking, and security


Posted On: Thursday, October 11, 2018
Compensation: 130,000.

Apply to this job