Big Data Architect
- Cranberry Township, PA
Data Architect/ Staff Engineer, Big Data
- Defines technology roadmap in support of product development roadmap
- Lead the design, architecture and development of multiple real time streaming data pipelines encompassing multiple product lines and edge devices
- Ensure proper data governance policies are followed by implementing or validating data lineage, quality checks, classification, etc.
- Provide technical leadership to agile teams – onshore and offshore: Mentor junior engineers and new team members, and apply technical expertise to challenging programming and design problems
- Resolve defects/bugs during QA testing, pre-production, production, and post-release patches
- Have a quality mindset, squash bugs with a passion, and work hard to prevent them in the first place through unit testing, test-driven development, version control, continuous integration and deployment.
- Ability to lead change, be bold, and have the ability to innovate and challenge the status quo.
- Conduct design and code reviews
- Analyze and improve efficiency, scalability, and stability of various system resources
- Operate within Agile Development environment and apply the methodologies
- Track technical debt and ensure unintentional technical debt is not created
- Recommends improvements to the software delivery cycle to help remove waste and impediments for the team
- Drives, promotes and measures team performance against the sprint and project goal.
- Works with the team to continuously improve in development practices and process
- Troubleshoots complex problems with existing or newly-developed software
- Mentoring and coaching of Software Engineers
Required Skills and Knowledge:
- Expert knowledge of data architectures, data pipelines, real time processing, streaming, networking, and security
- Proficient understanding of distributed computing principles
- Advanced knowledge of Big Data querying tools, such as Pig or Hive
- Expert understanding of Lambda Architecture, along with its advantages and drawbacks
- Proficiency with MapReduce, HDFS
Do you have the following experience?
- Bachelor’s Degree
- 12+ years’ experience in software engineering with 2+ years using public cloud
- 6+ years’ experience developing ETL processing flows using MapReduce technologies like Spark and Hadoop
- 4+ years’ experience developing with ingestion and clustering frameworks such as Kafka, Zookeeper, YARN
- 4+ years’ experience with building stream-processing systems, using solutions such as Storm or Spark-Streaming
- 2+ years’ experience with spark structured streaming
- 4+ years’ experience with various messaging systems
- 2+ years’ experience with Kafka
- 1+ years of DevOps experience
- 1+ years’ benchmarking experience
- Experience with integration of data from multiple data sources and multiple data types
One of our clients has a full-time opening for a passionate and hands-on Big Data Architect/Staff Engineer to join our Software Engineering Team located in Cranberry Township, PA.
As a platform team member, you will work with a team of engineers on our cloud data platform that streams data from a variety of health care software and hardware systems in real-time to create transformational recommendations and benchmarking across our customers. Our solutions help to drive improved financial performance, compliance, and better patient outcomes. Each day you will make a positive impact in healthcare, while working with lathe test technologies.
- Master Degree in Engineering/IT/Computer Science
- 1+ years’ experience with DataBricks
- 3+ years’ experience:
- NoSQL databases, such as HBase, Cassandra, MongoDB
- Big Data ML toolkits, such as Mahout, SparkML, or H2O
- Scala or Java Language as it relates to product development.
- 3+ Years’ DevOps experience in cloud technologies like AWS, CloudFront, Kubernetes, VPC, RDS, etc
- Management of Spark or Hadoop clusters, with all included services
- Experience Service Oriented Architecture (SOA) /microservices
Before you apply, please read this:
This position is a full-time (W2) job. Please do not apply if you are looking for a C2C or contract position.
This is not a remote work position.
please apply here: https://jobs.crelate.com/portal/devopsdna/job/4y4f4ephyqib38h3mqg3g1g6ww/BigDataArchitect?d=false