Big Data Engineer - Data Pipeline Architecture

Experience : 6-10 Yrs Job Location : Bangalore ( 3 days hybrid working from office) Notice : Serving Notice for a maximum of 30 days Job Description : As a Software Development Engineer, you will be responsible for expanding and optimising our data and data pipeline architecture as well as optimising data flow and collection for cross-functional teams. The ideal candidate is an experienced data pipeline design and data wrangler who enjoys optimising data systems and building them from the ground up. The Data Engineer will lead our software developers on data initiatives and will ensure optimal data delivery architecture is consistent throughout ongoing projects. They must be self-directed and comfortable supporting the data needs of multiple teams systems and products. The right candidate will be excited by the prospect of optimising or even re-designing our company's data architecture to support our next generation of products and data : - Create and maintain optimal data pipeline architecture - Assemble large complex data sets that meet functional / non-functional business requirements. - Identify design and implement internal process improvements : automating manual processes optimising data delivery, coordinating to re-design infrastructure for greater scalability etc. - Work with stakeholders including the Executive Product Data and Design teams to assist with data-related technical issues and support their data infrastructure needs. - Keep our data separated and secure - Work with data and analytics experts to strive for greater functionality in our data systems. - Support PROD systems Qualifications : - Must have About 6 - 11 years and at least 3 years of relevant experience with Bigdata. - Must have Experience in building highly scalable business applications, which involve implementing large complex business flows and dealing with huge amounts of data. - Must have experience in Hadoop, Hive, and Spark with Scala with good experience in performance tuning and debugging issues. - Hands-on experience in one of the cloud providers (AWS/Azure) is must - Good to have any stream processing Spark/Java Kafka. - Must have experience in the design and development of Big data projects. - Good knowledge of Functional programming and OOP concepts, SOLID principles, and design patterns for developing scalable applications. - Familiarity with build tools like Maven. - Must have experience with any RDBMS and at least one NoSQL database preferably PostgreSQL - Must have experience writing unit and integration tests using the scaliest - Must have experience using any versioning control system - Git - Must have experience with CI / CD pipeline - Jenkins is a plus - Databricks Spark certification is a plus. (ref:hirist.tech)

Location: bangalore, IN

Posted Date: 11/29/2024