Lead Data Engineer - Spark/Hadoop

Job Description : As a Lead Data Engineer, you will play a crucial role in driving data-driven initiatives within our organization. You will be responsible for designing, developing, and implementing scalable data solutions that support our business objectives. You will work closely with data scientists, analysts, and business stakeholders to ensure data quality, integrity, and accessibility. Key Responsibilities : - Develop and maintain robust data architectures and data models that align with business requirements and support data-driven decision-making. - Design, build, and optimize data pipelines to efficiently extract, transform, and load (ETL) data from various sources into our data warehouse or data lake. - Implement and manage data warehousing or data lake solutions to store, manage, and analyze large volumes of data. - Ensure data quality and integrity by implementing data validation, cleansing, and governance processes. Data Modeling : - Develop and maintain data models that effectively represent the organization's data assets and support analytical needs. - Collaborate with data analysts and scientists to provide data-driven insights and visualizations that support business decision-making. - Evaluate and select appropriate data technologies and tools, such as databases, data pipelines, and data visualization platforms. - Lead and mentor junior data engineers, providing guidance and support in their professional development. - Effectively communicate with stakeholders, including business leaders, data scientists, and analysts, to understand their data needs and deliver solutions that meet their expectations. Required Skills and Qualifications : - Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field. - 5 years of experience in data engineering roles. - Strong proficiency in SQL and data modeling techniques. - Experience with data warehousing and data lake solutions Snowflake, AWS Redshift, Azure Data Lake. - Expertise in ETL/ELT tools and frameworks Informatica, Talend, Apache Airflow. - Knowledge of data pipelines and data integration best practices. - Experience with cloud platforms AWS, Azure, GCP and cloud-based data services. - Familiarity with data quality and governance frameworks. - Strong problem-solving and analytical skills. - Excellent communication and interpersonal skills. - Ability to work independently and as part of a team. Preferred Skills : - Experience with big data technologies Hadoop, Spark, Kafka. - Knowledge of data science and machine learning concepts. - Experience with data visualization tools Tableau, Power BI. - Certification in data engineering or cloud platforms. (ref:hirist.tech)

Location: in, IN

Posted Date: 11/27/2024