Data Engineer - ETL

As a Data Engineer in our Data Science team, you will play a pivotal role in building and maintaining a central repository of datasets used for advanced analytics and machine learning. Your focus will be on designing efficient ETL processes, developing robust data pipelines, and ensuring data accessibility across the organization. Working closely with cross-functional teams, you will ensure that our cloud-based data repository remains enriched, scalable, and ready for actionable insights. Key Responsibilities : ETL Development : - Design, develop, and maintain efficient ETL processes for diverse datasets. - Implement and optimize data transformation and validation processes to ensure accuracy and consistency. - Collaborate with teams to understand data requirements and implement business logic. Data Pipeline Architecture : - Architect, build, and maintain scalable, high-performance data pipelines. - Explore and implement new technologies to enhance pipeline efficiency and reliability. - Develop pipelines for scraping data from publicly available web sources and other ad-hoc sector-specific datasets. Data Modeling : - Design and implement data models that support analytics and reporting. - Optimize database structures for enhanced performance and scalability. Data Quality and Governance : - Establish and maintain data quality checks and governance standards. - Work with stakeholders to define and enforce data integrity measures. Documentation and Communication : - Document ETL processes, data models, and associated workflows. Communicate technical concepts effectively to non-technical stakeholders. Cross-Functional Collaboration : - Partner with Quant teams and developers to optimize data pipelines. - Engage with external stakeholders to gather business requirements for database enrichment. Basic Qualifications : - Bachelor's degree in Computer Science, Information Technology, or a related field. - Minimum of 2 years of proven experience as a Data Engineer with expertise in ETL techniques. - Strong programming skills in Python, Java, or Scala. - Experience in scraping and transforming data from publicly available web sources. - Proficiency in SQL and familiarity with relational and non-relational databases. - Experience with cloud-based platforms such as AWS, Azure, or GCP. - Solid understanding of data warehousing concepts and architectures. - Knowledge of big data technologies such as Hadoop, Spark, and Kafka. - Expertise in data modeling tools and techniques. - Strong problem-solving, analytical, and communication skills. Preferred Qualifications : - Master's degree in Computer Science, Data Science, or a related field. - Experience with real-time data processing and streaming. - Familiarity with data governance and security best practices (ref:hirist.tech)

Location: bangalore, IN

Posted Date: 11/29/2024