Lead Data Engineer - ETL

Seeking for a Lead data Engineer with a wealth of knowledge in managing legacy SQL Server installations and creating ETL procedures with PySpark Notebooks and Microsoft Fabric. The ideal applicant will have a solid foundation in Spark-based development, exhibit a high level of SQL competence, and feel at ease working both independently and in teams, as well as managing other developers when necessary. Lead Data Engineer (Data Fabric) || (10-13 Y) || Hyderabad (Hybrid) || Quick Joiner || This role is critical and requires resources to have adequate hands on exp in MS Fabric in Projects and should be reflective in the resume too. Scope of Work : - Design, develop, and maintain ETL pipelines using PySpark Notebooks and Microsoft Fabric. - Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver efficient data solutions. - Migrate and integrate data from legacy SQL Server environments into modern data platforms. - Optimize data pipelines and workflows for scalability, efficiency, and reliability. - Provide technical leadership and mentorship to junior developers and other team members. - Troubleshoot and resolve complex data engineering issues related to performance, data quality, and system scalability. - Develop, maintain, and enforce data engineering best practices, coding standards, and documentation. - Conduct code reviews and provide constructive feedback to improve team productivity and code quality. - Support data-driven decision-making processes by ensuring data integrity, availability, and consistency across different platforms. Requirements : - Bachelor's or master's degree in computer science, Data Science, Engineering, or a related field. - Experience with Microsoft Fabric or similar cloud-based data integration platforms - any (SAP HANA Cloud / Databricks DIP / Vertex AI / SnowFlake / Dataiku / IBM Watson Studio / Dataloop, etc.) is a must. - 10 years of experience in data engineering, with a strong focus on ETL development using PySpark or other Spark-based tools. - Proficiency in SQL with extensive experience in complex queries, performance tuning, and data modeling. - Strong knowledge of data warehousing concepts, ETL frameworks, and big data processing. - Familiarity with other data processing technologies (e.g., Hadoop, Hive, Kafka) is an advantage. - Experience working with both structured and unstructured data sources. - Excellent problem-solving skills and the ability to troubleshoot complex data engineering issues. - Proven ability to work independently, as part of a team, and in leadership roles. - Strong communication skills with the ability to translate complex technical concepts into business terms. Highly Desired Skills : - Experience with Azure Data Services, including Azure Data Factory, Azure Synapse, or similar tools. - Knowledge of scripting languages (e.g., Python, Scala) for data manipulation and automation. - Familiarity with DevOps practices, CI/CD pipelines, and containerization (Docker, Kubernetes) is a plu (ref:hirist.tech)

Location: in, IN

Posted Date: 12/27/2024