India Infoline - Lead Data Engineer - Azure/Databricks

Key Responsibilities : - Data Lakehouse Development : Lead the design and implementation of our data lakehouse architecture using Azure and Databricks. - Data Modeling : Develop conceptual, logical, and physical data models utilizing star schema methodologies. - Pipeline Optimization : Build and optimize data pipelines using PySpark and Databricks, ensuring high performance and scalability. - Data Governance : Implement data governance frameworks and ensure compliance with data quality standards. - Tool Integration : Leverage Azure Data Factory (ADF) and Change Data Capture (CDC) mechanisms for efficient data ingestion and processing. - Unity Catalog Management : Utilize Unity Catalog for metadata management and data cataloging. - Team Leadership : Mentor junior engineers and collaborate with cross-functional teams to drive data initiatives. - Best Practices : Stay updated with the latest industry trends and integrate best practices into the data engineering & Experience : Minimum of 7 years in data engineering with at least 4 years focused on building data lakehouse. - Azure & Databricks Expertise : 3-4 years of hands-on experience with Azure services and Databricks platform. Technical Skills : Proficient in ADF, CDC, PySpark, and Unity Catalog. - Data Modeling : Strong knowledge in creating conceptual, logical, and physical data models, specifically using star schema. - Data Governance & Quality : Demonstrated experience in implementing data governance frameworks and ensuring data quality. - Pipeline Development : Expertise in building optimized data pipelines on Databricks. - Leadership Skills : Proven ability to lead projects and mentor team members. Education : Bachelor's or Master's degree in Computer Science, Information Systems, or a related field. Preferred Qualifications : Certifications : Azure or Databricks certifications. Soft Skills : Excellent communication, problem-solving abilities, and attention to detail. (ref:hirist.tech)

Location: in, IN

Posted Date: 11/24/2024