Senior Data Engineer - ETL/Python

We are seeking a highly skilled Senior Data Engineer to join our growing team. You will play a critical role in building and scaling our data platform to support complex data processing and analytics needs. This role requires expertise in designing and developing scalable systems that handle large volumes of batch and real-time data, enabling data-driven insights across the : - Architect, build, and maintain scalable data platforms for batch and real-time data processing. Optimize system performance and cost efficiency. - Develop and manage ETL/ELT pipelines for data ingestion, transformation, and distribution; ensuring scalability, resilience, and performance optimization. - Implement efficient data lakes and warehouses with modern formats like Hudi, Iceberg, or Delta Lake. - Design and implement real-time data processing solutions using Apache Kafka and Apache Flink. - Implement monitoring solutions and data quality frameworks to ensure data accuracy, reliability, and security. - Proactively identify and resolve performance bottlenecks. - Collaboration and Cross-Team Support: Work closely with data scientists, analysts, and product teams to build scalable data solutions. - Data Security and Compliance: Champion data security, integrity, and compliance measures. - Technical Leadership: Provide mentorship to junior engineers, and promote best practices in system design, coding, and deployment. - Automation and Infrastructure as Code (IaC): Automate infrastructure management using tools like Terraform and Ansible. Requirements : - 5 years of hands-on experience in data platform engineering. - Expertise in cloud environments (AWS preferred) and distributed processing frameworks (Apache Spark, Presto). - Strong SQL and Data Modeling experience. - Experience with DBT for data transformations and modeling. - Knowledge of modern data lake architectures and table formats (Hudi, Iceberg, Delta Lake). - Proficiency in Python and/or Java for developing scalable solutions. - Strong experience in real-time streaming technologies (Kafka, Flink). - Performance tuning and incremental improvement experience. - Excellent communication and leadership skills. - Bachelor's/Master's degree in Computer Science, Engineering, or a related field. - Experience building data platforms from scratch, particularly in startups. - Familiarity with DataOps and CI/CD pipelines in data engineering. - Hands-on experience with Terraform or Ansible for cloud infrastructure management. (ref:hirist.tech)

Location: bangalore, IN

Posted Date: 4/18/2025