Senior Data Engineer - Python/PySpark

Job Description : - Build data pipeline between source systems (RDS/SQL Server) to Snowflake on AWS - ETL data into AWS S3 - ELT Data into Snowflake - Realtime data integration through CDC and AWS managed Kafka streams. - Embedded Tableau reporting - Data transformations using SQL, Python, PySpark, and DBT - Through understanding of data engineering principles including data lakes, and data warehouse - Data orchestration using Apache Airflow - Data Analysis skills - Best practices in data management, data quality and data stewardship were applied to ensure data integrity and reliability. - Optimize data pipelines for productivity and efficiency by considering factors such as scalability, reliability, and cost effectiveness. - Troubleshoot and resolve data infrastructure and pipelines in a timely manner. - Bachelor's or master's degree in computer science, Engineering, or a related field. - Proven experience (6-10 years) as a Data Engineer - Proficiency in SQL for data extraction, transformation, and analysis across various database systems. - Strong programming skills in Python for data manipulation, scripting, and building data engineering solutions. (ref:hirist.tech)

Location: hyderabad, IN

Posted Date: 11/27/2024