Data Engineer - Python/ETL/SQL

Key Responsibilities : - Design, develop, and maintain robust and scalable data pipelines using Python/PySpark and Azure services. - Implement ETL/ELT processes to ingest, transform, and load data from various sources into our data lake and data warehouse. - Automate data pipelines using Azure Data Factory and other orchestration tools. - Design and implement efficient data models to support business requirements. - Optimize data structures for performance and scalability. - Perform data analytics tasks to provide insights and support business decision-making. - Develop reports and dashboards to visualize data and communicate findings. - Leverage Azure cloud services such as Azure Databricks, Azure Data Factory, Azure Data Lake Storage, and Azure DevOps to build and maintain our data platform. - Optimize Azure resources for cost-effectiveness and performance. - Implement and maintain CI/CD pipelines using Azure DevOps to automate code deployment and testing. - Ensure code quality and adherence to best practices. - Implement and maintain data security practices and methods, including API/web service security. - Ensure compliance with data security policies and regulations. - Work with relational databases such as SQL Server, Oracle, or MySQL. - Write and optimize SQL queries for data access and manipulation. - Develop interactive data applications and dashboards using Streamlit or RShiny (preferred). - Collaborate effectively with other engineers, analysts, and stakeholders. - Communicate clearly and concisely, both orally and in writing. - Troubleshoot and resolve data-related issues. - Identify and implement solutions to improve data quality and performance. Required Skills and Qualifications : Experience : 5 years of experience in data engineering. Technical Skills : - Strong proficiency in Python/PySpark and SQL. - Expertise in Azure cloud services, including Azure Databricks, Azure Data Factory, Azure Data Lake Storage, and Azure DevOps. - Knowledge of CI/CD principles and tools. - Good understanding of API/web service security and data security practices. - Experience working with relational databases (SQL Server, Oracle, MySQL) (ref:hirist.tech)

Location: in, IN

Posted Date: 4/18/2025