Big Data Engineer - Azure Data Services

Job Title : Big Data Engineer Location : Chennai, Bangalore, Hyderabad Key Responsibilities : - Design, develop, and implement data pipelines, leveraging Azure Data Factory, Azure Synapse Analytics, Azure SQL, and Azure Data Lake. - Create scalable, efficient, and robust data solutions to manage data ingestion through APIs and streaming data sources, optimizing performance and reliability. - Develop and deploy cloud-based analytics and data solutions to support various business intelligence and data analysis requirements. - Build infrastructure for data analysis and machine learning workflows using best practices in Azure cloud environments. - Build, test, and maintain efficient data pipelines to collect, manage, and transform large datasets from diverse sources, ensuring data quality and accessibility. - Use SQL scripting and Stream Analytics to handle real-time and batch processing requirements effectively. - Implement DevOps processes including CI/CD and Infrastructure as Code (IaC) to streamline development, testing, and deployment of data solutions. - Collaborate with DevOps teams to integrate automation and version control in data pipeline processes. - Develop and enforce security models to protect sensitive data within the cloud environment. - Ensure compliance with data privacy regulations and implement robust security protocols for all data solutions. - Analyze complex data sets to support decision-making and derive actionable insights. - Work closely with data analysts and business teams to understand requirements and troubleshoot any data issues promptly. Required Skills & Qualifications : - 5 years in IT service delivery with experience working in global onshore and offshore teams. - 3 years of hands-on experience in the Azure Analytics space, including Azure Data Factory, Azure Synapse, Azure SQL, and Data Lake. - Strong knowledge of SQL scripting and familiarity with Python or Spark-SQL for data processing tasks. - Experience with data ingestion, transformation, and pipeline development using API and streaming methods. - Familiarity with DevOps tools and practices, including CI/CD and infrastructure management. - Ability to develop and implement secure data models in cloud environments, with a focus on data protection and regulatory compliance. - Excellent data analysis, problem-solving, and troubleshooting skills, with attention to detail and a strong analytical mindset. - Ability to work effectively in a fast-paced, agile development environment and collaborate with cross-functional teams. - Certified as an Azure Data Engineer or higher, which is a plus. (ref:hirist.tech)

Location: in, IN

Posted Date: 11/21/2024