Cloud Analogy - Data Engineer - Python

Job Overview : We are looking for an experienced Data Developer to join our team. The ideal candidate will have a strong background in data development, with expertise in Python and various cloud technologies. This role requires hands-on experience in building data pipelines and designing data management solutions that leverage both relational and non-relational databases. The successful candidate will be a curious technology enthusiast who understands the intricacies of cloud computing, microservices, and streaming technologies. Key Responsibilities : 1. Data Pipeline Development : - Design, build, and maintain scalable data pipelines in a cloud environment using tools such as Azure Data Lake Storage, Azure Event Hubs, and Kafka. - Develop data ingestion processes for structured and unstructured data, ensuring data quality and integrity. 2. Database Management : - Work with SQL Server, Cosmos DB, and Mongo DB to design and optimize data management solutions. - Create and maintain database schemas, indexes, and stored procedures to enhance performance and data retrieval. 3. Data Solutions Design : - Design and develop data persistence solutions that cater to various application use cases, focusing on relational and non-relational databases. - Enhance data processing capabilities by implementing best practices and leveraging advanced technologies. 4. Streaming Technologies : - Utilize Spark Streaming and other streaming technologies to handle real-time data processing and event-driven architectures. - Collaborate with teams to ensure effective integration of streaming solutions with existing systems. 5. Cloud Computing : - Leverage cloud services, particularly Azure, to support data storage, processing, and analytics needs. - Stay current with emerging cloud technologies and trends to continuously improve data solutions. 6. Collaboration and Leadership : - Work closely with cross-functional teams to gather requirements and ensure successful project delivery. - Act as a hands-on engineering lead, mentoring junior developers and promoting best practices in data development. Required Skills and Qualifications : Experience : - Minimum of 4 years of experience as a data developer, with a strong focus on Python and data pipeline development. Technical Skills : - Proficient in Kafka, Spark Streaming, Azure SQL Server, Cosmos DB, and Mongo DB. - Hands-on experience with Azure services, including Azure Event Hubs and Azure Data Lake Storage. Data Management : - Solid understanding of relational and non-relational databases, with experience designing data models and persistence solutions. - Experience handling unstructured data and working within a data lake environment. Cloud and Microservices : - Familiarity with cloud computing concepts, especially Azure (preferred), AWS, and microservices architecture. - Knowledge of network and security best practices in cloud environments. (ref:hirist.tech)

Location: bangalore, IN

Posted Date: 11/21/2024