KrtrimaIQ Cognitive Solution - Data Engineer - Python/Spark/Hadoop

Job Description : We are looking for an Only immediate joiner and experienced Data Engineer with a strong background in Kafka, PySpark, Python/Scala, Spark, SQL, and the Hadoop ecosystem. The ideal candidate should have over 5 years of experience and be ready to join immediately. This role requires hands-on expertise in big data technologies and the ability to design and implement robust data processing solutions. Key Responsibilities: - Design, develop, and maintain scalable data processing pipelines using Kafka, PySpark, Python/Scala, and Spark. - Work extensively with the Kafka and Hadoop ecosystem, including HDFS, Hive , and other related technologies. - Write efficient SQL queries for data extraction, transformation, and analysis. - Implement and manage Kafka streams for real-time data processing. - Utilize scheduling tools to automate data workflows and processes. - Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions. - Ensure data quality and integrity by implementing robust data validation processes. - Optimize existing data processes for performance and scalability. Preferred Qualifications : - Experience with GCP - Knowledge of data warehousing concepts and best practices. - Familiarity with machine learning and data analysis tools. - Understanding of data governance and compliance standards. (ref:hirist.tech)

Location: bangalore, IN

Posted Date: 11/26/2024