Big Data Engineer - Python/PySpark

Job Summary : We are seeking a highly skilled and motivated Big Data Engineer to join our dynamic team. The ideal candidate will have hands-on experience in designing, developing, and optimizing scalable big data pipelines and solutions. This role requires proficiency in advanced big data tools, technologies, and programming languages. Key Responsibilities : - Design and implement scalable data processing pipelines using PySpark, Spark, and other big data technologies. - Develop and optimize data workflows for batch and real-time processing. - Implement and manage data lakes and data warehouses using Hive. - Work with large-scale distributed systems, particularly Hadoop ecosystems. - Write efficient and maintainable code using Scala and other relevant programming languages. - Collaborate with data scientists, analysts, and business stakeholders to deliver robust solutions. - Monitor, troubleshoot, and improve the performance of big data platforms. - Ensure data security, quality, and governance standards are met. Required Skills and Qualifications : - Strong expertise in PySpark, Spark, Scala, Hive, and Hadoop. - Proficiency in programming languages like Python and Java/Scala. - Solid understanding of distributed computing principles and big data ecosystems. - Experience with big data tools such as Kafka, Airflow, or Oozie (preferred). - Hands-on experience with cloud platforms like AWS, Azure, or Google Cloud (optional but a plus). - Familiarity with SQL and NoSQL databases. - Strong problem-solving skills and ability to optimize complex data pipelines. (ref:hirist.tech)

Location: in, IN

Posted Date: 11/28/2024