Techstar Software - Big Data Engineer - Spark/Hadoop

A Big Data Engineer Lead plays a critical role in managing and guiding the development of big data solutions within an organization. Here's a typical job description for this role. Job Title : Big Data Engineer Lead Exp : 5 Years Location : Hyderabad Notice Period : Immediate to 30days Job Summary : The Big Data Engineer Lead is responsible for overseeing the design, implementation, and management of large-scale data processing systems. This role involves leading a team of data engineers, collaborating with cross-functional teams, and ensuring the organization effectively utilizes big data technologies to derive actionable insights. Key Responsibilities : Team Leadership : - Lead and mentor a team of data engineers, fostering a collaborative and innovative work environment. - Provide technical guidance and support for team members in project execution Architecture Design : - Design and implement scalable big data architectures and solutions. - Ensure systems are optimized for performance and scalability. Data Processing : - Develop and maintain data pipelines for processing structured and unstructured data. - Utilize big data technologies (e.g., Hadoop, Spark, Kafka) to manage large datasets. Collaboration : - Work closely with data scientists, analysts, and stakeholders to understand data needs and requirements. - Communicate effectively with technical and non-technical teams. Quality Assurance : - Establish data governance and data quality standards. - Implement testing and validation processes for data integrity. Innovation and Strategy : - Stay updated with industry trends and emerging technologies in big data. - Propose and implement best practices for data engineering and analytics. Documentation and Reporting : - Create and maintain documentation for data architecture, processes, and systems. - Report on project progress and performance metrics to upper management. Qualifications : Education : - Bachelor's degree in computer science, Engineering, Data Science, or a related field; a Master's degree is a 5 years of experience in data engineering or related fields, with at least 2 years in a leadership role. - Proven experience with big data technologies (e.g., Hadoop, PySpark , Hive, NoSQL databases). Technical Skills : - Proficiency in programming languages such as Python or Java, and Scala. - Experience with Cloud Platform CI/CD Pipelines - Experience with cloud platforms (e.g., AWS, Azure, Google Cloud) and data warehousing solutions. Soft Skills : - Strong analytical and problem-solving abilities. - Excellent communication and interpersonal skills. - Ability to manage multiple projects and priorities effectively. Preferred Skills : - Familiarity with machine learning frameworks and tools. - Knowledge of data visualization tools (e.g., Tableau, Power BI). - Experience with DevOps practices and CI/CD pipelines. Work Environment : This position may require occasional overtime and the flexibility to adapt to changing project requirements. Remote or hybrid work options may be available. (ref:hirist.tech)

Location: hyderabad, IN

Posted Date: 11/27/2024