Data Engineer - SQL/Python

Role Overview : We are looking for a Data Engineer with at least 10 years of hands-on experience in building and optimizing data pipelines, data processing systems, and analytical tools. The ideal candidate will have deep technical knowledge in working with big data frameworks, especially Apache Spark, along with strong proficiency in SQL and Python. You will play a key role in designing, developing, and maintaining scalable data solutions that support various data-driven initiatives within the organization. Key Responsibilities : - Data Pipeline Development : Design, implement, and maintain end-to-end data pipelines to efficiently ingest, process, and store large datasets from various sources. - Data Processing : Work with large-scale datasets using Apache Spark for data processing and transformations. Optimize Spark jobs for performance and scalability. - SQL Development : Write efficient, optimized SQL queries for data extraction, transformation, and analysis. Work with large, complex datasets and databases. - Python Scripting : Develop Python scripts and automation tools to support data engineering tasks, such as data validation, cleaning, and integration. - Cloud Integration : Leverage cloud platforms (AWS, Azure, GCP) for data storage, processing, and scaling. Familiarity with cloud data lakes, data warehouses, and other cloud-based services. - Collaboration & Reporting : Collaborate with data scientists, business analysts, and other stakeholders to understand business requirements and deliver data solutions. - Performance Optimization : Ensure that data systems are scalable, reliable, and efficient. Continuously monitor and improve performance of data pipelines and processes. - Best Practices & Documentation : Maintain clear, up-to-date documentation of data workflows, data architecture, and related systems. Adhere to coding and process standards. - Troubleshooting & Support : Troubleshoot issues within data pipelines and resolve them quickly to ensure smooth data operations. Qualifications : Education : Bachelor's or Master's degree in Computer Science, Information Technology, Data Engineering, or a related field. Experience : - 10 years of hands-on experience in Data Engineering or related fields. - Strong expertise with Apache Spark for distributed data processing and data pipeline development. - Advanced proficiency in SQL (e.g., PostgreSQL, MySQL, or similar database systems), including complex joins, aggregations, and performance tuning. - Proficient in Python for data engineering tasks, such as scripting, automation, and working with APIs. - Experience working with Big Data Technologies such as Hadoop, Hive, or other distributed computing frameworks. - Familiarity with Cloud platforms (AWS, Azure, GCP) for managing data solutions. Skills : - Deep understanding of data structures, algorithms, and data processing workflows. - Strong knowledge of data modeling, ETL (Extract, Transform, Load) processes, and database management. - Expertise in optimizing data pipelines for performance, scalability, and cost-efficiency. - Ability to work with version control systems (e.g., Git) and code review practices. - Excellent problem-solving and debugging skills. - Ability to work in a fast-paced, collaborative environment with cross-functional teams. Nice to Have : - Experience with Data Warehousing (e.g., Redshift, Snowflake). - Experience with Kafka or other streaming platforms. - Familiarity with Containerization (e.g., Docker, Kubernetes). (ref:hirist.tech)

Location: noida-gautam-buddha-nagar, IN

Posted Date: 11/28/2024