HiLabs - Data Engineer - Spark/Python

Job Description : Key Responsibilities : Data Pipeline Development : - Design, develop, and optimize scalable ETL (Extract, Transform, Load) processes to ingest and process data from diverse sources. - Implement real-time and batch data processing systems using tools like Apache Spark, Kafka, or similar technologies. Data Warehousing : - Build and maintain data warehouses/lakes for structured and unstructured data, leveraging platforms such as AWS Redshift, Snowflake, or Google BigQuery. - Design and implement database schemas to ensure efficient querying and data retrieval. Data Quality and Governance : - Ensure data quality by implementing monitoring, validation, and auditing frameworks. - Collaborate on data governance strategies to maintain compliance with organizational and regulatory Optimization : - Optimize database performance and query execution for large-scale datasets. - Implement caching and indexing strategies to enhance system responsiveness. Tools and Technology : - Develop solutions using programming languages such as Python, Scala, or Java. - Leverage cloud platforms (AWS, Azure, or GCP) for scalable and secure data storage and processing. Required Qualifications : Education : Bachelor's or Master's degree in Computer Science, Engineering, or a related field. Experience : - 4-6 years of professional experience in data engineering, data architecture, or a related role. - Hands-on experience with distributed computing frameworks such as Hadoop, Spark, or Kafka. - Strong proficiency in SQL and experience with database management systems (e., PostgreSQL, MySQL, or Oracle). - Expertise in at least one programming language (Python, Scala, or Java). - Familiarity with version control tools like Git and CI/CD pipelines. - Experience with cloud services like AWS (S3, EMR, Glue), Azure (Data Factory, Synapse), or GCP (BigQuery, Dataflow). Soft Skills : - Excellent problem-solving abilities and analytical skills. - Strong communication skills to interact effectively with technical and non-technical stakeholders. Preferred Qualifications : - Certification in cloud technologies (AWS Certified Data Analytics, Google Professional Data Engineer, or Azure Data Engineer Associate). - Knowledge of big data tools like Apache Hive, Presto, or Airflow. - Experience with data visualization tools like Tableau or Power BI (ref:hirist.tech)

Location: pune, IN

Posted Date: 12/26/2024