Idexcel Technologies - Big Data Engineer - Data Pipelines & Streaming Solutions

Job Description : Data Engineer We are looking for an experienced Data Engineer with strong expertise in building and managing data pipelines and streaming solutions. Mandatory Skills : Apache Spark : Strong expertise in Spark-based data processing. Python : Proficiency in developing and maintaining data pipelines. Databases & SQL : Hands-on experience in working with relational databases and writing optimized SQL queries. GCP Components : Experience with GCP tools like BigQuery, DataProc, Airflow, or Cloud Composer (at least one project). Preferred Skills : Apache Airflow : Working knowledge of orchestration and scheduling workflows. Kafka/Spark Streaming : Familiarity with real-time data streaming tools and concepts. Responsibilities : - Design, develop, and optimize scalable data pipelines using Apache Spark and Python. - Implement workflows and orchestration using Airflow. - Leverage GCP tools to process and analyze large-scale datasets. - Write and optimize complex SQL queries for data extraction and transformation. - Work with real-time data streaming solutions like Kafka or Spark Streaming (if required). Experience : Minimum 6 years in relevant technologies. Location : Open to remote or onsite roles depending on the project. (ref:hirist.tech)

Location: bangalore, IN

Posted Date: 11/29/2024