Senior ETL Developer - AWS Cloud Services

Job Summary : We are seeking an experienced Senior ETL Developer with expertise in designing, implementing, and maintaining ETL processes using AWS services, specifically - AWS Glue- and - EventBridge- . The ideal candidate will play a key role in transforming and integrating complex data systems, ensuring data quality, and optimizing performance across cloud-based infrastructures. Responsibilities : - Design and Develop ETL Pipelines : Build scalable, efficient ETL workflows using AWS Glue to extract, transform, and load data from various sources into data lakes or warehouses (e.g, Amazon S3, Redshift). - Event-Driven Architecture : Implement event-driven data processing solutions using AWS Event Bridge to trigger ETL workflows and automate data flows across systems. - Data Integration : Work on data ingestion, transformation, and orchestration by integrating data from multiple sources, including structured and unstructured datasets. - Performance Optimization : Monitor and fine-tune ETL jobs for performance, reliability, and scalability, ensuring low-latency and efficient data processing. - Data Quality and Governance : Implement robust data quality checks, validations, and data governance best practices within the ETL processes. - Troubleshooting and Maintenance : Diagnose and resolve issues related to ETL processes, cloud infrastructure, and data pipelines, ensuring high system availability. - Automation and Orchestration : Leverage AWS Lambda, Step Functions, and Event Bridge to automate and schedule data pipelines in a cost-effective manner. - Collaboration with Data Teams : Work closely with data architects, engineers, and analysts to understand data requirements and deliver solutions that align with business goals. - Documentation and Reporting : Maintain clear documentation of ETL workflows, data lineage, and technical processes for auditing and transparency purposes. Qualifications : - 4 years of experience in ETL development, with a focus on cloud-based services. - Expertise in AWS Glue for creating and managing ETL pipelines, utilizing Glue Data Catalog, Crawlers, and Glue Jobs. - Proficient with AWS Event Bridge for event-driven architecture and data processing. - Strong experience with AWS Lambda, Step Functions, and CloudWatch for orchestration and monitoring. - Experience with Python and PySpark for custom ETL scripting and transformations. - Familiarity with AWS S3, Redshift, and RDS for data storage and processing. - Knowledge of SQL for querying and data manipulation. - Strong problem-solving skills and ability to troubleshoot ETL processes and data integration issues. - Experience with data governance, data quality checks, and compliance in data processing. - Excellent communication skills, with the ability to collaborate effectively across teams. Note : - Looking for less notice period candidates. (ref:hirist.tech)

Location: andhra-pradesh, IN

Posted Date: 11/24/2024