Experis
Lead Data Engineer
Job Location
Farringdon, United Kingdom
Job Description
Lead Data Engineer
Location: Remote
6 months
£(Apply online only) per day - Umbrella only
Key Responsibilities:
Lead Data Pipeline Development: Design and optimize end-to-end data pipelines using BigQuery, Google Cloud Storage, Dataproc, Composer (Airflow), and PySpark.
Cloud Architecture & Solution Design: Architect secure and scalable data solutions on GCP, ensuring alignment with business needs.
Collaboration & Leadership: Mentor data engineers, work with cross-functional teams, and translate business requirements into technical solutions.
Optimization & Troubleshooting: Optimize data processing for performance and cost-efficiency.
Continuous Innovation: Stay up-to-date with GCP technologies, implementing best practices and new tools for pipeline improvement.
Skills & Experience Required:
IT experience: 10+ years of experience in IT and development.
Programming: 5+ years of experience in Python, SQL, PySpark, or SparkSQL.
GCP Experience: 3+ years with BigQuery, Google Cloud Storage, Dataproc, Composer (Airflow), and Cloud Pub/Sub.
Data Engineering: Expertise in ETL pipelines, data transformation, and data governance.
Leadership: Experience leading development teams, mentoring engineers, and managing technical projects.
DevOps & Agile: Familiar with DevOps practices, CI/CD pipelines, and tools like Jira, Azure DevOps, and GitHub.
Desirable:
Certifications such as Google Cloud Professional Data Engineer are a plus.
All profiles will be reviewed against the required skills and experience. Due to the high number of applications we will only be able to respond to successful applicants in the first instance. We thank you for your interest and the time taken to apply
Location: Farringdon, GB
Posted Date: 11/15/2024
Location: Remote
6 months
£(Apply online only) per day - Umbrella only
Key Responsibilities:
Lead Data Pipeline Development: Design and optimize end-to-end data pipelines using BigQuery, Google Cloud Storage, Dataproc, Composer (Airflow), and PySpark.
Cloud Architecture & Solution Design: Architect secure and scalable data solutions on GCP, ensuring alignment with business needs.
Collaboration & Leadership: Mentor data engineers, work with cross-functional teams, and translate business requirements into technical solutions.
Optimization & Troubleshooting: Optimize data processing for performance and cost-efficiency.
Continuous Innovation: Stay up-to-date with GCP technologies, implementing best practices and new tools for pipeline improvement.
Skills & Experience Required:
IT experience: 10+ years of experience in IT and development.
Programming: 5+ years of experience in Python, SQL, PySpark, or SparkSQL.
GCP Experience: 3+ years with BigQuery, Google Cloud Storage, Dataproc, Composer (Airflow), and Cloud Pub/Sub.
Data Engineering: Expertise in ETL pipelines, data transformation, and data governance.
Leadership: Experience leading development teams, mentoring engineers, and managing technical projects.
DevOps & Agile: Familiar with DevOps practices, CI/CD pipelines, and tools like Jira, Azure DevOps, and GitHub.
Desirable:
Certifications such as Google Cloud Professional Data Engineer are a plus.
All profiles will be reviewed against the required skills and experience. Due to the high number of applications we will only be able to respond to successful applicants in the first instance. We thank you for your interest and the time taken to apply
Location: Farringdon, GB
Posted Date: 11/15/2024
Contact Information
| Contact | Human Resources Experis |
|---|