Data Architect - Python/PySpark

Must Have : - 8 years of overall IT experience and 3 years as a data Architect. - Development experience in the core tools and technologies like SQL, Python, AWS (Lambda, Glue, S3, Redshift, Athena, IAM Roles & Policies) , PySpark used by the solution services team. - Design end-to-end data architectures using Databricks, ensuring they meet scalability, reliability, and security requirements. - Manage and optimize Databricks clusters, workspaces, and notebooks to ensure efficient processing and storage. - Implement best practices for data governance, security, and compliance on the Databricks platform. - Provide technical leadership and guidance on Databricks and related big data technologies, mentoring junior team members. - Design and oversee the implementation of data integration processes to ingest, clean, and transform data from various sources into the Databricks environment. - Monitor and tune data architecture and Databricks performance to handle large-scale data workloads. - Implement strategies for cost optimization and efficient resource utilization within the Databricks environment. - Stay up-to-date with the latest Databricks features and industry trends to continuously improve data architectures. - 3 years of experience in Agile Development and code deployment using Github & CI-CD pipelines. - 2 years of experience in job orchestration using Airflow. - Expertise in the design, data modelling, creation, and management of large datasets/data models - Ability to work with business owners to define key business requirements and convert to technical specifications - Ensure successful transition of applications to service management team through planning and knowledge transfer - Leads cross-functional product development teams, acting also as a senior consultant to provide system and technical advice. - Partner with and influence vendor resources on solution development to ensure understanding of data and technical direction for solutions as well as delivery - Excellent problem-solving skills with a strong attention to detail. - Effective communication skills, with the ability to explain complex technical concepts to non-technical stakeholders. (ref:hirist.tech)

Location: bangalore, IN

Posted Date: 11/22/2024