Data Quality Engineer - Teradata/Azure Databricks

Job Summary : Seeking a Data Quality Engineer to ensure data accuracy and consistency in a Teradata to AWS Databricks migration. The role involves data validation, reconciliation, automation, and quality framework implementation. Key Responsibilities : 1. Data Validation & Testing : - Develop and execute data validation and reconciliation tests for relational databases. - Ensure data completeness, accuracy, and consistency during the Teradata to AWS Databricks migration. - Validate data extracted from mainframe systems and ensure correctness in Databricks. - Perform schema validation, anomaly detection, and root cause analysis for data inconsistencies. - Monitor and troubleshoot data transformation, ingestion, and processing pipelines to identify issues early. 2. Automation & Quality Frameworks : - Design and implement automated data quality checks to streamline validation processes. - Develop end-to-end automation solutions for data reconciliation using PySpark, Python, and SQL. - Create reusable data quality frameworks for ongoing monitoring and verification. - Ensure automated alerts and reporting mechanisms are in place for data discrepancies. 3. Collaboration & Best Practices : - Work closely with Data Engineers, ETL Developers, and Business Analysts to ensure smooth data migration. - Define data quality metrics, benchmarks, and governance standards. - Establish best practices for data validation in cloud-based environments (AWS Databricks). - Document test cases, automation scripts, and validation reports for auditing and compliance Required Skills : - SQL, PySpark, Python for data validation and automation. - Relational Databases, Teradata, AWS Databricks. - Automation tools and frameworks for data quality. - Mainframe data extraction validation (nice to have). (ref:hirist.tech)

Location: bangalore, IN

Posted Date: 4/20/2025