AI Test Architect

About the Role

Qualitest is looking for an experienced AI Test Architect to join our growing AI & Data Testing practice. This is an exciting opportunity to define and implement the end-to-end testing strategy for a cloud-based, data-driven AI Ops platform.

You will play a key role in ensuring data quality, workflow reliability, automation robustness, and AI model accuracy — collaborating closely with engineering, data science, and DevOps teams to deliver high-performing, trusted AI solutions.

Key Responsibilities

• Define and implement comprehensive end-to-end test strategies for data ingestion, pipelines, workflows, AI models, and cloud deployments.

• Establish testing methodologies for data validation, workflow orchestration, model verification, and system integration.

• Drive adoption of automation-first approaches across CI/CD and MLOps pipelines.

• Develop validation frameworks for structured, semi-structured, and unstructured data.

• Implement automated checks for data integrity, transformation accuracy, schema compliance, lineage, and reconciliation.

• Test workflow orchestration tools such as Airflow, Step Functions, Lambda, and EventBridge.

• Collaborate with data science teams to design and execute model validation tests covering accuracy, drift, bias, and fairness.

• Architect automation frameworks and synthetic data generation strategies for AI/ML pipeline testing.

• Integrate testing within DevOps/MLOps workflows to ensure continuous quality and model reliability.

• Ensure performance, scalability, resilience, and security testing of cloud-native AI systems.

• Define and monitor quality KPIs and dashboards for data and AI testing outcomes.

• Provide technical leadership and mentoring to QA and engineering teams in AI testing best practices.

Must-Have Skills & Experience

• Strong foundation in software testing principles, test architecture, and automation frameworks.

• Hands-on experience with data testing, ETL/ELT pipelines, and big data environments.

• Good understanding of the AI/ML lifecycle — from model training and validation to deployment and monitoring.

• Practical experience in testing AI/ML models for accuracy, drift, explainability, and fairness.

• Proficiency in Python, PyTest, Robot Framework, or similar test automation tools.

• Familiarity with data quality tools such as Great Expectations, Soda, or Deequ.

• Experience working with AWS services — including S3, Glue, Lambda, SageMaker, Athena, Redshift, and Step Functions.

• Strong analytical and problem-solving skills.

• Excellent communication, documentation, and stakeholder management abilities.

• Proven ability to lead cross-functional quality initiatives in complex, fast-paced environments.

Why Qualitest?

• Be part of the world’s leading independent AI & quality engineering company.

• Work with cutting-edge technologies and enterprise-scale AI projects.

• Collaborate with global experts and innovative clients across multiple industries.

• Access continuous learning, upskilling, and certification opportunities.

• Enjoy a supportive culture that values excellence, innovation, and teamwork.