Senior QA Automation Engineer, AI & LLM Testing

Job Description

We are looking for a Senior QA Automation Engineer with strong hands-on experience in test automation, API testing, and AI/LLM quality evaluation.

• Design, develop, and maintain automated test suites for web applications, APIs, and AI-powered features.
• Evaluate LLM outputs for response accuracy, relevance, consistency, and hallucination risks.
• Create prompt tests and quality checks using Promptfoo or similar LLM evaluation frameworks.
• Build end-to-end UI automation using Playwright, following page object models and maintainable test architecture principles.
• Develop reusable Python scripts for API testing, automation utilities, and LLM quality evaluation.
• Create and maintain Postman collections for REST API testing, including environments, assertions, and CI/CD integration.
• Use GitHub Copilot or similar AI coding assistants to improve test coverage, speed up automation development, and identify quality gaps.
• Collaborate closely with product managers, developers, and stakeholders to define test strategies, communicate risks, and improve quality practices.
• Actively contribute to Agile ceremonies, including refinement sessions, sprint planning, retrospectives, and daily collaboration.

Qualifications

• Strong experience in QA Automation and software testing.
• Hands-on experience testing AI or LLM-based features, including prompt testing, output validation, response accuracy, and hallucination mitigation.
• Experience with Promptfoo or similar LLM evaluation tools/frameworks.
• Strong proficiency in Playwright for end-to-end UI automation.
• Solid knowledge of page object patterns, reusable test components, and scalable test architecture.
• Strong Python scripting skills for automation, API helpers, and custom quality checks.
• Experience with Postman and REST API testing, including collections, assertions, environments, and CI/CD integration.
• Practical experience using GitHub Copilot or similar AI coding assistants.
• Experience working in Agile environments such as Scrum or Kanban.
• Strong communication skills, with the ability to explain test strategies and quality risks to technical and non-technical stakeholders.

Nice to have:

• Experience with mobile test automation using Appium, XCUITest, or similar tools.
• Understanding of iOS and Android platform-specific testing considerations.
• Experience integrating automated tests into CI/CD pipelines.
• Familiarity with MCP or agent-assisted testing approaches in Playwright.

Additional Information

Project: a global consulting company delivering digital and AI solutions for businesses in financial services, healthcare, consumer goods, technology, energy, and travel.