Case study · AI QA and API testing
Making a confidential AI platform safer to ship
A fast-moving AI startup needed practical QA support without slowing product delivery. testology combined API automation, manual testing, regression checks, and AI-focused review so the team could keep releasing with clearer evidence and fewer unknowns.
- Client
- Confidential AI platform
- Engagement
- Freelance QA support, 10 to 20 hours per month
- Focus
- API testing, manual testing, regression, AI workflow assurance
- Delivery
- BDD scenarios, automated checks, reproducible defect reports
The challenge
The client was moving quickly, adding new API-backed features and AI-assisted workflows in a product where trust matters. They needed flexible QA help that could fit around a startup cadence: enough structure to protect releases, but not a heavy process that got in the way.
APIs carried the real product logic
Many important risks lived below the interface: permissions, response shape, data updates, error handling, and the way services behaved under ordinary user journeys. UI checks alone would not have been enough.
AI behaviour needed human judgement
For the AI implementation, the work could not rely on exact string matching. testology reviewed whether outputs were useful, consistent with the workflow, and safe enough for the context in which they appeared.
What testology delivered
The work combined repeatable automation with hands-on exploratory testing. Automated checks covered the stable, high-value paths; manual testing covered the places where product judgement, AI behaviour, and user experience still needed human eyes.
- 98
- BDD scenarios reviewed across the test assets
- 187
- API interactions exercised by automated scenarios
- 86
- explicit response-time checks in the suite
- 15
- negative or permission-focused checks
- 11
- schema and contract-style validations
Scenario coverage by area
BDD scenarios were spread across core workflows, platform behaviour, access control, AI-facing flows, and supporting services. Areas are anonymised.
API methods exercised
The automation covered read, create, update, patch, and delete behaviour rather than checking only happy-path responses.
Quality signals built into the suite
Checks were not limited to “does it return 200?”. The suite also captured speed, contract shape, and failure behaviour.
Automation and human judgement
The engagement balanced fast, repeatable checks with manual testing where AI behaviour and user experience needed context.
- API automation
- Regression testing
- Manual exploratory testing
- AI behaviour review
The approach
Map the risk
Identify the workflows most likely to damage trust if they broke: access, data changes, service responses, AI-assisted outputs, and high-use user journeys.
Automate the stable checks
Build BDD-style API scenarios around repeatable behaviours, including response status, response body, schema shape, and basic performance expectations.
Test the AI manually
Review AI implementation behaviour with human judgement: whether the feature produced useful outputs, handled edge cases, and stayed aligned with the intended workflow.
Feed releases with evidence
Document bugs clearly, retest fixes, and keep regression checks current so the team could move quickly without losing sight of quality risk.
The outcome
The client gained a QA layer that matched the pace of a high-growth startup: lean enough to run part-time, structured enough to protect important releases, and practical enough for engineers to act on. Part-time has limits, though. At 10 to 20 hours a month not every release got a full pass, so the hours went to the flows most likely to hurt if they broke.
The goal was not to create a bigger test suite. It was to create a test suite the team could trust.
More reliable API releases
Repeatable checks helped catch broken contracts, unexpected response changes, permission issues, and regression risks earlier in the delivery cycle.
Clearer AI quality feedback
Manual review added judgement where automation alone was too brittle, especially around AI outputs, edge cases, and feature behaviour.
Better developer handoff
Defects were documented with enough context to reproduce, prioritise, fix, and retest without long back-and-forth conversations.
Building AI features into your product?
We test the APIs, workflows, prompts, and edge cases that decide whether users trust what you ship. We find it before your users do.