Data analytics application testing
A dashboard that renders is not a dashboard that is right. Most analytics bugs ship quietly, looking exactly like data.
Analytics products have a testing problem the rest of software mostly avoids: the output looks plausible either way. A checkout that fails is obvious. A revenue chart that is 4% low because the pipeline drops rows with null country codes just looks like a slow month. Nobody files a bug against a number that seems believable.
So we test analytics with known answers. Fixture datasets, small enough to total by hand, go through the full pipeline, and every chart, rollup and export must reproduce the numbers we computed independently. It is unglamorous arithmetic. It is also how you catch the classics: nulls counted as zeros, timezone shifts moving revenue across midnight, a filter that quietly applies to one panel but not its neighbour.
Daylight saving deserves special mention. Twice a year, daily aggregates get a 23-hour and a 25-hour day, and any rollup written with naive dates puts a slice of one day's activity into the next. If your customers span time zones, we test the boundaries where 'today' means three different things.
We also chase disagreement between surfaces. The dashboard says one figure, the CSV export another, the API a third, usually because each applies filters or sampling at a different stage. Your users will spot this eventually, in a board meeting, with less patience than we have.
Fair warning: this kind of testing is slower to set up than clicking around the UI, and it cannot repair what poor data quality upstream has already destroyed. Rubbish in, confidently presented rubbish out. But once the fixture packs exist they run on every release, and the numbers people quote in that board meeting stay worth quoting.
Known-answer testing
Fixture datasets with hand-computed totals run through the full pipeline on every release.
Time & timezone logic
DST boundaries, week starts and differing definitions of 'today' tested across zones.
Cross-surface consistency
Dashboards, CSV exports and APIs checked against the same expected numbers.
Filters & drill-downs
Filter combinations, sampling and drill-down paths verified against source data.
Frequently asked questions
How do you test whether a dashboard's numbers are actually correct?
By computing them a second way. A dashboard that renders beautifully and loads fast can still be wrong, and wrong-but-plausible numbers are the most dangerous defect class in analytics because people make decisions on them. We reconcile displayed figures against the source data independently: filters, date boundaries, timezone handling and aggregation logic all checked against a known dataset with known answers.
What are the most common bugs you find in analytics products?
Off-by-one date ranges (does 'last 30 days' include today?), timezone drift that splits one day's events across two, double-counting after joins, filters that silently drop null values, and totals that don't match their own breakdowns. None of these crash anything. All of them erode the only thing an analytics product sells: trust in the numbers.
Can you test data pipelines, not just the front end?
Yes, at the contract level: given a known input dataset, does each stage produce the expected output, and what happens with late-arriving data, duplicates, schema changes and malformed records? Deep platform engineering on your specific stack stays with your team; what we bring is systematic adversarial input and independent verification that the end-to-end story holds.
How do you test performance with large datasets?
With generated data at realistic scale, because a dashboard that's instant with 10,000 rows can time out at 10 million, and your best customer is by definition the one with the most data. We test query performance, pagination and export behaviour at the volumes your top-decile accounts will reach next year, not the volumes your demo account holds today.
Why do my dashboards show different numbers for the same metric?
Usually one of three causes: different metric definitions baked into different reports, different refresh times (a nightly warehouse against a live CRM), or an untested transformation change upstream. None of these fix themselves; they need agreed metric definitions and regression tests that compare outputs across tools. Once two 'correct' numbers circulate, stakeholders stop trusting both, so treat the first mismatch as a defect rather than a quirk.
How do you stop upstream schema changes breaking dashboards?
You cannot stop source teams changing things, but you can stop the breakage being silent. Data contracts or schema tests at the ingestion boundary fail loudly the moment a column is renamed or retyped, before bad data propagates into reports. Teams without this guard report upstream changes routinely breaking core pipelines; teams with it turn a week of firefighting into a same-day fix.
Why does nobody react to our failing data quality tests?
Alert fatigue. Hundreds of undifferentiated test failures train a team to ignore all of them, which is worse than having no tests. The fix is stratification: a small set of critical checks that block the pipeline, warnings for the rest, and deletion of any check nobody has ever acted on. An independent review of your test suite is often the quickest way to separate signal from noise.
Ready to improve your software quality?
Tell us about your product and we'll get back to you with a plan.
Contact Us