HelpMum Africa, a health-tech organization dedicated to reducing maternal and infant mortality, today announced the release of HelpMumMamaBench, a counterfactual benchmark built specifically to evaluate how AI models reason about maternal and paediatric diagnosis not just whether they get individual answers right, but whether that reasoning actually holds up when a case changes in a clinically meaningful way.
The benchmark is fully open-findings on some of today’s most advanced AI models
Why It Matters
Most existing medical AI benchmarks test one clinical question at a time, in isolation. That approach can hide a dangerous blind spot: a model may answer correctly on a standard case, then fail completely when a single decisive clinical detail shifts — even though, on paper, its accuracy score looks strong.
HelpMum MamaBenchwas built to expose exactly that gap. It consists of 434 clinical narratives, arranged in 217 matched pairs, spanning 371 maternal and paediatric conditions. Every narrative was individually authored from scratch by HelpMum Africa’s Medical team, written in first-person, patient-reported language rather than adapted from existing datasets. Each pair links a base clinical case to a counterfactual version, a case with the fewest possible changes needed to shift the correct diagnosis, testing whether an AI system’s reasoning is genuinely sound, or simply pattern-matching its way to a right answer.
What the Research Found
In everyday clinical evaluation, two patients can describe their symptoms in strikingly similar terms and still be facing entirely different, equally serious conditions; the correct diagnosis often turns on one or two decisive details buried in the rest of the presentation. HelpMum MamaBench is built to test exactly that: whether an AI system’s diagnosis holds up when a case is altered just enough to flip the correct answer, or whether the model was really just recognizing a familiar pattern rather than reasoning through the clinical evidence.
The research team put this to the test across eight configurations spanning four leading AI models. Each model was first scored on how often it got the “standard case” right, its base accuracy, and then scored again on how often it still got the right answer once a clinically decisive detail had shifted. Across every model tested, the gap between those two numbers ranged from 16 to 28 percentage points. In practical terms: a model that appeared roughly 80% accurate under conventional testing was, in some cases, barely above 60% reliable once its reasoning was actually put under pressure — and one widely capable model dropped from 77% apparent accuracy to just 47% once its answers were checked for consistency.
The team also tested whether a common technique called retrieval-augmented generation (RAG) — where an AI system pulls in reference material before answering, rather than relying solely on what it learned during training — would close this gap on its own. It didn’t. Models using standard RAG performed no better on the robustness test than comparable models without it, because the retrieval step tended to surface nearly identical reference material for both the standard case and its altered counterpart, giving the model no new signal to catch the shift.
To address that directly, the team introduced Evidence-Anchored RAG (EA-RAG), a new method that changes what gets retrieved rather than simply retrieving more. Instead of pulling in generally similar reference material, EA-RAG explicitly identifies the clinical details most likely to change a diagnosis, checks whether the retrieved evidence actually covers them, and fills any gaps before the model reasons through its answer. Applied to one of the strongest models in the study, EA-RAG reduced diagnostic failures on altered cases by more than a fifth, while leaving the model’s standard-case accuracy essentially unchanged.
Even with that improvement, the research team was candid about the limits of what’s been solved: the best-performing configuration in the entire study, the top model enhanced with EA-RAG — still got the wrong answer on roughly one in every five altered cases.
“We wanted a benchmark that told us the truth, even when the truth is uncomfortable,” said Dr. Abiodun Adereni, Founder of HelpMum Africa. “An AI system that looks accurate on a leaderboard but can’t be trusted to catch a life-threatening shift in a mother’s or child’s condition isn’t ready for real clinical use. HelpMum MamaBench exists to make that gap visible, and to push the whole field toward closing it.”
“This started as a question we couldn’t answer with existing benchmarks: does a model actually understand a diagnosis, or has it just memorized the shape of one? MamaBench is our attempt to ask that question properly — and EA-RAG is our first real attempt to answer it.” said Adewuyi Ayomide, Research Engineer at HelpMum Africa.
Open and Available Now
In keeping with HelpMum Africa’s approach to open, standards-based health innovation, MamaBench’s full dataset is released publicly and free to use with no restricted access, no paywall. The accompanying evaluation code is in final preparation and will be released shortly.
- Research paper:https://arxiv.org/abs/2607.14385
- Dataset:https://huggingface.co/datasets/HelpMum-Personal/MamaBench
