Consensus Built on Quicksand: How Aggregating Flawed Research Hardens Error Into Fact

There is a particular kind of intellectual confidence that emerges not from the strength of a single finding, but from the apparent convergence of many. When dozens of studies point in the same direction, when a meta-analysis synthesizes thousands of participants across multiple continents, the conclusion feels secure — not merely probable, but established. It is precisely this confidence that makes the problem so difficult to diagnose, and so consequential when it goes unaddressed.

The crisis being described here is not the familiar story of a single fraudulent paper or a sensationalized headline that outpaced its evidence. It is something structurally deeper: the possibility that the very instruments science built to correct individual errors — systematic reviews, meta-analyses, evidence hierarchies — are, under certain conditions, machines for compounding them.

The Aggregation Assumption

The logic behind meta-analysis is genuinely compelling. Any single experiment is vulnerable to idiosyncratic noise — a peculiar sample, an unusual measurement environment, a researcher's unconscious choices. Pool enough studies together, the reasoning goes, and those individual quirks cancel out, leaving behind a signal that no single laboratory could reliably detect on its own.

This logic holds, but only under a critical assumption: that the errors distributed across individual studies are random. If the distortions are systematic — if they share a common directional bias — then aggregation does not average them away. It concentrates them. A meta-analysis of twenty studies, each nudged in the same direction by the same publication pressures, the same measurement conventions, or the same theoretical priors, does not produce a cleaner picture of reality. It produces a more authoritative-looking version of the original distortion.

Researchers have a name for this: garbage in, garbage out. But the phrase undersells the severity of what happens at scale, because garbage entering a meta-analysis does not exit labeled as such. It exits wearing the imprimatur of synthesis.

The Citation Cascade

Consider how a moderately flawed study moves through the scientific literature. Published in a reputable journal, it accumulates citations — some from researchers who engaged with it critically, many from those who encountered it only through a secondary summary or an abstract. Each citation is a small vote of confidence, and over time, those votes accrue into something that functions socially like established fact.

When a meta-analyst then surveys the field, that study appears not as a contested finding but as a data point. Its methodological weaknesses, if they were ever publicly noted, are rarely carried forward into the aggregated record. The meta-analysis reports an effect size and a confidence interval. Downstream readers — clinicians, policymakers, science journalists writing for audiences in Chicago or Sacramento — encounter the conclusion stripped of its genealogy.

This is the citation cascade: a process by which a finding's influence grows faster than scrutiny of its foundations. In fields where empirical research directly informs practice — nutrition science, social psychology, certain corners of clinical medicine — the downstream consequences are not merely academic. Dietary guidelines have been shaped by meta-analyses whose constituent studies shared systematic biases. Behavioral interventions have been scaled nationally on the basis of effect sizes that evaporated when independent teams attempted to reproduce them.

Heterogeneity and the Illusion of Coherence

One of the more technically sophisticated problems within meta-analytic practice involves heterogeneity — the degree to which individual studies are actually measuring the same thing. Two experiments nominally investigating "the effect of sleep deprivation on cognitive performance" may operationalize both the intervention and the outcome in ways that are, at a mechanistic level, barely comparable. One study uses a twelve-hour deprivation protocol in a controlled laboratory setting; another uses self-reported sleep duration collected via smartphone survey.

Statistical tests exist to detect heterogeneity, and responsible meta-analysts apply them. But the incentive structure of scientific publishing rewards coherent narratives over complicated ones. A meta-analysis that concludes "the evidence is deeply inconsistent and the effect may not be real" faces a steeper path to publication than one that reports a clean pooled effect. The result is a literature that systematically underreports the degree to which its component studies are talking past one another.

When heterogeneity is masked rather than addressed, the meta-analysis presents a false unity. The pooled effect size is a mathematical artifact — the average of apples, oranges, and the occasional persimmon — rather than an estimate of any real-world quantity.

The Feedback Loop Between Theory and Evidence

Perhaps the most philosophically troubling dimension of this problem involves the relationship between theoretical expectation and empirical selection. Researchers conducting systematic reviews must make decisions at every stage — which studies to include, how to handle outliers, which statistical models to apply. These decisions are rarely arbitrary; they are guided, consciously or not, by prior beliefs about what the evidence should show.

This is not misconduct. It is cognition. But it introduces a feedback loop in which theoretical priors shape the evidence base that is then used to confirm those priors. A field that collectively believes a particular mechanism is real will, through entirely defensible individual choices, tend to construct a literature that supports that belief — even if the underlying phenomenon is weaker than the consensus suggests, or does not exist at all.

The philosopher of science might recognize this as a variant of the Duhem-Quine problem: when an experiment fails to confirm a hypothesis, it is never entirely clear whether the hypothesis is wrong or whether one of the auxiliary assumptions surrounding the experiment is at fault. Meta-analysis does not resolve this underdetermination. It inherits it from every study it absorbs.

Toward a More Skeptical Synthesis

None of this is an argument against systematic review as a practice. The alternative — treating each study as an island, making decisions on the basis of individual findings without any attempt at synthesis — would be far worse. The point is more precise: that the authority granted to meta-analytic conclusions should be calibrated to the quality of the underlying literature, and that this calibration is currently, in many fields, badly miscalibrated in the direction of overconfidence.

Several researchers have begun developing what might be called adversarial synthesis — a methodological stance that treats the prior literature not as a resource to be mined but as a body of claims to be challenged. This involves pre-registering the meta-analysis protocol before examining the data, conducting sensitivity analyses that test how conclusions shift when questionable studies are excluded, and reporting heterogeneity honestly even when it complicates the narrative.

These are not radical proposals. They are, in a sense, merely the application of scientific skepticism to the process of scientific aggregation itself — a recognition that the tools we use to find truth are not exempt from the scrutiny we apply to everything else.

The most durable scientific consensus is not the one that accumulates citations most rapidly. It is the one that survives the most determined attempts to dismantle it. Until the machinery of synthesis is held to that standard, the confidence we place in aggregated evidence will remain, in too many cases, a confidence built on ground that has not been tested nearly as thoroughly as it appears.