Researchers shoot holes in study touted for confirming 'masks work' in curbing COVID

An acclaimed study on the effectiveness of masks in reducing symptomatic COVID-19 is facing new scrutiny after a researcher highlighted the minuscule infection differences between "treatment" and control groups randomized across 600 Bangladeshi villages.

Accused of design flaws and overstating its findings when it was released in late August, the study's newly released data show only 20 more symptomatic COVID cases in the villages that didn't receive masks and related education, reminders and "role modeling by community leaders."

In a total study population of 342,126 adults, 1,106 people in the control group tested positive, compared to 1,086 in the treatment group. The latter group represented 52% of the study population.

"I have a hard time going from these numbers to the assured conclusions that 'masks work' that was promulgated by the media or the authors after this preprint [not yet peer reviewed] appeared," University of California Berkeley professor Ben Recht, who studies machine learning, wrote in an essay last week.

He said he was frustrated that the "raw number of seropositive cases" was left out of the preprint by researchers led by Yale University economists Jason Abaluck and Ahmed Mobarak, preventing him from "computing standard statistical analyses of their results."

The researchers posted the replication code and data in early November, long after media coverage touting "the largest randomized trial to demonstrate the effectiveness of surgical masks, in particular, to curb transmission of the coronavirus."

In light of the full release, "a complex intervention including an educational program, free masks, encouraged mask wearing, and surveillance in a poor country with low population immunity and no vaccination showed at best modest reduction in infection," Recht said.

The newly provided raw numbers exacerbate other weaknesses of the study, according to Recht, who was also initially skeptical of the research because of its "statistical ambiguity."

The study was not blinded, did not exclude pre-intervention infections, and was "highly complex" because of the mixed interventions, he said. 

The three-percentage-point differential between household visit consent rates for the treatment and control groups, by itself, "could wash away the difference in observed cases," he explained, adding that relative measures of risk are "[o]ne of the dark tricks of biostatistics," which unlike hard case counts have a tendency to exaggerate effects.

'How robust can this possibly be?'

The UC Berkeley professor's analysis drew attention on Twitter, including from Harvard Medical School epidemiologist Martin Kulldorff, whose own skepticism of the protective power of masks for unvaccinated elderly people got him suspended by Twitter for a month.

"One of the problems of the study is that despite the vast size of the study, the primary endpoint depends on ~5000 blood samples collected" each from the treatment and control groups, Philadelphia cardiologist Anish Koka wrote in a related thread.

Appreciate authors of the 🇧🇩 RCT finally releasing raw data.

Dismayed at their topline conclusion on mask effectiveness that generated so much buzz

Out of ~340,000 ppl in mask and control arm.. the difference in symptomatic cases was 20 over 8 weeks.https://t.co/ZIMCNDXibZ

— Anish Koka (@anish_koka) November 24, 2021

"So we are left to extrapolate from a 20 case difference tested in ~10,000 patients to a 300,000 patient study," he continued. "But how robust can this possibly be?"

Koka noted that Yale's Abaluck, a lead author, floated the idea of fining people for not wearing government-supplied cloth masks, the least effective kind, early in the pandemic. "It seems a bit much to go from these small differences to the police tracking down and fining people who don't mask in public," the cardiologist wrote.

Abaluck also argued that if cloth masks reduced spread by 10%, that would more than justify an estimated cost per unit of $1. 

1) Federal or state governments should immediately purchase a dozen cloth masks for everyone and strongly encourage everyone to wear masks when in public (perhaps even fine them if they don't).

— Jason Abaluck (@Jabaluck) March 27, 2020

The Bangladeshi research performed a year later found the interventions reduced "symptomatic seroprevalence" by 9.3% in the treatment group, but also that cloth masks specifically had "an imprecise zero" effect and surgical masks were statistically insignificant for age groups under 50.

George Mason University economist Tyler Cowen, founder of the economics blog Marginal Revolution and dubbed "one of the most influential economists" of the 2000s by The Economist, pointed his readers to Recht's analysis.

"With more data transparency, [the Bangladeshi study] does not seem to be holding up very well," he wrote Sunday, cautioning that "at the aggregate social level we are quite far from knowing how well masks work."

Abaluck called Recht's analysis "deeply flawed" in a lengthy email to Just the News Monday. He apparently gave the same response to Cowen's post Tuesday, emphasizing the study was now undergoing review at Science.

Recht agrees that "our intervention led to a roughly 10% reduction in symptomatic seropositivity (going from 12% to 41% of the population masked)," Abaluck wrote. "Taking this estimate at face value, going from no one masked to everyone masked would imply a considerably larger effect" — a 33% reduction in COVID. 

"People underwhelmed by the absolute number of cases prevented need to ask, what did you expect if masks are as effective as the observational literature suggests?" he wrote. 

Abaluck agreed that "survey response bias is a potential concern" but emphasized that the "direction of the bias is unclear — individuals might be more attuned to symptoms in the treatment group." His research group has obtained funding to "replicate the entire study and collect blood spots from symptomatic and non-symptomatic individuals to partially mitigate this bias."

Recht didn't respond to a query for his response to Abaluck's rebuttal but gave a short response in a Twitter thread Tuesday, saying it "highlights how conventions in science can vary widely from field to field," in this case economics and medicine.

These differences show why "we should pay more attention to effect size, bias, and confounding [variables] than any convoluted statistical argument."

At @MargRev, @tylercowen has posted a rebuttal by @jabaluck to my blog on the Bangladesh RCT. Notably, Abaluck confirms there is only a 20 case difference between the control and treatment arms. https://t.co/aOVxPlJjVl 1/5

— Ben Recht (@beenwrekt) November 30, 2021

Abaluck responded in his own lengthy thread by claiming Recht misunderstood the research trial. "The bottom-line is that the most natural and powered way of interpreting the study results remains asking whether people were less likely to have Covid in the treatment group, which is exactly what we do," he wrote.

While Abaluck believes Recht "is well-intentioned," the UC Berkeley professor should have sought answers directly from Abaluck "so I could clarify that we had good reasons for many of the choices he criticizes."