Questionable diet findings from the Nutrition Researchers Guild: how can we learn from this unscientific manipulation of statistics?

AAre you confused by the conflicting results of “research” on the effects of certain foods on our health? It would hardly be surprising. First, butter is the enemy; then it’s solid margarine. Does caffeine Good or bad for your heart? For a time, beta-carotene supplements were thought to prevent cancer – until they increased the risk of lung cancer in smokers. And finally, does a woman’s diet at conception determine the sex of her fetus?

When you dive deep into the methodology of the studies that have produced such findings, it’s no surprise that they’re inconsistent or implausible.

Consider who performs such research.

A branch of pseudo-science has become something of a throwback to a phenomenon that began centuries ago – a “guild», a formal group with a common interest, or a set of artisans or traders who have common interests and standards. (Or, as discussed below, a lack statistical standards.)

Entry into a guild was originally by apprenticeship where the pay was low while learning a trade. There were corporations of goldsmiths, and in Holland, even corporations of painters. Sometimes trade guilds spanned large regions and were often aligned or protected by governments. Medieval cities supported guild monopolies.

Today, certain scientists – or, as they are more accurately described, “pseudoscientists” – have formed what amounts to a guild of researchers whose intention, it seems, is to undermine the science of feed.

Here’s what they do: They perform what is called “cohort studieswhich include the results of questionnaires probing the food consumption habits of respondents. A cohort study identifies a group of people and follows them longitudinally over time, asking serial questions and testing associations or correlations – for example, whether certain food consumption patterns predispose to heart disease or dementia or affect longevity.

England”The life planwas one of the first cohort studies, funded by the UK government. For many years, he tracked all children born within a time window. Finally, new cohorts were formed and followed. Data from the questions asked and answered in The Life Project Cohort of 1958 resulted in some 2,500 papers and are still ongoing.

Credit: Alamy

The type and amount of food eaten arguably affects health, so it makes sense that cohort researchers would want to collect information about foods that could be linked to later health outcomes. To facilitate this, nutrition researchers at Harvard University have developed a semi-quantitative analysis”food frequency questionnaireor FFQ in the mid-1980s.

The FFQ is a self-administered food questionnaire that asks people how often they eat specific foods. The initial FFQs included 61 foods. The choice of this particular number was probably intentional, as there is a 95% chance of getting at least one positive result if you ask 61 (independent) questions. 95% is generally defined as the level necessary for a study to statistically make its point.

Over the years, many cohort studies have been initiated; and FFQs with more and more foods represented have become a standard feature of many of them. In a research project from the National Association of Scholars which reviewed a set of 105 papers on the health effects of red meat, the number of foods included ranged from 14 to 280, with a median of 51.

meat beware imgcache rev ed cdfbfd c fbc dc d bdb x
Credit: Health Initiative

Questionnaires could ask, for example, how many citrus fruits, broccoli, avocado, etc. the subject has consumed, and what health problems – hypertension, depression, angina, asthma, obesity, erectile dysfunction, glaucoma, etc. – he or she experienced . The number of entries in both categories is large and therefore the possible associations are vast.

This presents something of a statistical dilemma – really a quagmire – because nutritional epidemiologists rarely, if ever, adjust their statistical analysis to account for the number of questions examined and the likelihood that a certain fraction of the supposed correlations will occur. by chance. In other words, performing a large number of statistical tests—that is, looking for correlations in a study—produces many false positives by chance.

What are you worried about now?

Another complicating factor is that the supposed correlations can be sliced ​​and diced by sex, age, race, etc., and also at multiple times. To shed some light on this conundrum, the just-released landmark report of the National Association of Scholars (NAS) project, Shifting Sands: Faulty Science and Dangerous Regulations, examined how these kinds of chance correlations produce spurious and irreproducible conclusions that affect various areas of government policy and the regulation of federal agencies. The report applies Multiple Testing and Multiple Modeling (MTMM) to assess whether a given body of research has been affected by such defective practices, in which case the claim would be considered unreliable.

Non-statisticians might not appreciate the significance of this finding, but it poses a fundamental problem: without proper adjustment for the number of claims examined, the research results of nutrition researchers are often irreproducible. In other words, however compelling or potentially important the findings may seem, they really don’t make sense.

What does this flawed approach of researchers using food frequency questionnaires have to do with their guilding? They keep their research data private; they do not criticize each other on dubious statistical methods; they sell their monopoly “expertise” to the government in exchange for subsidies; and they monitor their trade themselves (i.e. they arbitrate each other’s papers). They also protect their own: several years ago, for example, fourteen of the “members of the guild” required that an article criticizing the research of one of the members be withdrawn.

screenshot am

Nutrition researchers have built a very comfortable, profitable, and practical pseudo-science guild whose members know, or should know, that in the interest of increasing their number of papers, they sometimes publish non-reproducible results. And in doing so, they fill the scientific literature with tens of thousands of studies based on worthless food frequency questionnaires that confuse and scare the public.

Take, for example, the 2008 article, “You are what your mother eats: evidence that maternal preconception diet influences fetal sex in humans.” He got a lot of attention by making the genetically implausible claim that women who eat more cereal for breakfast are more likely to have a boy. This result is easily explained by chance. What was the article based on? On observational (cohort) studies.

This is not an isolated example. Claims from cohort studies (e.g., FFQ) regarding human health effects of various interventions such as vitamin supplementation or low-fat diets have not been supported by large randomized clinical trials.

What can we do about this situation? It’s hard, because the status quo represents collusion between dishonest researchers themselves, predatory journals that publish anything for high fees, and university departments that overlook malpractice. They all “benefit” in one way or another from easy to generate but often specious research. Until one or more of them break the cycle, nothing will change.

Perhaps the funding agencies, who have the power – and the responsibility – to police scientific integrity, will start doing their job and crush the guild. We are not optimistic.

Henry I. Miller, physician and molecular biologist, is principal investigator at Pacific Research Institute. He was a founding director of the FDA’s Office of Biotechnology and a research associate at the NIH. Find Henry on Twitter @henryimiller

Dr. S. Stanley Young is a Fellow of the American Statistical Association and the American Association for the Advancement of Science. He is an adjunct professor of statistics at North Carolina State University, the University of Waterloo, and the University of British Columbia, where he co-directs thesis work.

Leave a Comment