Problems with 30 of 107 papers reviewed

Three reviews of HER2 as a prognostic factor in breast cancer have been published by The Oncologist, in 1998, 2003, and 2009.

In the 1998 paper, 10 of the 47 studies were mishandled; correcting the errors overturned the review's conclusion that HER2 is independently prognostic. 

The error rate increased in subsequent reviews. The 2003 update added 34 more papers and 10 new errors. The most recent review, published in 2009, added 26 papers and 10 more new errors. 

All told, in the 2009 review, of the 107 papers reviewed, a total of 30 (28%) are either miscoded or should not have been included:

• 10 papers coded 'Yes' for multivariate significance should be 'No'
•  7 papers coded 'Yes' for multivariate significance should be 'NA' 
• 11 papers should not have been included 
•  2 papers coded 'No' for multivariate significance should be 'Yes' 

Appendix A below defines what constitutes an error. Appendix A also enumerates and explains the 30 errors contained in the 2009 review.

Stuffing the ballot box

First author acknowledges possibility of errors, disputes none of them

Jeffrey Ross, the first author on all three reviews, acknowledged the first two might contain errors. Regarding the 1998 review, Ross wrote in email: "It is certainly possible that the studies you have cited were not perfectly listed in my manuscript from so many years ago.” 

With respect to the 2003 review, Ross wrote: "I have no reason to believe that your conclusions are not correct and that there were scattered errors in the meta-analysis of the published literature in our 2003 manuscript."

However, contacted regarding the most recent, 2009 paper, Ross wrote: "Due to time constraints, I am unable at this time to either agree or disagree with your analysis..." In PubMed, the 2009 review is cited 133 times.

No response from The Oncologist

According to the Committee on Publication Ethics (COPE) guidelines, journal editors should consider issuing a correction if "a small portion of an otherwise reliable publication proves to be misleading (especially because of honest error)." 

Three emails documenting possible issues in the Ross et al. reviews, sent to Martin Murphy, executive editor at The Oncologist, have not been answered. The Oncologist is a member of COPE.

Appendix A

Papers counted as representing an error were either miscoded or inappropriately included. Note the 2009 review includes all the papers and errors contained in the 1998 and 2003 reviews.

Miscoding

Inappropriate Inclusion

Enumeration of Errors

84. Lal et al.: Yes to Exclude

Correlates HER2 with other biomarkers not clinical outcomes, as the title suggests: "Correlation of HER-2 Status With Estrogen and Progesterone Receptors and Histologic Features in 3,655 Invasive Breast Carcinomas"

85. Huang et al.: Yes to Exclude

Correlates HER2 with other biomarkers not clinical outcomes, as the title suggests: "Association between tumour characteristics and HER-2/neu by immunohistochemistry in 1362 women with primary operable breast cancer"

87. Ariga et al.: Yes to Exclude 

Correlates HER2 with other biomarkers not clinical outcomes, as the title suggests: "Correlation of Her-2/neu Gene Amplification with Other Prognostic and Predictive Factors in Female Breast Carcinoma"

89. Prati et al.: Yes to Exclude

Correlates HER2 with other biomarkers not clinical outcomes, as the title suggests: "Histopathologic Characteristics Predicting HER-2/neu Amplification in Breast Cancer"

90. Tanner et al.:Yes to NA

The study does not include a multivariate analysis of HER2 as an independent prognostic factor. In the paper's only multivariate analysis, all the patients were HER2+: 

91. Diallo et al.: Yes to Exclude

Correlates HER2 with other biomarkers not clinical outcomes.

99. Sandri et al. Yes to Exclude

Examines HER2 in serum, as the title suggests: "Serum EGFR and serum HER-2/neu are useful predictive and prognostic markers in metastatic breast cancer patients treated with metronomic chemotherapy"

101. Sunami et al.: Yes to Exclude

Correlates HER2 with other biomarkers not clinical outcomes, as the title suggests: "Estrogen receptor and HER2/neu status affect epigenetic differences of tumor-related genes in primary breast tumors"

106. Ludovini et al.: Yes to No

Found HER2 by IHC and FISH significant in univariate analysis. But only serum HER2 was found prognostic in the multivariate analysis. (See table 5.) 

Biology’s holy grail, a full mechanistic understanding of the workings of life, is beyond reach according to two recent papers. Computer models that closely replicate the phenomena of a single cell are not possible, and the goal has been dropped.

Over the last decade, researchers have tried to grapple with biological complexity by modeling less complicated organisms. Yeast proved too complex and was replaced by organisms with smaller and smaller genomes, all the way down to tiny Mycoplasma pneumoniae. Unable to reduce genomes any further, scientists have radically reduced expectations for models instead.

In Science last month, researchers described the “popular view,” in which “we progress linearly, from conceptual to ever more detailed models.” The popular, linear view is no more. From now on, models “should be judged by how useful they are and what we can learn from them,” according to the paper’s authors, “not by how close we are to the elusive ‘whole cell model’.”

Alex Mogilner, one of authors and a professor at UC Davis, believes some future discovery might make the whole cell model again possible. “Never say never,” he advised. However, a paper from the Institute for Systems Biology forecloses the possibility for all time:

[N]o practically conceivable model will ever represent all possible physical parameters in a system, nor will enough data ever exist to fully constrain them all. It is also experimentally infeasible to measure, and technically prohibitive to model all possible phenomena in a cell, all possible environmental contexts, and all possible genetic perturbations.

There will be no in silico model of a cell, one that fully recapitulates cell behavior and substitutes for wet lab experiments. “Anyone who thinks we can ever obtain a completely deterministic view of an organism will have a hard job to convince me,” said Marc Kirschner, chair of the systems biology department at Harvard. “It is probably true that the number of equations to describe the events in a single cell is so large that this approach will never work,” according to Kirschner. He does hope to be able to predict “to some accuracy” particular responses of a system.

The implications for the future have yet to be worked out, although Mogilner and colleagues observed that such models were envisioned as enabling personalized medicine. For historical purposes, however, these papers bring an end to a monumentally successful, physics-based program for biology that began roughly a century ago.

Biologist Thomas Hunt Morgan successfully pioneered the methods of physics in biology, elucidating the role of chromosomes in heredity. This “turned out to be extraordinarily simple,” as he wrote in 1919, and nature was entirely approachable. “[I]f the world in which we live were as complicated as some of our friends would have us believe,” Morgan wrote, “we might well despair that biology could ever become an exact science.”

Shortly thereafter, physics underwent a crisis of faith as the discipline moved from an intuitive, mechanistic basis into a new and unsettling quantum era which renounced the Newtonian ideal of casually linking everything in space and time. When DNA was discovered decades later, the Newtonian paradise was regained. As a theoretical physicist turned biologist Max Delbrück said in his Nobel Prize lecture:

It might be said that Watson and Crick’s discovery of the DNA double helix in 1953 did for biology what many physicists hoped in vain could be done for atomic physics: it solved all the mysteries in terms of classical models and theories, without forcing us to abandon our intuitive notions about truth and reality.

Not long after, Lee Hood decided to become a biologist after reading an article by Francis Crick in Scientific American. Crick wrote how “the sequence of the bases acts as a kind of genetic code…” which was unknown. Many years later, Hood expressed the belief that “the core of biology is ultimately knowable, and hence, we start with a certainty that is not possible in the other disciplines,” like physics. He forecast being able to predict the behavior of a biological systems “given any perturbation.” His lab at Caltech invented the DNA sequencer.

A draft sequence of the human genome was published in 2000 and Hood founded his Institute for Systems Biology (ISB). The same year, Matt Ridley published his best-selling Genome which predicted a leap from knowing “almost nothing about our genes to knowing everything,” which he described as “the greatest intellectual moment in history. Bar none."

For the next dozen years, researchers from ISB heaved with might and main to realize Hood’s vision. Instead, they now say it is unattainable.

Undoubtedly, there will be disbelief. But Robert Millikan, a founding father of Caltech, didn’t want to believe in Einstein’s photoelectric effect. He won a Nobel Prize for being wrong and proving Einstein right.

This may still be one of the greatest intellectual moments in history, just not what we expected.

Image of yeast adapted from Nelson et al. DOI: 10.1073/pnas.0910874107

The 20th century broke open both the atom and the human genome. Physics deftly imposed mathematical order on the upwelling of particles. Now, in the 21^st century, systems biology aims to fit equations to living matter, creating mathematical models that promise new insight into disease and cures. But, after a decade of effort and growth in computing power, models of cells and organs remain crude. Researchers are retreating from complexity towards simpler systems. And, perversely, ever-expanding data are making models more complicated instead of accurate. To an extent, systems biology, rather than climbing upwards to sparkling mathematical vistas, is stuck in a mire of its own deepening details.

Read the rest....

Celebrex photo (Allison Turrell)

Although medicine has never been more advanced, progress against the two biggest killers, cancer and heart disease, now comes more slowly than it did decades ago. A skeptical reader suggested that recent breakthroughs in depression, diabetes and arthritis present a truer picture of medical advance. But surveying these other diseases in fact reveals similar contours: the steepest gains occur in the past with progress flattening toward the present.

Depression

No antidepressant, however new, surpasses the very first, imipramine, which dates to 1952. Although we are on the third or fourth generation of antidepressants, the newest agents don’t outperform the oldest. Savella, the most recently approved antidepressant, worked the same as imipramine in a meta-study of seven clinical trials. Same for Effexor. Same for Zoloft. Same for Prozac. Unsurprisingly, the newer agents are about the same in terms of efficacy compared with each other. For severe, psychotic depression, imipramine remains the treatment of choice.

Having more therapeutic options greatly aids the art of treating an individual patient, and welcome progress has come on side effects. But we see refinement not revolution. Conceivably research emphasis actually shifts to side effects when progress on defeating a given disease slows. “Polypharmacy,” prescribing more than one drug for a condition, is another such symptom visible in antidepressants.

The reduced (or sometimes merely different) side effects of newer antidepressants come courtesy of very substantial advances in chemistry and receptor biology. Modern compounds strike their targets far more precisely than the older, less discriminating, “dirtier” agents like imipramine. However, consummate skills at the microscopic level are not matched by an understanding of the overall machinery of depression despite a vast expansion of neurological knowledge. In some respects, depression appears more complicated than ever, a tangled skein catching up many different biological pathways, looping through environmental and genetic factors. Breaking free of a complex disease, whether cancer or depression, requires more than snipping the single threads we have found so far.

Diabetes

In diabetes, insulin clearly wins the prize for biggest breakthrough, one achieved in 1923 before there were clinical trials. But none were needed. Insulin very straightforwardly saves the lives of type 1 diabetics who would die without it. Living with diabetes has since become progressively less and less difficult. Insulin has been improved many times, but these are refinements, like the recent fast-acting or long-release variants.

After insulin, the largest absolute gain in glycemic control came in the late 1950s and early 1960s with the introduction of sulfonylureas. For type 2 diabetes, these drugs were frontline until 1995 when metformin jumped to the front because of its greatly reduced side effects. The sulfonylureas actually provide better glycemic control than metformin but ultimately they exhaust the pancreas. After metformin came a series of new drug classes: glitazones in 1999, GLP-1 analogs like Byetta in 2005 and DPP-4 inhibitors like Januvia in 2006. These drugs exploit recently acquired, fine-grain knowledge of biochemical pathways. By contrast, we don't really know how metformin works except that it acts on the pancreas. But the new drugs aren’t “better” than older ones. Instead they provide useful (albeit more expensive) treatment alternatives.

As with depression, increased mastery of microbiological detail in diabetes hasn’t translated into mastery of the disease, a cure. Remarkably, gastric bypass surgery somehow controls type 2 diabetes quickly and not simply because of weight loss. A drug mimicking such an effect would likely be the biggest diabetes breakthrough ever. But the mysterious target has been lurking since 1987. Although today’s suite of diabetes therapies enables more precise glycemic control than ever before, for now, the largest advances are more than half a century old.

Rheumatoid Arthritis

A wave of new biological therapies has washed over rheumatoid arthritis, establishing a new high water mark in relief and remission. For all their scientific brilliance, however, biologicals back up an older drug, methotrexate. In use as early as 1967, methotrexate became the drug of choice in the mid-1980s and it remains first line therapy today.

The most important new biologicals, the TNF inhibitors, arrived in 1998 when the FDA approved Enbrel. However, in head-to-head trials, TNF inhibitors work about the same as methotrexate. Enbrel, for example, performed marginally better than methotrexate in one study while Humira did marginally worse in another. Not surprisingly, the differences between these biologicals are negligible. However, in patients where methotrexate has not worked, adding a TNF inhibitor can make an enormous difference. Remicade plus methotrexate boosted response rates to 52% compared to 17% with methotrexate alone.

The new agents also work much faster and appear to provide much greater protection against joint damage. Still, the biologicals haven’t beaten methotrexate and usually join them in combination therapy. In the case of Remicade, methotrexate is a requirement.

The newest biological for rheumatoid arthritis, Actemra, approved in January, 2010, supplants no other drugs but occupies the last line of treatment, after the TNF inhibitors. Actemra joins the other recently-approved biologicals Rituxan and Orencia sitting at the end of the bench.

 The benefits of new drugs for heart disease, breast cancer, depression, diabetes as well as rheumatoid arthritis have declined over time, with the largest gains realized decades ago. Methotrexate, dating to the early 1980s, provided the biggest gain, followed by TNF inhibitors in the 1990s, with the newest drug Actemra last and least.

However, we are heir to decades of medical advances. The best time in all of human history to have any chronic disease is now (or, better yet, tomorrow). Surprisingly, though, we are adding less to this legacy than previous generations—and we’re a bit oblivious about what’s happening.

----------------------------------------------------------

Related:

If heart drugs keep improving, will we be able to tell?

Life expectancy, medical progress, birthday wishes (video)