Problems with 30 of 107 papers reviewed
Three reviews of HER2 as a prognostic factor in breast cancer have been published by The Oncologist, in 1998, 2003, and 2009.
In the 1998 paper, 10 of the 47 studies were mishandled; correcting the errors overturned the review's conclusion that HER2 is independently prognostic.
The error rate increased in subsequent reviews. The 2003 update added 34 more papers and 10 new errors. The most recent review, published in 2009, added 26 papers and 10 more new errors.
All told, in the 2009 review, of the 107 papers reviewed, a total of 30 (28%) are either miscoded or should not have been included:
- 10 papers coded 'Yes' for multivariate significance should be 'No'
- 7 papers coded 'Yes' for multivariate significance should be 'NA'
- 11 papers should not have been included
- 2 papers coded 'No' for multivariate significance should be 'Yes'
Appendix A below defines what constitutes an error. Appendix A also enumerates and explains the 30 errors contained in the 2009 review.
Stuffing the ballot box
The 11 papers which should not have been included accounted for 7,511 (19%) of the 39,730 patients in the 2009 review. Of the 7,511, the review reported 7,213 supported HER2 as independently prognostic. A single paper, Lal et al. (2005) contributed 3,655 patients to the review, more than twice as many as the
next largest study. Like nine of the 11 erroneously included papers,
Lal et al. examined the correlation of HER2 with other biomarkers, not HER2 and clinical outcomes.
First author acknowledges possibility of errors, disputes none of them
Jeffrey Ross, the first author on all three reviews, acknowledged the first two might contain errors. Regarding the 1998 review, Ross wrote in email: "It is certainly possible that the studies you have cited were not perfectly listed in my manuscript from so many years ago.”
With respect to the 2003 review, Ross wrote: "I have no reason to believe that your conclusions are not correct and that there were scattered errors in the meta-analysis of the published literature in our 2003 manuscript."
However, contacted regarding the most recent, 2009 paper, Ross wrote: "Due to time constraints, I am unable at this time to either agree or disagree with your analysis..." In PubMed, the 2009 review is cited 133 times.
No response from The Oncologist
According to the Committee on Publication Ethics (COPE) guidelines, journal editors should consider issuing a correction if "a small portion of an otherwise reliable publication proves to be misleading (especially because of honest error)."
Three emails documenting possible issues in the Ross et al. reviews, sent to Martin Murphy, executive editor at The Oncologist, have not been answered. The Oncologist is a member of COPE.
Appendix A
Papers counted as representing an error were either miscoded or inappropriately included. Note the 2009 review includes all the papers and errors contained in the 1998 and 2003 reviews.
Miscoding
This examination focuses solely on the reporting of HER2 having independent prognostic value in a multivariate analysis. The reviews misclassified the findings of 19 papers.
Perhaps most remarkably, seven of the 19 did not report performing a multivariate analysis of HER2 as a prognostic factor.
Ten papers did perform such an analysis but found HER2 did not predict clinical outcomes although the reviews categorized the 10 as finding HER2 to be independently prognostic.
Two studies were reported as finding HER2 not prognostic when the papers did find it prognostic. Strangely, one of these false negatives was a paper co-authored by Jeffrey Ross, i.e. he seems to have miscoded one of his own papers.
Inappropriate Inclusion
The three reviews mostly examined papers that included some clinical outcome, such as disease free survival, in HER2 positive and HER2 negative patients. However, particularly in the 2009 review, studies of HER2 were reviewed that did not include any clinical outcome. Of 11 papers that should not have been included, nine correlated HER2 with other biomarkers, not clinical outcomes.
Inclusion of one the 11 papers, Wright et al., resulted in a double-counting (in Gullick et al.) of a single cohort.
In the last of the 11, Sandri et al., the paper examined HER2 in serum whereas the other studies in the review were of HER2 overexpression or amplification in tumor cells. The review's conclusions are only for overexpression and/or amplification.
Enumeration of Errors
Numbers correspond to the study number from Table 1 of the 2009 review.
2. Berger et al.: Yes to Exclude
Correlates HER2 with other biomarkers not clinical outcomes, as the title suggests: "Correlation of c-erbB-2 Gene Amplification and Protein Expression in Human Breast Carcinoma with Nodal Status and Nuclear Grading."
4. Wright et al.: Yes to Exclude
One of three studies incorporated in Gullick et al. (1991), also in the review. As a result, the same 185 patients are counted twice.
9. Battifora et al.: Yes to No
The paper reports: "Stepwise Cox Regression: This analysis identified independent prognostic factors of DFS and OS when all variables were considered together. Independent predictors of DFS included stage of disease, histology, and nuclear grade. Nuclear grade and stage were the only significant predictors of OS."
13. Lovekin et al.: Yes to No
The paper reports: “Multivariate analysis (Cox, 1972) was used to identify whether c-erbB-2 was of independent prognostic significance. In the context of the temporal variables, tumour size and lymph node stage, cell membrane staining was found to have independent significance as a prognostic factor but significance was lost when histological grade was included in the analysis."
15. Dykins et al.: Yes to NA
No multivariate analysis
19. Paterson et al.: Yes to No
The paper does not state HER2 is independently prognostic in a multivariate analysis or provide the statistics relevant to such a statement. The authors do suggest possible confounding of prognostic factors: “our study design precluded direct determination of the interrelationships of c-erbB-2 [HER2] amplification with conventional disease parameters.”
21. Molina et al.: Yes to NA
No multivariate analysis
28. Press et al.: Yes to NA
No multivariate analysis
30. Descotes et al.: Yes to Exclude
Correlates HER2 with other biomarkers not clinical outcomes, as the title suggests: “Correlation study between Her-2/neu amplification and prognostic factors.”
33. Têtu et al.: Yes to No
The paper reports that HER2 was predictive of treatment resistance: “The difference in survival rates between cases was only significant among patients submitted to adjuvant chemotherapy or hormone therapy."
46. Charpin et al.: Yes to NA
No multivariate analysis
54. Scorilas et al.: No to Yes
Tables 2 and 3 show HER2 overexpression prognostic in multivariate analyses of early relapse and overall survival.
59. Agrup et al.: Yes to NA
No multivariate analysis
67. Jukkola et al.: Yes to No
The abstract reports: "In multivariate regression analysis, only tumour size and nodal involvement were risk factors for poor survival when analysed separately together with c-erbB-2 and receptor status..."
Section 3.2 states: "In multivariate Cox stepwise regression analysis, tumour size and nodal involvement emerged as independent prognostic factors when analysed separately in combination c-erbB-2, indicating a 2.9 (90% CI 1.9-4.4) risk of death in node-positive patients. For patients with tumour sizes T3 or T4 the risk of death was 2.7 (90% CI 1.4-5.1) and 4.8 (90% CI 2.5-9.5), respectively, c-erbB-2 status did not reach significance in this model, nor when analysed in combination with tumour size, nodal involvement and receptors."
69. Rudolph et al.: Yes to No
HER2 only emerges as prognostic if CR is removed: "When all variables that attained statistical significance in the univariate analysis were included in the multivariate model, the CR was the first and most significant independent indicator of both AOS and DFS (P .0001; Table 3). Next to CR, only PR status was found to be an independent prognostic factor, albeit of borderline significance."
71. Pinto et al.: Yes to No
HER2 is not independently prognostic: "C-erbB-2 is an independent prognostic indicator when evaluated in conjunction with ploidy and SPF."
73. Horita et al.: Yes to NA
No multivariate analysis
74. Suo et al.: Yes to No
HER2 is only prognostic when combined with EGFR or HER4. See Table 5.
76. Rosenthal et al.: No to Yes
A paper on which Ross is senior author found "Multivariate analysis of the combined LN+ and LN− lobular and ductal cases revealed that HER-2/neu amplification (P 0.002), pathologic stage (P < 0.0001), and node positivity (P < 0.0001) were all independent predictors of disease-related death."
78. Spizzo et al.: Yes to No
The paper states: "Multivariate analysis for DROS revealed that nodal status, EpCAM overexpression, tumor size and histological grade were significant prognostic factors. Hormone receptor expression and Her-2/neu overexpression were not significant predictors of DROS. For DFS, nodal status, Ep-CAM overexpression, tumor size and progesterone receptor expression were significant prognostic factors. Her-2/neu overexpression, histologic grade and estrogen receptor expression had no prognostic value for disease-free survival (Table III)."
81. Taucher et al.: Yes to Exclude
Correlates HER2 with other biomarkers not clinical outcomes.
84. Lal et al.: Yes to Exclude
Correlates HER2 with other biomarkers not clinical outcomes, as the title suggests: "Correlation of HER-2 Status With Estrogen and Progesterone Receptors and Histologic Features in 3,655 Invasive Breast Carcinomas"
85. Huang et al.: Yes to Exclude
Correlates HER2 with other biomarkers not clinical outcomes, as the title suggests: "Association between tumour characteristics and HER-2/neu by immunohistochemistry in 1362 women with primary operable breast cancer"
87. Ariga et al.: Yes to Exclude
Correlates HER2 with other biomarkers not clinical outcomes, as the title suggests: "Correlation of Her-2/neu Gene Amplification with Other Prognostic and Predictive Factors in Female Breast Carcinoma"
89. Prati et al.: Yes to Exclude
Correlates HER2 with other biomarkers not clinical outcomes, as the title suggests: "Histopathologic Characteristics Predicting HER-2/neu Amplification in Breast Cancer"
90. Tanner et al.:Yes to NA
The study does not include a multivariate analysis of HER2 as an independent prognostic factor. In the paper's only multivariate analysis, all the patients were HER2+:
91. Diallo et al.: Yes to Exclude
Correlates HER2 with other biomarkers not clinical outcomes.
99. Sandri et al. Yes to Exclude
Examines HER2 in serum, as the title suggests: "Serum EGFR and serum HER-2/neu are useful predictive and prognostic markers in metastatic breast cancer patients treated with metronomic chemotherapy"
101. Sunami et al.: Yes to Exclude
Correlates HER2 with other biomarkers not clinical outcomes, as the title suggests: "Estrogen receptor and HER2/neu status affect epigenetic differences of tumor-related genes in primary breast tumors"
106. Ludovini et al.: Yes to No
Found HER2 by IHC and FISH significant in univariate analysis. But only serum HER2 was found prognostic in the multivariate analysis. (See table 5.)