Ethical concerns arise in secondary data analysis because of which of the following?

1. Kerr NL. HARKing: Hypothesizing after the results are known. Pers Soc Psychol Rev. 1998;2(3):196–217.

   CAS  PubMed  Google Scholar 

2. Border R, Johnson EC, Evans LM, et al. No support for historical candidate gene or candidate gene-by-interaction hypotheses for major depression across multiple large samples. Am J Psychiatry. 2019;176(5):376–87.

   PubMed  PubMed Central  Google Scholar 

3. Duncan LE, Keller MC. A critical review of the first 10 years of candidate gene-by-environment interaction research in psychiatry. Am J Psychiatry. 2011;168(10):1041–9.

   PubMed  PubMed Central  Google Scholar 

4. Seibold H, Czerny S, Decke S, et al. A computational reproducibility study of PLOS ONE articles featuring longitudinal data analyses. PLoS ONE. 2021;16(6):e0251194. //doi.org/10.1371/journal.pone.0251194.

   CAS  Article  PubMed  PubMed Central  Google Scholar 

5. Botvinik-Nezer R, Holzmeister F, Camerer CF, et al. Variability in the analysis of a single neuroimaging dataset by many teams. Nature. 2020;582:84–8.

   CAS  PubMed  PubMed Central  Google Scholar 

6. Orben A, Przybylski AK. The association between adolescent well-being and digital technology use. Nat Hum Behav. 2019;3(2):173.

   PubMed  Google Scholar 

7. Munafò MR, Nosek BA, Bishop DV, et al. A manifesto for reproducible science. Nat Hum Behav. 2017;1(1):0021.

   PubMed  PubMed Central  Google Scholar 

8. Nuzzo R. How scientists fool themselves–and how they can stop. Nature News. 2015;526(7572):182.

   CAS  Google Scholar 

9. Bishop DV. The psychology of experimental psychologists: Overcoming cognitive constraints to improve research: The 47th Sir Frederic Bartlett lecture. Q J Exp Psychol. 2020;73(1):1–19.

   Google Scholar 

10. Greenland S. Invited commentary: The need for cognitive science in methodology. Am J Epidemiol. 2017;186(6):639–45.

    PubMed  Google Scholar 

11. De Vries Y, Roest A, de Jonge P, Cuijpers P, Munafò M, Bastiaansen J. The cumulative effect of reporting and citation biases on the apparent efficacy of treatments: The case of depression. Psychol Med. 2018;48(15):2453–5.

    PubMed  PubMed Central  Google Scholar 

12. Nickerson RS. Confirmation bias: A ubiquitous phenomenon in many guises. Rev Gen Psychol. 1998;2(2):175–220.

    Google Scholar 

13. Franco A, Malhotra N, Simonovits G. Publication bias in the social sciences: Unlocking the file drawer. Science. 2014;345(6203):1502–5.

    CAS  PubMed  Google Scholar 

14. Silberzahn R, Uhlmann EL, Martin DP, et al. Many analysts, one data set: Making transparent how variations in analytic choices affect results. Adv Methods Pract Psychol Sci. 2018;1(3):337–56.

    Google Scholar 

15. Simmons JP, Nelson LD, Simonsohn U. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci. 2011;22(11):1359–66.

    PubMed  Google Scholar 

16. Metcalfe J, Wheat, K., Munafo, M., Parry, J. Research integrity: A landscape study: UK Research and innovation 2020.

17. Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. Proc Natl Acad Sci. 2018;115(11):2600–6.

    CAS  PubMed  PubMed Central  Google Scholar 

18. Wagenmakers E-J, Wetzels R, Borsboom D, van der Maas HL, Kievit RA. An agenda for purely confirmatory research. Perspect Psychol Sci. 2012;7(6):632–8.

    PubMed  Google Scholar 

19. Chambers CD. Registered reports: A new publishing initiative at Cortex. Cortex. 2013;49(3):609–10.

    PubMed  Google Scholar 

20. Nosek BA, Beck ED, Campbell L, et al. Preregistration is hard, and worthwhile. Trends Cogn Sci. 2019;23(10):815–8.

    PubMed  Google Scholar 

21. Kaplan RM, Irvin VL. Likelihood of null effects of large NHLBI clinical trials has increased over time. PLoS One. 2015;10(8):e0132382.

    PubMed  PubMed Central  Google Scholar 

22. Allen C, Mehler DM. Open science challenges, benefits and tips in early career and beyond. PLoS Biol. 2019;17(5):e3000246.

    CAS  PubMed  PubMed Central  Google Scholar 

23. Scheel AM, Schijen MR, Lakens D. An excess of positive results: Comparing the standard psychology literature with registered reports. Adv Methods Pract Psychol Sci. 2021;4(2):25152459211007468.

    Google Scholar 

24. Schäfer T, Schwarz MA. The meaningfulness of effect sizes in psychological research: differences between sub-disciplines and the impact of potential biases. Front Psychol. 2019;10:813.

    PubMed  PubMed Central  Google Scholar 

25. Protzko J, Krosnick J, Nelson LD, et al. High replicability of newly-discovered social-behavioral findings is achievable. PsyArXiv. 2020. doi://doi.org/10.31234/osf.io/n2a9x

26. Small DS, Firth D, Keele L, et al. Protocol for a study of the effect of surface mining in central appalachia on adverse birth outcomes. arXiv.org. 2020

27. Deshpande SK, Hasegawa RB, Weiss J, Small DS. Protocol for an observational study on the effects of playing football in adolescence on mental health in early adulthood. arXiv preprint 2018

28. Twins Early Development Study. TEDS Data Access Policy: 6. Pre-registration of analysis. //www.teds.ac.uk/researchers/teds-data-access-policy#preregistration. Accessed 18 March 2021

29. Van den Akker O, Weston SJ, Campbell L, et al. Preregistration of secondary data analysis: a template and tutorial. PsyArXiv. 2019. doi://doi.org/10.31234/osf.io/hvfmr

30. Chambers C, Tzavella L. Registered reports: past, present and future. MetaArXiv. 2020. doi://doi.org/10.31222/osf.io/43298

31. McIntosh RD. Exploratory reports: A new article type for cortex. Cortex. 2017;96:A1–4.

    PubMed  Google Scholar 

32. Scheel AM, Tiokhin L, Isager PM, Lakens D. Why hypothesis testers should spend less time testing hypotheses. Perspect Psychol Sci. 2020;16(4):744–55.

    PubMed  PubMed Central  Google Scholar 

33. Colhoun HM, McKeigue PM, Smith GD. Problems of reporting genetic associations with complex outcomes. Lancet. 2003;361(9360):865–72.

    PubMed  Google Scholar 

34. Hughes RA, Heron J, Sterne JAC, Tilling K. Accounting for missing data in statistical analyses: Multiple imputation is not always the answer. Int J Epidemiol. 2019;48(4):1294–304. //doi.org/10.1093/ije/dyz032.

    Article  PubMed  PubMed Central  Google Scholar 

35. Goldstein BA. Five analytic challenges in working with electronic health records data to support clinical trials with some solutions. Clin Trials. 2020;17(4):370–6.

    PubMed  Google Scholar 

36. Goldin-Meadow S. Why preregistration makes me nervous. APS Observer. 2016;29(7).

37. Lash TL. Preregistration of study protocols is unlikely to improve the yield from our science, but other strategies might. Epidemiology. 2010;21(5):612–3. //doi.org/10.1097/EDE.0b013e3181e9bba6.

    Article  PubMed  Google Scholar 

38. Lawlor DA. Quality in epidemiological research: should we be submitting papers before we have the results and submitting more hypothesis-generating research? Int J Epidemiol. 2007;36(5):940–3.

    PubMed  Google Scholar 

39. Vandenbroucke JP. Preregistration of epidemiologic studies: An ill-founded mix of ideas. Epidemiology. 2010;21(5):619–20.

    PubMed  Google Scholar 

40. Pingault J-B, O’reilly PF, Schoeler T, Ploubidis GB, Rijsdijk F, Dudbridge F. Using genetic data to strengthen causal inference in observational research. Nat Rev Genet. 2018;19(9):566.

    CAS  PubMed  Google Scholar 

41. Fanelli D. Negative results are disappearing from most disciplines and countries. Scientometrics. 2012;90(3):891–904.

    Google Scholar 

42. Greenwald AG. Consequences of prejudice against the null hypothesis. Psychol Bull. 1975;82(1):1.

    Google Scholar 

43. Scott KM, Kline M. Enabling confirmatory secondary data analysis by logging data checkout. Adv Methods Pract Psychol Sci. 2019;2(1):45–54. //doi.org/10.1177/2515245918815849.

    Article  Google Scholar 

44. Steegen S, Tuerlinckx F, Gelman A, Vanpaemel W. Increasing transparency through a multiverse analysis. Perspect Psychol Sci. 2016;11(5):702–12.

    PubMed  Google Scholar 

45. Del Giudice M, Gangestad SW. A traveler’s guide to the multiverse: Promises, pitfalls, and a framework for the evaluation of analytic decisions. Adv Methods Pract Psychol Sci. 2021;4(1):2515245920954925.

    Google Scholar 

46. Simonsohn U, Simmons JP, Nelson LD. Specification curve: descriptive and inferential statistics on all reasonable specifications. SSRN. 2015. //doi.org/10.2139/ssrn.2694998.

    Article  Google Scholar 

47. Rohrer JM, Egloff B, Schmukle SC. Probing birth-order effects on narrow traits using specification-curve analysis. Psychol Sci. 2017;28(12):1821–32.

    PubMed  Google Scholar 

48. Masur P. How to do specification curve analyses in R: Introducing ‘specr’. 2020. //philippmasur.de/2020/01/02/how-to-do-specification-curve-analyses-in-r-introducing-specr/. Accessed 23rd July 2020.

49. Masur PK, Scharkow M. specr: Conducting and visualizing specification curve analyses: R package. (2020).

50. Kiyonaga A, Scimeca JM. Practical considerations for navigating registered reports. Trends Neurosci. 2019;42(9):568–72.

    CAS  PubMed  Google Scholar 

51. McPhetres J. What should a preregistration contain? PsyArXiv. (2020).

52. Lakens D. Equivalence tests: A practical primer for t tests, correlations, and meta-analyses. Soc Psychol Personal Sci. 2017;8(4):355–62.

    PubMed  PubMed Central  Google Scholar 

53. Anderson ML, Magruder J. Split-sample strategies for avoiding false discoveries: National Bureau of Economic Research2017. Report No.: 0898-2937.

54. MacCoun R, Perlmutter S. Blind analysis: Hide results to seek the truth. Nature. 2015;526(7572):187–9.

    CAS  PubMed  Google Scholar 

55. MacCoun R, Perlmutter S. Blind analysis as a correction for confirmatory bias in physics and in psychology. Psychological science under scrutiny 2017. p. 295-322.

56. Rubin DB. The design versus the analysis of observational studies for causal effects: Parallels with the design of randomized trials. Stat Med. 2007;26(1):20–36.

    PubMed  Google Scholar 

57. Claesen A, Gomes SLBT, Tuerlinckx F, Vanpaemel W. Preregistration: Comparing dream to reality. 2019.

58. Schönbrodt FD, Wagenmakers E-J. Bayes factor design analysis: Planning for compelling evidence. Psychon Bull Rev. 2018;25(1):128–42.

    PubMed  Google Scholar 

59. Lakens D, Scheel AM, Isager PM. Equivalence testing for psychological research: A tutorial. Adv Methods Pract Psychol Sci. 2018;1(2):259–69.

    Google Scholar 

60. Lakens D, McLatchie N, Isager PM, Scheel AM, Dienes Z. Improving inferences about null effects with Bayes factors and equivalence tests. J Gerontol Ser B. 2020;75(1):45–57.

    Google Scholar 

61. Gopalakrishna G, ter Riet G, Vink G, Stoop I, Wicherts J, Bouter L. Prevalence of questionable research practices, research misconduct and their potential explanatory factors: a survey among academic researchers in The Netherlands. 2021.

62. Goldacre B, Drysdale, H., Powell-Smith, A., Dale, A., Milosevic, I., Slade, E., Hartley, H., Marston, C., Mahtani, K., Heneghan, C. The compare trials project. 2021. //compare-trials.org. Accessed 23rd July 2020.

63. Mathieu S, Boutron I, Moher D, Altman DG, Ravaud P. Comparison of registered and published primary outcomes in randomized controlled trials. JAMA. 2009;302(9):977–84.

    CAS  PubMed  Google Scholar 

64. Rubin M. Does preregistration improve the credibility of research findings? arXiv preprint 2020.

65. Szollosi A, Kellen D, Navarro D, et al. Is preregistration worthwhile? Cell. 2019.

66. Quintana DS. A synthetic dataset primer for the biobehavioural sciences to promote reproducibility and hypothesis generation. Elife. 2020;9:e53275.

    PubMed  PubMed Central  Google Scholar 

67. Weston SJ, Ritchie SJ, Rohrer JM, Przybylski AK. Recommendations for increasing the transparency of analysis of preexisting data sets. Adv Methods Pract Psychol Sci. 2019;2(3):214–27.

    PubMed  PubMed Central  Google Scholar 

68. Thompson WH, Wright J, Bissett PG, Poldrack RA. Meta-research: dataset decay and the problem of sequential analyses on open datasets. Elife. 2020;9:e53498.

    PubMed  PubMed Central  Google Scholar 

Page 2

From: Protecting against researcher bias in secondary data analysis: challenges and potential solutions