UvA-DARE (Digital Academic Repository) Questionable and Open Research Practices: Attitudes and Perceptions among Quantitative Communication Researchers

Recent contributions have questioned the credibility of quantitative communication research. While questionable research practices (QRPs) are believed to be widespread, evidence for this belief is, primarily, derived from other disciplines. Therefore, it is largely unknown to what extent QRPs are used in quantitative communication research and whether researchers embrace open research practices (ORPs). We surveyed first and corresponding authors of publications in the top-20 journals in communication science. Many researchers report using one or more QRPs. We find widespread pluralistic ignorance: QRPs are generally rejected, but researchers believe they are prevalent. At the same time, we find optimism about the use of open science practices. In all, our study has implications for theories in communication that rely upon a cumulative body of empirical work: these theories are negatively affected by QRPs but can gain credibility if based upon ORPs. We outline an agenda to move forward as a discipline.

have, however, brought many of the previously acceptable research practices (e.g., Bem, 1987) under scrutiny-to the point where they are now considered to conflict with the contemporary practices (Vermeulen & Hartmann, 2015).In response, communication researchers have called on the scientific community to become more transparent and adopt new open research practices (ORPs) (Bowman & Keene, 2018;Dienlin et al., 2020;Lewis Jr, 2020).
Research practices that are not transparent have been termed "Questionable Research Practices" (QRPs).They include undisclosed selective reporting of outcome variables/studies, outlier removal, data imputation, hypothesizing after the results are known (HARKing), analytical decisions to arrive at a statistically significant effect (p-hacking), and artificially inflating effect sizes (Vermeulen & Hartmann, 2015).Ultimately, QRPs "increase the likelihood of finding support for a false hypothesis" (John et al., 2012, p. 524) and make it less likely a finding will replicate (Asendorpf et al., 2013).QRPs also affect theories in communication which "search for general 'laws' about human behavior" (Chaffee & Hochheimer, 1985, p. 86) and are formed by research programs (Benoit & Holbert, 2008).On the other hand, ORPs such as preregistration, replication, and sharing of data and materials will, according to some, increase the transparency and, ultimately, the credibility of communication research (Dienlin et al., 2020;Lewis Jr, 2020).
If QRPs are indeed prevalent in quantitative communication research, this would have implications for how the field of communication research is regarded, both by outsiders as well as the scholars who belong to it.An outsider belonging to another scientific community may consider it grounds to dispute the validity of communication theories, citing the lack of evidentiary value.Likewise, a scholar of communication research may find that their understanding of the fundamental communication process formulated in the theory is most likely flawed.
However, at this point, it is largely unknown whether there is a problem in the prevalence of QRPs in quantitative communication research.We also don't know whether researchers accept the premises of the proposed solutions, i.e., ORPs.With a few notable exceptions (Elson & Przybylski, 2017;Matthes et al., 2015;Vermeulen et al., 2015), most of the literature on the prevalence of QRPs and acceptability of ORPs are based upon evidence from other disciplines, particularly psychology.
The studies that did examine the prevalence of QRPs in communication have relied on content analyses of research articles, which, for instance, examine whether the distribution of p-values clusters around .05.These analyses demonstrate that QRPs are fairly widespread (Matthes et al., 2015;Vermeulen et al., 2015).Content analyses have some limitations: (a) they are based upon the reported analyses.Many QRPs happen behind the scenes, and are, therefore, not recorded; (b) they cannot distinguish accidental omissions of details from a scholar's intent to do wrong; and (c) they cannot give information about the perceptions researchers have about the field's research practices.A survey-based approach can answer whether researchers report questionable and ORPs, how acceptable they find these practices, and how common they think these practices are.
Our project examines the self-reported prevalence and acceptance of QRPs and ORPs.Here, we report the results of a large survey (N ¼ 1039) among quantitative communication researchers-in particular first and/or corresponding authors from articles published in top-20 communication journals (Song et al., 2020) between 2010 and 2020.

Questionable Research Practices
Questionable research practices are often, consciously or unconsciously, used to "successfully" uncover a p-value of .05 or another arbitrary statistical threshold that is considered proof of a researcher's hypotheses (Gelman & Loken, 2014;Levine, 2013).Here, we discuss a selection of QRPs that are particularly relevant for quantitative communication research.It is important to note that most research practices we discuss are defensible strategies once openly reported but become "questionable" if they are not reported or done in a posthoc way.
While collecting their data, researchers may peek at the data at different points during data collection.If the results are not yet significant, they may collect more data.If they find significant results, they may stop data collection at that point.Matthes et al. (2015) find that only four out of 239 experimental papers, in a selection of communication journals, reported a priori power analyses.While this does not imply that peeking is common, it does suggest that only a few studies have a pre-specified stopping rule when to halt data collection.
Various questionable research practices also apply to the analysis of data.Researchers may impute missing data points without reporting that those data were imputed, e.g., through multiple imputations, mean substitution, and so on.People may also drop outliers or influential cases from their analyses while not disclosing this.Matthes et al. (2015) found that 22% (N ¼ 53) of the coded experimental papers reported case removal.In 3 of these 53 cases, the authors said cases were removed because they were outliers.But obviously, it is not clear in how many cases outliers were removed without disclosure.Other QRPs during data analyses are: trying different statistical tests, such as ordinary least squares (OLS) versus ordinal logit to reach statistical significance or dropping variables to reach statistically significant results (John et al., 2012).These QRPs are difficult to track in the literature as a published paper generally does not disclose which alternative analyses were tried but not reported.
The process of presenting posthoc findings as evidence of an a priori hypothesis is known as Hypothesizing After the Results are Known (HARKing, Kerr, 1998).Generations were trained that HARKing is good research practice, as exemplified in the primer on writing a scientific journal article by Bem, (1987, p. 171-172): "There are two possible articles you can write: (a) the article you planned to write when you designed your study or (b) the article that makes the most sense now that you have seen the results.They are rarely the same, and the correct answer is (b)."Bem's (1987) option (b) is acceptable when researchers are transparent about the fact that findings are exploratory and the interpretation is posthoc.But option (b), which amounts to HARKing, is a QRP if it is not made explicit that exploratory findings are presented as expected from the start (Vermeulen & Hartmann, 2015).
Finally, QRPs happen in reporting the results.Researchers, deliberately or mistakenly, round down p-values to meet a pre-specified threshold (e.g., reporting p ¼.054 as p .05).Incorrect downward rounding of p-values is also documented in quantitative communication research (Vermeulen et al., 2015).Non-publication is the QRP where whole studies or key variables that failed to reach statistical significance (e.g., p .05) are not published (Levine et al., 2008).Franco et al. (2014) report direct evidence for non-publication among experiments that were funded by Time-sharing Experiments for the Social Sciences-a project which requires funded projects to upload study data and documentation.Survey-experiments with a statistically significant treatment effect were 40 percentage points more likely to get published than experiments with null results.

Self-reported Questionable Research Practices
Content analysis is a valuable method for identifying certain QRPs such as publication bias, non-publication of studies or outcome variables, and rounding (Matthes et al., 2015;Vermeulen et al., 2015).But other QRPs, such as peeking at data, imputing missing data, or running additional analyses, are harder to isolate with a content analysis of the published literature since these QRPs are generally not disclosed in the paper.A content analysis of the published literature can also not distinguish accidental omissions of details or mistakes from a scholar's intent to do wrong.Finally, content analyses do not provide information about the perceptions researchers have about QRPs.
An alternative approach has been to survey researchers and ask them about the extent to which they have engaged in QRPs.This approach also offers insight into the perceptions researchers have about the field's research practices.Surveys in psychology (Agnoli et al., 2017;John et al., 2012;Rabelo et al., 2020), criminology (Chin et al., 2021), ecology and evolutionary biology (Fraser et al., 2018), and education (Makel et al., 2021) have found that QRPs are widespread-see Supplementary Appendix B.3.These surveys also measure whether researchers find these QRPs acceptable and whether they think these practices are prevalent in their discipline.However, there has not been a similar survey conducted among communication researchers. 2Here, we test three preregistered research questions: What is the prevalence of self-reported QRPs among communication researchers?(RQ1); How appropriate do communication researchers find QRPs?(RQ2); What is the perceived prevalence of QRPs? (RQ3)

Pluralistic Ignorance and QRPs
We think it is important to study the perceived prevalence of QRPs (see RQ3) to provide insight into the prospects for change.Research in social psychology indicates that perceived social norms are a powerful constraint on potentially problematic behavior (Tankard & Paluck, 2016).Yet, decades of public opinion research has shown that people are pluralistically ignorant-that is, they don't have an accurate impression of that what other people think and do (for an overview, Gunther & Chia, 2001).Research on pluralistic ignorance shows that a majority of people will personally reject a behavior but believe that it is widely prevalent (Gunther & Chia, 2001;Shamir & Shamir, 1997).O'Gorman (1975) illustrated pluralistic ignorance among white Americans who personally rejected racial segregation but believed that racial segregation was widely supported among other white Americans.Communication research has also seen evidence of pluralistic ignorance: Gunther and Chia (2001), for instance, found that people who were supportive of the use of primates in laboratory research overestimated public opposition and perceived the press coverage as largely unfavorable.
The literature on research practices has not engaged with the theory of pluralistic ignorance to the best of our knowledge.In terms of QRPs, due to pluralistic ignorance, one might expect that people would indicate they reject the QRPs but believe they are prevalent.Moreover, if people believe that QRPs are widespread, they are less likely to believe that they are doing anything inappropriate.We will test these possible patterns of pluralistic ignorance in this study but should note that the expectations for pluralistic ignorance were not preregistered and should thus be considered exploratory.

Sub-group Differences in QRPs
We preregistered tests of differences in QRPs for three different subgroups.There are three perspective about the relationship between career stage and QRP usage.First, the use of QRPs is equivalent across career stages, which is documented by some (Chin et al., 2021;Makel et al., 2021).Second, early career researchers may engage more in QRPs because they are more involved in data collection and analyses while under pressure to publish.Early career researchers were, for instance, more likely to be an author of a retracted paper (Fanelli et al., 2015).Yet, there is no evidence that early career researchers self-report more engagement in QRPs compared to later career researchers (Agnoli et al., 2017;Chin et al., 2021;Makel et al., 2021;Martinson et al., 2005;Rabelo et al., 2020).A third, and final, perspective is that late-career researchers engage more in QRPS compared to early career researchers.Later career researchers have had more opportunities to engage in QRPs as they have longer research careers (Martinson et al., 2005).Moreover, late-career researchers-compared to early career researchers-might be less worried about the possible negative consequences of engaging in QRPs and the risk of being caught doing so (Martinson et al., 2005).Later career researchers were indeed more supportive of QRPs than early career researchers (Chin et al., 2021).When it comes to self-reported use of QRPs, surveys confirm that those in later career stages reported more QRPs than those in early career stages (Agnoli et al., 2017;Fraser et al., 2018;Martinson et al., 2005;Rabelo et al., 2020).Therefore, we expect lower self-reported QRPs at early vs. later career stage (H1).Do questionable research practices differ across the world?Among studies that tested this question there was no systematic difference in QRPs across different regions of the world (Agnoli et al., 2017;Makel et al., 2021;Rabelo et al., 2020).In line with this, we expect equivalence in the self-reported QRPs across regions (continents) in the world.(H2).
QRPs might also differ across sub-fields.In psychology, the only discipline where this has been studied, there was no consistent pattern of QRPs being more present in some compared to other sub-fields (Agnoli et al., 2017;John et al., 2012;Rabelo et al., 2020).We have no a priori reason to expect that there are sub-field differences in the use of QRPs.We, therefore, expect equivalence in the self-reported QRPs across sub-fields (ICA-divisions) (H3).
Preregistration means that researchers outline and register their hypotheses, design, and analysis plan, generally, before data collection (DeAngelis et al., 2005;Laine et al., 2007;Nosek et al.,2018).Preregistration prevents researchers from chasing pvalues and selective reporting.Replication means that a researcher tries to replicate a published finding "by duplicating the methodology as exactly as possible" (Chambers, 2019, p. 48).Replication provides the empirical evidence that informs theories in communication (Benoit & Holbert, 2008;Boster, 2002) and tests the foundations of our claims (Chambers, 2019).They are, however, rare in communication (Keating & Totzkay, 2019;Kelly et al., 1979) and often lead to the conclusion that key findings do not replicate or that the effects are much weaker than assumed (e.g., Camerer et al., 2018;Chung & Fink, 2018).Sharing data means researchers make the data they collected publicly available in a repository (Bowman & Spence, 2020).Sharing data does not seem to be a common practice as long as journals don't mandate this, see for instance Elson and Przybylski (2017).A related open science practice is to make studies publicly available so that they are not behind paywalls.
Heretofore, it is largely unknown how communication scholars engage in and evaluate open science practices.We test the following research questions: What is the prevalence of self-reported open science practices among communication researchers?(RQ4); How appropriate do communication researchers find open science practices?(RQ5); What is the perceived improvement of open science practices for communication research?(RQ6).
We preregistered tests of differences in ORPs based upon career stage.Early career researchers might be more receptive to engaging in open science practices (Allen & Mehler, 2019;Farnham et al., 2017).First, in a field like communication, where ORPs are now being discussed and implemented, it could be beneficial for early career researchers to be first movers and adopt ORPs.As a consequence, their work could be evaluated more positively (Allen & Mehler, 2019).Second, open data and materials can help early career researchers be acknowledged for their efforts, establish collaborations and facilitate exchanges.Third, the infrastructure that facilitate ORPs such as the Open Science Framework "can help [early career researchers] in documenting their work, improving workflows, supporting collaborations, and ultimately progressing their training" (Allen & Mehler, 2019, pp. 7-8).Finally, "early career researchers have the least commitment toward professional hierarchy" which would make them more likely to adopt ORPs, while they are also "highly involved in data collection and analysis"-where ORPs can be applied (Farnham et al., 2017, p. 1).
Early career researchers are indeed supportive of ORPs (Chin et al., 2021;Nicholas et al., 2020;Toribio-Flo ´rez et al., 2021) and frequently share their data (Campbell et al., 2019) and publish in open access journal (Nicholas et al., 2020). 3 Therefore, we preregistered that we expect higher self-reported open science practices at early vs. later career stage (H4).

Preregistration
The hypotheses, design, and planned analyses were preregistered before data collection started on the Open Science Framework (https://osf.io/nyxv4/).For one deviation from the preregistered design, see Supplementary Appendix A.1.

Sample
The target population were the 4,664 first and corresponding authors that have (co-)authored a paper that relied upon quantitative methods in one of the top-20 journals in communication science in the past ten years (based upon, Song et al., 2020).Going forward, we define them as quantitative communication researchers as they are researchers who have published a quantitative article in a top-20 journal in the field of communication. 4 We focus on quantitative researchers because the research practices we study might be less applicable to qualitative researchers.
Between September 15 and 23, 2020, 4,664 email invitations were sent: 507 bounced so we contacted 4,157 researchers. 5We sent two reminders to each researcher-roughly seven and 14 days after the first invitation.In total, 1,174 (28.24%) of the sample completed the informed consent and a few people (0.1%, 8/ 1,166) did not agree with the terms.Of the 1,166 people, 1,039 (89.49%) indicated that their work was quantitative.As preregistered, respondents who indicated their work was qualitative (N ¼ 122, 10.51%) were routed to the end of the survey.In total, 872 people (83.9%) completed the survey.But some items have more responses in the analyses due to item non-response. 6To conclude, our AAPOR response rate 1 is 21% and our AAPOR response rate 2 is 28%.Our sample is diverse in terms of career stage, region and ICA division with which one identifies-see Supplementary Appendix A.6.
All quantitative researchers were assigned to a block of questions about questionable research practices and questions about ORPs in random order.In the block with QRPs, participants were exposed to five of the nine QRPs listed in Table 1.For each QRP, respondents were asked four survey questions.Engagement in the practice was measured as a response to the question: "Have you ever engaged in this practice?"on a five-point Likert scale ranging from "Never" (1) to "Almost always" (5).Acceptability of the practice was measured with the question: "What's your opinion of this practice?"scored on a seven-point Likert scale ("Very unacceptable" (1) to "Very acceptable" ( 7)).Perceived prevalence of the practice was measured as a response to the question, "Please estimate the percentage of communication researchers who you believe have engaged in this practice on at least one occasion" which was scored on a slider ranging from 0 -100.Finally, an open-ended response was also permitted, with the prompt "Do you have any additional thoughts related to this research practice?"For all survey items, participants could choose to answer questions but were not forced to answer.Filling in missing data points without reporting that those data were imputed, e.g., through multiple imputation, mean substitution, etc. 3 Excluding data points, such as outliers, after first checking the impact on statistical significance.4 Not reporting studies or key variables that failed to reach statistical significance (e.g., p .05). 5 Reporting a set of results as the complete set of analyses when other analyses were also conducted but these are not reported.6 Reporting an unexpected finding or a result from exploratory analysis as having been predicted from the start.7 Adopting another type of statistical analysis after the analysis initially chosen failed to reach statistical significance.For instance, using OLS instead of logit.8 Adding or dropping covariates in order to reach statistical significance (e.g., p .05) on a key variable.9 Rounding off a p-value to meet a pre-specified threshold (e.g., reporting p ¼.054 as p ¼.05). 1 Preregistering research plans prior to data collection. 2 Replicating the work of other researchers.3 Sharing data you collected to a publicly accessible, online repository.4 Posting copies of your research so that it is not behind a paywall, e.g., on your website, SSRN, or OSF.
We asked four ORPs-see the last four rows of Table 1.In response to each question, we asked the respondents about their engagement, the perceived prevalence, and any open-ended responses.And we asked about their approval of the ORP, worded, "How much would communication research be improved if more people adopted this practice?"answered on a four-point Likert scale ranging from "Not at all" (1) to "A great deal" (4).
Upon completing the QRP and ORP blocks, we measured whether respondents' primary appointment is in communication, their current academic position, the continent where they work, and their sub-field (see Supplementary Appendix A.7).In return for their participation, participants could donate $5 to a charity. 7

QRPs among Quantitative Communication Researchers
In Figure 1, we present the results for the self-reported use, acceptability, and perceived prevalence of QRPs.The table in the top of Figure 1 provides the mean, standard deviation, 95% confidence intervals and responses per research practices for the self-reported use and acceptance of the nine QRPs.The lower half of Figure 1 plots the results for the perceived prevalence of the QRPs.Here the black dot is the mean, the bar represents the 95% confidence interval, and the raw data (dots) and half violin plot provide insight into the data distribution, and the completed responses are listed in the box on the bottom of the panel.
A manual thematic analysis of the open-ended responses was conducted to supplement the quantitative results.Following a bottom-up approach, comments were categorized into one or more high-level arguments supporting or opposing different questionable and ORPs.Here we highlight some of these findings, but a full overview can be found in Supplementary Appendix B.1.
When it comes to collecting more data for a study after first inspecting whether the results are statistically significant, 23.4% report that they have done this at least once.On average, peeking is considered somewhat unacceptable (M ¼ 3.28; SD ¼ 1.71) and perceived as not being very common.We see divided opinions in the open-ended responses as well.In 23% of open-ended responses (31/133), researchers were more lenient in suggesting that collecting additional data might be okay if the study were underpowered.32% (43/133) responses suggested that better practices exist such as pilot studies and power analyses, which, if followed, would ensure that data peeking was not an issue.
Imputing missing values without reporting them was reported by 9% of the researchers.Researchers believe that this practice is unacceptable (M ¼ 1.97; SD ¼ 1.32) but prevalent to some extent (M ¼ 23.94, SD ¼ 20.59).In the openended responses, 43% (45/105) suggest that missing data imputation is acceptable when reported but is unacceptable when unreported.
Roughly a third of our respondents (34%) reported having excluded data points, such as outliers, after first checking the impact on statistical significance at least once.Respondents considered this practice somewhat unacceptable (M ¼ 2.99, SD ¼ 1.66) and researchers think a sizeable percentage of colleagues engages in this practice (M ¼ 41.55, SD ¼ 26.18).However, the open-ended responses paint a divided community.On the one hand, 25% (38/150) of the responses justified that sometimes excluding outliers may be acceptable, depending on when it is done and the appropriateness for a method.On the other hand, 29% (44/150) of the openended responses considered that this would be unacceptable unless the authors are transparent about excluding outliers.
In total, 45% of the researchers indicated that they had adopted another type of statistical analysis after the analysis initially chosen failed to reach statistical significance.Researchers are ambivalent about this practice (M ¼ 3.39, SD ¼ 1.59) and view a sizeable percentage of colleagues engage in this practice (M ¼ 47.70, SD ¼ 27.32).In the open-ended answers, 40% (67/167) of the researchers justified their responses by pointing out that this is often done to fix mistakes.A further 28% (46/167) suggested that it could be appropriate if all analyses were reported in the Supplementary Appendix.
Adding or dropping covariates to reach statistical significance (e.g., p .05) on a key variable is something that 46% of the researchers at least did once.It is considered unacceptable (M ¼ 2.94, SD ¼ 1.61) but relatively prevalent in the discipline (M ¼ 53.16, SD ¼ 27.17).In the open-ended responses, 16% (26/157) suggested that this practice would be okay if it were motivated by theory, not fishing for statistical significance.A total of 24% (37/157) suggested that all deviations should be reported in the paper.
In our sample, 46% report that they at least once engaged in hypothesizing after the results are known (HARKing).HARKing is considered somewhat unacceptable (M ¼ 3.10, SD ¼ 1.65) but is seen as relatively common in the discipline (M ¼ 51.55, SD ¼ 26.60).In the open-ended responses, 25% (44/173) attributed the blame to the prevailing publication bias, while 19% (33/173) argued that HARKing was acceptable in an exploratory analysis.
Rounding p-values off to meet a pre-specified threshold (e.g., reporting p ¼.054 as p ¼.05) has been done at least once by 24% of the researchers.Rounding is considered somewhat unacceptable (M ¼ 2.77, SD ¼ 1.65) and relatively uncommon (M ¼ 37.93, SD ¼ 27.58).In the open-ended responses, 9% (12/124) were optimistic that APA and other journal style guides could reinforce the requirement to report exact p-values up to at least three decimal places, which would render this practice less common.
Not reporting studies or key variables that failed to reach statistical significance (e.g., p .05) is something that 60% of the researchers did at least once.The modal response to this question was "occasionally."Researchers find this practice relatively acceptable (M ¼ 3.27, SD ¼ 1.42). 8But the lack of interest in non-significant results among journal editors and reviewers is according to 46% (81/177) of the openended responses an explanation for this practice.A concern that is also echoed by others in communication research (Levine, 2013).
Reporting a set of results as the complete set of analyses when other analyses were also conducted but not reported yields a similar pattern.Sixty-four percent of researchers indicated that they had done this at least once.Researchers think this research practice is relatively acceptable (M ¼ 3.87, SD ¼ 1.73) and prevalent (M ¼ 60.52, SD ¼ 29.52).The open-ended responses portray the mixed feelings about this practice: 29% (64/215) blame such choices on article space limitations, while 24% (52/215) note that this practice is no longer acceptable as online appendices offer unlimited space to report additional analyse.
How does the field of communication compare to other disciplines?On average, QRPs in quantitative communication research do not seem very different from those documented in other disciplines (see Supplementary Appendix B.3 for a comparison to other disciplines).If anything, HARKing (46%) seems to be a bit more prevalent in communication compared to other fields. 9However, given differences in timing, sampling, question-wording, study design, etc., appropriate caution must be taken when comparing surveys.
One might also wonder whether some people engage in all QRPs, while others never engage in QRPs.An exploratory test-presented in Supplementary Appendix B.4-shows that some QRPs such as not reporting key variables, not reporting all analyses, HARKing, changing statistical tests, adding, or dropping variables, excluding data are positively correlated with each other, while data peeking, imputing data and excluding data are relatively independent of these other QRPs.Hence, engaging in one QRP does not necessarily mean a researcher engages in other QRPs. 10  Is there Evidence for Pluralistic Ignorance?Pluralistic ignorance suggests that many people indicate they reject the QRPs but believe QRPs are prevalent in the discipline.We find widespread pluralistic ignorance.Following the common analytical strategy to test for pluralistic ignorancesee for instance, O'Gorman (1975)-we plot scatter plots for the 9 QRPs in Figure 2. In each QRP, we plot the extent to which researchers find the QRP acceptable (xaxis) and the perceived prevalence of the QRP (y-axis).Pluralistic ignorance would suggest we see a higher density of observations above the diagonal.To further illustrate this, each panel of Figure 2 contains a box which indicates the percentage of individuals that are above the diagonal.Indeed, we find-with exception of data peeking-that people personally don't find the QRP acceptable, but they think it is widely prevalent in the discipline.A similar pattern is found when we replace acceptability for engagement.People personally don't engage in the QRP but believe others do-due to space constraints, these results are presented in Supplementary Appendix B.6.
We also test pluralistic ignorance in a mixed effects model specification.We run a model with perceived prevalence as the dependent variable and included a random intercept for the respondents, the person-level covariates (career stage and discipline) as well as fixed effects for research practice and the acceptance of the QRPs and the engagement in the QRPs (see Supplementary Appendix B.9 for the full model results).This model, like the scatter plot, shows strong evidence for pluralistic ignorance as there is a relatively strong and positive effect of acceptance-recoded to range from 0 (very unacceptable) to 1 (very acceptable)-of the QRP (b ¼.15, p <.01) on perceived prevalencescored from 0 (0 percent) to 1 (100 percent).Likewise, self-reported engagement-scored from 0 (never) to 1 (almost always) in QRPs has an even stronger positive effect on perceived prevalence of the QRP (b ¼.47, p <.01). 11To conclude, there is widespread pluralistic ignorance among quantitative communication researchers.
Are there Sub-group Differences in QRPs?Turning to the preregistered subgroup differences, Table 2 provide the results of a series of preregistered one-sided t-tests where we compare self-reported use of QRPs among early versus later career stage researchers (H1).The tests are under-powered as less than  Note: Preregistered, one-sided t-tests are reported the preregistered 620 respondents completed the different questions 12 -hence some caution is needed. 13Turning to the results, we find that early-career researchers engage less in data peeking (M ¼ 1.29) compared to those at a later career stage (M ¼ 1.56; t(417) ¼À3.41, p(one-sided) ¼ .001).However, for the other QRPs, we do not find statistically significant differences in self-reported QRPs between researchers at the early versus later career stage.Based upon these results, we reject Hypothesis 1. 14  We preregistered that we expected equivalence in the self-reported QRPs across regions (continents) in the world (H2).However, we only had a sizeable number of observations for researchers from the United States and Europe.Therefore, we only test whether self-reported QRPs in the United States and Europe are equivalent.Across nine QRPs, we find consistent evidence that self-reported QRPS are equivalent across the two continents (see Supplementary Appendix B.10).While we cannot test H2 across all continents, we confirm H2 for the continents where we had enough data to test our hypothesis.We also explored whether self-reported QRPs differ between those whose primary affiliation is in communication versus all other fields.We find no systematic evidence that our results differ based upon affiliation using equivalence tests (Supplementary Appendix B.11) or mixed effects models (see Supplementary Appendix B.9). 15  We preregistered equivalence in the self-reported QRPs across sub-fields (H3).However, we have, with a few exceptions, very small numbers of observations in each sub-field (ICA-discipline).This means we cannot compare QRPs across a wide range of sub-fields.Therefore, we decided not to formally test this hypothesis.

ORPs among Quantitative Communication Researchers
In Figure 3, we plot the results for the perceived prevalence, usefulness, and the selfreported use of ORPs.Forty-seven percent of the researchers in our sample have at least once preregistered a study.They believe that preregistration would somewhat improve communication research (M ¼ 2.91, SD¼.88) but think it is still relatively uncommon in the discipline (M ¼ 22.78, SD ¼ 20.39).A majority of our sample-58% of the respondents-indicated that they have, at least once, tried to replicate the work of other researchers.This was also the modal response to that question.Researchers are positive that this practice would improve communication research but indicate that replication is rare in the discipline (M ¼ 28.57, SD ¼ 22.81).
Data sharing is relatively common, as 64% of our respondents indicated that they shared their data at least once.The modal response to this question is that researchers do this occasionally.Like the other open science practices, researchers perceive data sharing as relatively uncommon (M ¼ 30.53,SD ¼ 20.56), while they think that data sharing would improve the discipline (M ¼ 3.25, SD¼.73).
Finally, 85% of our respondents posted a paper on a public repository at least once, which was also the modal response to that question.Researchers think this practice improves communication research (M ¼ 3.33, SD¼.78) but believe that some, but not all, researchers do this (M ¼ 52.25, SD ¼ 23.43).
The open-ended responses to the ORP questions are insightful in understanding the challenges in instituting open practices.Across ORPs, an average of 18% (42/ 237) contend that tenure committees and journals offer little incentive for ORPs.Given the lack of incentive, researchers (15%, or 35/237) lamented that the process was often time-consuming, expensive, and of little utility.
We did not find evidence in line with our expectation that self-reported open science practices among early versus later career stage researchers should be higher (H4, see bottom panel of Table 2).If anything, the pattern is opposite, because later career researchers report greater use of ORPs.Therefore we reject hypothesis 4. 16

Perceptions and Misperceptions about Research Practices
The open-ended responses offer a great deal of insight into the perceptions and misperceptions that researchers have about questionable and ORPs.
According to many researchers, QRPs exist because of the prevalent publication bias-that is, the lack of interest from journal editors and reviewers in studies with non-significant results (see also, Levine, 2013).This was a common answer for rationalizing the exclusion of results (55%, or 81/146 responses) and HARKing (27%, or 44/159 responses).A second related perception expressed in the open-ended responses for reporting analyses selectively (33% or 64/191), indulging in HARKing (27% or 44/159), and selectively reporting methods (44% or 81/181) was that these decisions are made because of journals' space limitations and requests by reviewers.
However, a promising sign is that researchers do perceive and advocate open science practices as a solution to QRPs.On average, 32% (75/237) of researchers agreed that these were good practices that they emulated and encouraged.Moreover, an average of 12% (23/191) of the responses suggested that the additional methods and analyses could be reported as supplementary and unexpected findings since online appendices offer unlimited space.
We also identified a few misperceptions that may explain the presence of questionable research practices and the slow uptake of open science practices.In the open-ended responses corresponding to each of the QRPs, an average of 10% (15/ 156) researchers expressed uncertainty about why specific questionable practices were frowned upon.For instance, respondents considered that choosing to not report all analyses (8% or 15/181), and adding covariates (17% or 27/157) was justified in order to make sure all results reported align with the narrative of the paper.
Second, many researchers are unfamiliar with what open science practices entail.For example, when discussing preregistration, as many as 26% (71/268) indicate that they or their colleagues were unfamiliar with the concept.Finally, some researchers are less convinced by the benefits of ORPs, such as preregistration (22%, 59/268) and data sharing (14%, 33/232).For instance, 13% (35/268) argued that preregistration reduces the value of more inductive research and/or the possibility for scientific discovery.Another 11% (20/268) consider preregistration irrelevant for analyses of secondary data.

Discussion
Our results paint a decidedly mixed portrait of the state and future of quantitative communication research.While many respondents reported having used a QRP in the past, QRPs are generally considered unacceptable, and many respondents were inclined toward open science practices.Further, the results show that the perceived and actual prevalence of QRPs depends on the QRP in question.Very few respondents reported engaging in unreported data imputation, outlier removal, or data peeking.A relatively large proportion of respondents reported engaging in QRPs related to the analyses of the studies (excluding data, changing statistical tests, adding variables), the interpretation of the results (HARKing), and the reporting of the study (not reporting key variables or full analyses).Open-ended responses revealed that respondents engaged in such behavior in response to publication pressures.At the same time, researchers perceive QRPs to be common among their colleagues, but this pessimism overestimates the real picture.That said, respondents are optimistic that the open science agenda will improve communication research.Quantitative communication researchers are, however, not yet fully embracing the open science agenda and do not believe that their colleagues are either.
Our study, of course, has a few limitations.The self-reported use measures of QRPs in this study are subject to social desirability bias.John et al. (2012) found that for the most sensitive QRPs (e.g., HARKing), the self-reported use went up by 7% when truth-telling incentives were implemented.If anything our estimates should thus be seen as conservative estimates.
Another caveat is that we asked if researchers "ever" engaged in QRPs.In the open-ended responses, researchers often indicated that they engaged in a QRP in the past but now realize that this is unacceptable.Future iterations of this survey could ask, for instance, whether researchers have used any of these practices within the past year(s).
Our survey's response rate is quite high for an online survey, and there is little evidence that, on average, response rates affect nonresponse bias (Hendra & Hill, 2019).Appropriate caution must, however, be taken when generalizing the results beyond this sample.First, we limited our study to quantitative communication researchers as some of the QRPs would be less relevant to qualitative researchers (Dienlin et al., 2020).Second, we had relatively few early career researchers in our sample, perhaps because we sampled from researchers who published a paper in the top-20 journals.Third, some ICA-divisions were better represented than other ICA-divisions.

Implications
The questions communication researchers study have, perhaps more than ever before, substantial social and policy implications.As such, the work of communication researchers is more than simple scientific curiosity and informs policymakers, politicians, or the society at large.Hence, it is crucial that we base our conclusions on transparent evidence.
As our survey documents, the discipline does not always live up to that standard.A non-trivial percent of researchers report using one or more QRPs.Findings that are derived via QRPs are likely not to replicate (Asendorpf et al., 2013;John et al., 2012) and could negatively affect the credibility of the research (Anvari & Lakens, 2018).
It is important to consider the broader implications of the widespread use of QRPs in communication.The conclusion that QRPs are widespread also threatens the validity of entire research programs, not only an individual study.Let us illustrate this with an example.It is widely accepted that media effects are conditional (Klapper, 1960;McQuail, 2010;Valkenburg & Peter, 2013).Imagine a research program-i.e., a series of studies-that documents a conditional media effect by heavily relying upon QRPs.The conditional media effect is derived using peeking (e.g., is the conditional effect in the right direction and statistically significant), p-hacking (e.g., specifying models until the conditional effects "works"), HARKing (e.g., presenting an unexpected conditional effect as expected from the start) and publication bias (e.g., only publishing studies that confirm the conditional effect).A research program like this has, ultimately, little evidentiary value.As a consequence, our understanding of the fundamental communication processes-and communication theories, in general-is most likely flawed.
We also find evidence for widespread pluralistic ignorance when it comes to QRPs.While people personally reject QRPs, they think they are widespread.Documenting pluralistic ignorance, our study contributes to the literature on this topic (Gunther & Chia, 2001;Merton, 1968;O'Gorman, 1975) showing its widespread presence in the domain of questionable research practices.At the same time, perceived social norms are a powerful constraint on changing behavior (Tankard & Paluck, 2016).Widespread adoption of open-science badges and other public markers may increase the perceived prevalence of ORPs (Kidwell et al., 2016), and, subsequently, the abandonment of QRPs.
While researchers in our sample are skeptical about the current norms in the discipline, they are also optimistic about the use of open science practices in communication research.Our respondents largely agree that the adoption of open science practices could make research more transparent and thereby increase replicability and credibility of communication research.However, scholars may need to avoid selectively implementing some ORPs which suit their needs while avoiding others.For instance, while researchers support greater transparency in reporting analyses and results, they were more skeptical about the importance of preregistering analyses as a way to control for confirmation biases.
The adoption of ORPs can bring the behavior of quantitative communication researchers' more in line with some of the core values of scientific research as defined by Merton (1942).Sharing of data, methods and research articles aligns with the principle of communality, which holds that sharing of evidence, discussion and open exchange is what defines science.Preregistration and replication could facilitate disinterestedness, which captures the idea that researchers should tell it like it is even if this challenges conventions, contradicts with earlier findings or hurts a reputation.Open data and materials as well as replication facilitate organized skepticism where researchers should critically evaluate all evidence.
What to do next?We have a series of recommendations to spur the uptake of open science practices.Obviously individual researchers bear some responsibility.We encourage researchers to use a declaration form that they can apply to document the processes followed in conducting the research reported in a manuscript.This form can be submitted with a manuscript or posted to a public repository in the interest of transparency.The consensus-based checklist for improved transparency developed by Aczel et al. (2020) is a form useful for quantitative communication researchers, see http://www.shinyapps.org/apps/TransparencyChecklist/.To illustrate this, we have added a completed checklist for this article to our OSF page (see, https://osf.io/b8yt6/).
We believe that institutions-funding agencies, knowledge organizations, journal editors, and hiring and promotion committees-bear the bulk of the responsibility for advocating a change in the research paradigm.As such, institutions and hiring or promotion committees could educate, encourage, and incentivize their faculty and students to engage in ORPs.Teaching classes on research ethics to graduate students, organizing panels on this topic during conferences and rewarding open science practices in hiring and promotion decisions are just some examples.Second, journals and reviewers could discourage QRPs by de-incentivizing them or making them more challenging to attempt.For instance, they could encourage online appendices documenting additional results.Perhaps most importantly given the reasons our respondents stated in the open-ended results for using QRPs, journals should be more receptive to publishing null results (Levine, 2013), which will decrease publication bias.Finally, journals could do more to implement open science practices in their journals.For example, they could implement the registered reports format, mandate data sharing statements and data analysis documentation, and encourage replication studies.
Changes are already underway.Leading journals in communication have published discussions of the open science agenda (Dienlin et al., 2020;Lewis Jr, 2020) (Kidwell et al., 2016;Nosek et al., 2018).Already a few journals-such as Communication Research Reports and Journal of Media Psychology-allow registered reports as a submission format.Other journals, like Political Communication, require data sharing (where possible).
To conclude, we are optimistic about the future of quantitative communication research.Sunlight is the best disinfectant, and by increasing awareness of QRPs and their prevalence, we are confident the credibility of the field will increase.
10.In Supplementary Appendix B.5, we discuss the correlations between acceptability of the different QRPs and the perceived prevalence of the QRPs and the correlation coefficients between self-reported use, acceptability, and perceived prevalence of each QRP.11.It is noteworthy that the intraclass correlation coefficient (ICC) for the QRP model shows some variation (ICC ¼.43) between respondents.12.To be sufficiently powered (b ¼ .8,one-sided p ¼ .05,d ¼ .2),we needed 620 complete responses per QRP.We assigned each respondent to five of the nine QRPs and set our target sample size to N ¼ 1000.Yet, as a reviewer pointed out, with 1000 completes we would only get a net sample size per QRP of 556 (1,000 times 5=9).A sample size of 556 gives us a power of .7 to detect a small effect size (d ¼ .2,one-sided p ¼ .05).Instead, we needed a target sample size of 1,127 to end up with the planned sample size of 620 respondents per QRP.13.Posthoc sensitivity analyses show that we could reliably (power.8)detect effect sizes that range between d ¼ .25 (Not reporting full analyses) and d ¼ .22(data peeking) given our one-sided alpha value of .1 (see Supplementary Appendix B.7).These effect sizes are not much smaller than the preregistered effect size of d ¼ .2. 14.Using two alternative-non-preregistered-modeling strategies, equivalence tests (Supplementary Appendix B.8) and mixed effects models (Supplementary Appendix B.9), we also find no evidence in support of hypothesis H1.Note that the ICC for the QRP models shows marginal variation between respondents.15.The intra class correlation coefficients for the ORP model shows marginal variation between respondents (see Supplementary Appendix B.9). 16.Using two alternative-non-preregistered-modeling strategies, namely equivalence test (Supplementary Appendix B.8) and mixed effects models (Supplementary Appendix B.9), we also find no evidence for hypothesis 4.

Figure 1 .
Figure 1.Self-reported QRPs: Descriptive Statistics of Self-reported Use and Opinion (both in the Table) and Perceived Prevalence of QRPs (in Figure).Note: The figure plots the perceived prevalence of QRPs.The dot is the mean, the bar represents the 95% confidence interval, and the raw data (dots) and half violin plot provide insight into the data distribution.The number of respondents that completed the particular survey item is listed in the box on the bottom within each panel.Notes: Mean of research practices.Standard deviation in parenthesis.For stacked barplots, see Supplementary Appendix B.2.

Figure 2 .
Figure 2. Test of Pluralistic Ignorance: Perceived Prevalence of QRP and Acceptability of the QRP.Note: Scatterplot per QRP with an the x-axis the acceptance of the QRP and on the y-axis the perceived prevalence.The dots are observations.The diagonal line is plotted for easy of interpretation.

Figure 3 .
Figure 3. Self-reported ORPs: Descriptive Statistics of Self-reported Use, Opinion (both in the Table) and Perceived Prevalence of ORPs (in Figure).Note: Perceived prevalence of ORPs.The dot is the mean, the bar represents the 95% confidence interval, and the raw data (dots) and half violin plot provide insight into the data distribution.The number of respondents that completed the particular survey item is listed in the box on the bottom within each panel.Notes: Mean of research practices.Standard deviation in parenthesis.For stacked barplots, see Supplementary Appendix B.1.
, and the International Communication Association devoted its 2020 conference theme to Open Science.Journals like the Journal of Communication and Communication Research have implemented the open science badges-which have been shown to increase the adoption of the open science agenda in other fields

Table 1 .
Questionable Research Practices and Open Research Practices

Table 2 .
Preregistered t-Tests of Differences in Self-reported QRPs and ORPs between Early and Later Career Researchers