Understanding the difference between risk of bias and concerns regarding applicability in the reference standard domain of QUADAS-2: A methodological review [Poster]

Background Diagnostic test accuracy (DTA) systematic reviews bring together findings from DTA studies to summarise the accuracy of a diagnostic test. Studies included in a DTA review should be assessed for risk of bias and applicability concerns, because a review of biased studies, or studies that do not apply directly to the review question, could result in misleading conclusions. Studies are most commonly assessed with the QUADAS-2 tool. Anecdotal evidence has suggested that researchers sometimes struggle to differentiate between risk of bias and applicability. Here, we investigate this distinction for the reference standard domain. Objectives To highlight challenges with the assessment of the applicability of the reference standard within Cochrane DTA reviews, including occurrences in which risk of bias issues are mistaken for applicability issues. We will identify examples of applicability concerns, discuss the distinction between risk of bias and applicability, and explore where the reference standard domain intersects with other domains. Methods DTA reviews were eligible for inclusion if they were published in the Cochrane Library, had used QUADAS-2, and had at least one study rated as “high concerns” for applicability of the reference standard domain. From each review, we extracted title, authors, publication date, URL, population, index test(s), target condition, reference standard, outcomes, objective, and the rationale provided by the authors for “high concerns” applicability judgements for the reference standard domain. One reviewer assessed these reasons and coded them as either being correctly or incorrectly related to applicability (i.e. actually a risk of bias issue). We also recorded any other issues that arose as part of the applicability assessment. Review selection and data extraction was checked by a second reviewer. Results This review is in progress. We have identified 50 DTA reviews that meet our inclusion criteria. Results will be presented at the conference. Conclusions Understanding the use of a risk of bias and applicability tool in research is important to ascertain the usefulness and limitations of the tool. The findings of this review will inform the development of QUADAS-3, which in turn may positively affect the robustness of evidence production.