INTRODUCTION Diagnostic test accuracy (DTA) systematic reviews bring together findings from DTA studies to summarise the accuracy of a diagnostic test. Studies included in a DTA review should be assessed for risk of bias and applicability concerns, be cause a review of biased studies, or studies that do not ap ply directly to the review question, could result in misleading conclusions. Studies are most commonly assessed with the QUADAS-2 tool. Anecdotal evidence has suggested that re searchers sometimes struggle to differentiate between risk of bias and applicability. Here, we investigate this distinction for the patient selection and index test domains. OBJECTIVES To develop a framework for assessing the applicability of the study target population and the study index test to the re view defined target populations and index test(s). We aimed to explore review authors’ applicability assessments for the QUADAS-2 patient selection and index test domains, to in form the framework. METHODS DTA reviews were eligible for inclusion if they were pub lished in the Cochrane Library, had used QUADAS-2, and had at least one study rated as “high concerns” for applicability of the patient selection or index test domain. Review selection was checked by a second reviewer. From each review, we extracted article identifiers such as title, authors and publication date, extracted the primary objectives an el ements of review questions: population, index test(s), target condition, reference standard. For each review, we extract ed the rationale provided by the authors for “high concerns” applicability judgements for the respective domains. We also extracted author’s rules how to assess applicability concern with QUADAS-2, whenever it was tailored to the review top ic. One reviewer assessed these rationales and rules, which were verified by another. We also recorded any other issues that arose as part of the applicability assessment, such as sub optimal reporting or erroneous applicability assessment. Two reviewers categorized the rationales inductively into themes, which will be discussed in the QUADAS steering group. The f inal framework will be informed by the identified themes and thorough group discussions. RESULTS This review is in progress. Of the 186 available Cochrane DTA reviews, 123 met our inclusion criteria: 110 for the pa tient selection domain, 75 the index test domain. The data ex traction process is ongoing and final results will be presented at the conference. The majority of the more recent Cochrane DTA reviews include guidance tailored to the review’s top ic. Review authors typically have a broad review question, whereas signaling questions tailored to the review’s topic are more restrictive. Several themes emerged in the patient se lection domain. These include: study setting not matching the review question, study unit not matching the review study unit (lesion level versus patient level); study’s target condition not meeting the review’s target condition (in-vitro versus in-vivo); study’s disease spectrum not covering the review’s targeted disease spectrum (i.e. due to sampling methods, choice of selection criteria; inappropriate exclusions) and study’s indication for testing not sufficiently matching the reviews indication. In the index test domain, themes identified so far include a mismatch in the study and the review with respect to: index test technology; test protocol; thresholds to define the target condition; clinical background of the examiner; experience of the examiner; use of consensus test interpretation rather than use of the interpretation of a single examiner, use of clinical information. In both domains, a number of reviews misunderstood QUADAS-2 applicability or provided insufficient information on the rationale for high concerns. Once all themes are identified, the framework will be developed and present ed as SISMEC. CONCLUSIONS Clear sources of applicability concerns are identifiable, but several Cochrane review authors struggle to adequately identify and report them. At SISMEC, we will present the applicability framework to guide review authors in their assessment of applicability concerns for the QUADAS patient selection and index test domains.
Developing a Framework for Assessing the Applicability of the Target Population and Index Test in Diagnostic Research
Rutjes Anne W S;Lachi Alessio;
2025-01-01
Abstract
INTRODUCTION Diagnostic test accuracy (DTA) systematic reviews bring together findings from DTA studies to summarise the accuracy of a diagnostic test. Studies included in a DTA review should be assessed for risk of bias and applicability concerns, be cause a review of biased studies, or studies that do not ap ply directly to the review question, could result in misleading conclusions. Studies are most commonly assessed with the QUADAS-2 tool. Anecdotal evidence has suggested that re searchers sometimes struggle to differentiate between risk of bias and applicability. Here, we investigate this distinction for the patient selection and index test domains. OBJECTIVES To develop a framework for assessing the applicability of the study target population and the study index test to the re view defined target populations and index test(s). We aimed to explore review authors’ applicability assessments for the QUADAS-2 patient selection and index test domains, to in form the framework. METHODS DTA reviews were eligible for inclusion if they were pub lished in the Cochrane Library, had used QUADAS-2, and had at least one study rated as “high concerns” for applicability of the patient selection or index test domain. Review selection was checked by a second reviewer. From each review, we extracted article identifiers such as title, authors and publication date, extracted the primary objectives an el ements of review questions: population, index test(s), target condition, reference standard. For each review, we extract ed the rationale provided by the authors for “high concerns” applicability judgements for the respective domains. We also extracted author’s rules how to assess applicability concern with QUADAS-2, whenever it was tailored to the review top ic. One reviewer assessed these rationales and rules, which were verified by another. We also recorded any other issues that arose as part of the applicability assessment, such as sub optimal reporting or erroneous applicability assessment. Two reviewers categorized the rationales inductively into themes, which will be discussed in the QUADAS steering group. The f inal framework will be informed by the identified themes and thorough group discussions. RESULTS This review is in progress. Of the 186 available Cochrane DTA reviews, 123 met our inclusion criteria: 110 for the pa tient selection domain, 75 the index test domain. The data ex traction process is ongoing and final results will be presented at the conference. The majority of the more recent Cochrane DTA reviews include guidance tailored to the review’s top ic. Review authors typically have a broad review question, whereas signaling questions tailored to the review’s topic are more restrictive. Several themes emerged in the patient se lection domain. These include: study setting not matching the review question, study unit not matching the review study unit (lesion level versus patient level); study’s target condition not meeting the review’s target condition (in-vitro versus in-vivo); study’s disease spectrum not covering the review’s targeted disease spectrum (i.e. due to sampling methods, choice of selection criteria; inappropriate exclusions) and study’s indication for testing not sufficiently matching the reviews indication. In the index test domain, themes identified so far include a mismatch in the study and the review with respect to: index test technology; test protocol; thresholds to define the target condition; clinical background of the examiner; experience of the examiner; use of consensus test interpretation rather than use of the interpretation of a single examiner, use of clinical information. In both domains, a number of reviews misunderstood QUADAS-2 applicability or provided insufficient information on the rationale for high concerns. Once all themes are identified, the framework will be developed and present ed as SISMEC. CONCLUSIONS Clear sources of applicability concerns are identifiable, but several Cochrane review authors struggle to adequately identify and report them. At SISMEC, we will present the applicability framework to guide review authors in their assessment of applicability concerns for the QUADAS patient selection and index test domains.I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.