El-Osta, Austen;
Webber, Iman;
Alaa, Aos;
Bagkeris, Emmanouil;
Mian, Saba;
Sharabiani, Mansour;
Majeed, Azeem;
(2021)
What is the suitability of clinical vignettes in benchmarking the performance of online symptom checkers? An audit study.
MedRxiv: Cold Spring Harbor, NY, USA.
Preview |
Text
Bagkeris_2021.07.29.21261320.full.pdf Download (401kB) | Preview |
Abstract
Objective: To assess the suitability of primary care vignettes in benchmarking the performance of online symptom checkers / Design: Observational study using publicly available, free online symptom checkers / Participants: Three symptom checkers (Healthily, Ada and Babylon) that provided consultations in English. 139 standardized patient vignettes were compiled by RCGP. Three independent GPs interpreted the vignettes to arrive at a “Gold Standard” consisting of 3 dispositions and divided into one of three categories of triage urgency: (1) emergency care required, (2) primary care required and (3) self-care. / Main outcome measures: Six professional non-medical and lay inputters simulated 2774 standardized patient evaluations using 3 online symptom checkers (OSC). We recorded when OSC provided a triage recommendation and whether it correctly recommended the appropriate triage recommendation across three categories of triage urgency (emergency care, primary care or self-care). We collected data on whether the solution appeared within the first 3 dispositions in each of the standards across 2774 standardized patient evaluations. / Results: When benchmarked against the Gold Standard, Healthily provided an appropriate triage recommendation 61.9% of the time compared to 45.3% and 42.4% of the time for Babylon and Ada respectively. There was poor agreement between OSC consultation outcome and Gold Standard dispositions. When compared to the Gold Standard, Healthily gave an unsafe “under-triage” recommendation 28.6% of the time overall across the three categories compared to 43.3% for Ada and 47.5% for Babylon (P<0.001). / Conclusions: OSCs recommended ‘very unsafe’ triages only <4% of the time suggesting that the online consultation tools are generally working at a safe level of risk. Primary care vignettes are a helpful tool to support development of OSC, but not ideally suited to benchmark the performance of different OSC. Real-world evidence studies involving general practice are recommended to benchmark the performance of OSC in the community setting. /
| Type: | Working / discussion paper |
|---|---|
| Title: | What is the suitability of clinical vignettes in benchmarking the performance of online symptom checkers? An audit study |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1101/2021.07.29.21261320 |
| Publisher version: | https://doi.org/10.1101/2021.07.29.21261320 |
| Language: | English |
| Additional information: | The copyright holder for this preprint is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY-NC-ND 4.0 International license (http://creativecommons.org/licenses/by-nc-nd/4.0/). |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Population Health Sciences > UCL GOS Institute of Child Health > Population, Policy and Practice Dept UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Population Health Sciences > UCL GOS Institute of Child Health UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences |
| URI: | https://discovery-pp.ucl.ac.uk/id/eprint/10147713 |
Archive Staff Only
 |
View Item |
