Pavlou, Menelaos;
Ambler, Gareth;
Omar, Rumana Z;
Goodwin, Andrew T;
Trivedi, Uday;
Ludman, Peter;
de Belder, Mark;
(2023)
Outlier identification and monitoring of institutional or clinician performance: an overview of statistical methods and application to national audit data.
BMC Health Services Research
, 23
(1)
, Article 23. 10.1186/s12913-022-08995-z.
Preview |
Text
Pavlou_s12913-022-08995-z.pdf Download (1MB) | Preview |
Abstract
Background: Institutions or clinicians (units) are often compared according to a performance indicator such as in-hospital mortality. Several approaches have been proposed for the detection of outlying units, whose performance deviates from the overall performance. Methods: We provide an overview of three approaches commonly used to monitor institutional performances for outlier detection. These are the common-mean model, the ‘Normal-Poisson’ random effects model and the ‘Logistic’ random effects model. For the latter we also propose a visualisation technique. The common-mean model assumes that the underlying true performance of all units is equal and that any observed variation between units is due to chance. Even after applying case-mix adjustment, this assumption is often violated due to overdispersion and a post-hoc correction may need to be applied. The random effects models relax this assumption and explicitly allow the true performance to differ between units, thus offering a more flexible approach. We discuss the strengths and weaknesses of each approach and illustrate their application using audit data from England and Wales on Adult Cardiac Surgery (ACS) and Percutaneous Coronary Intervention (PCI). Results: In general, the overdispersion-corrected common-mean model and the random effects approaches produced similar p-values for the detection of outliers. For the ACS dataset (41 hospitals) three outliers were identified in total but only one was identified by all methods above. For the PCI dataset (88 hospitals), seven outliers were identified in total but only two were identified by all methods. The common-mean model uncorrected for overdispersion produced several more outliers. The reason for observing similar p-values for all three approaches could be attributed to the fact that the between-hospital variance was relatively small in both datasets, resulting only in a mild violation of the common-mean assumption; in this situation, the overdispersion correction worked well. Conclusion: If the common-mean assumption is likely to hold, all three methods are appropriate to use for outlier detection and their results should be similar. Random effect methods may be the preferred approach when the common-mean assumption is likely to be violated.
| Type: | Article |
|---|---|
| Title: | Outlier identification and monitoring of institutional or clinician performance: an overview of statistical methods and application to national audit data |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1186/s12913-022-08995-z |
| Publisher version: | https://doi.org/10.1186/s12913-022-08995-z |
| Language: | English |
| Additional information: | Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data. |
| Keywords: | Outlier detection, Funnel plot, Random effects model, Overdispersion |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Maths and Physical Sciences UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Maths and Physical Sciences > Dept of Statistical Science |
| URI: | https://discovery-pp.ucl.ac.uk/id/eprint/10163299 |
Archive Staff Only
 |
View Item |
