Session 8: Assessing Error Rates in Multiple Examiner Groups using Regression Methods
Presentation Type
Invited
Student
No
Track
Forensic Statistics
Abstract
I
The receiver operating characteristic (ROC) curve is used to measure classification accuracy of tests which yield ordinal or continuous scores. Ordinal scores occur commonly in medical imaging studies and more recently in black-box studies on forensic identification accuracy (Phillips et al., 2018). To assess the accuracy of radiologists in medical imaging studies or the accuracy of forensic examiners in biometric studies, one needs to estimate the ROC curves from the ordinal scores and also account for the covariates related to the radiologists or forensic examiners. In this talk, we propose a homogeneity test for ordinal ROC curves to determine differences in accuracy among multiple rater groups while adjusting for covariates. The test relies on the derived covariance structure among the estimated covariate-specific ordinal ROC curves based on ordinal ROC regression. We conducted extensive simulation studies to evaluate the finite sample performance of the proposed test. The simulation results show that estimated ROC curves are consistent and the empirical coverage of the confidence intervals is close to the nominal level. Our proposed test is applied to a face recognition study in which participants include facial examiners, facial reviewers, super-recognizers, fingerprint examiners and students. We find that there exits difference in accuracy among five rater groups. Ad-hoc pairwise comparison tests are then conducted by establishing confidence bands of difference among ROC curves. Those pairwise tests identify statistically significant differences in ROC curves among five participant groups.
Start Date
2-7-2023 11:00 AM
End Date
2-7-2023 12:00 PM
Session 8: Assessing Error Rates in Multiple Examiner Groups using Regression Methods
Pasque 255
I
The receiver operating characteristic (ROC) curve is used to measure classification accuracy of tests which yield ordinal or continuous scores. Ordinal scores occur commonly in medical imaging studies and more recently in black-box studies on forensic identification accuracy (Phillips et al., 2018). To assess the accuracy of radiologists in medical imaging studies or the accuracy of forensic examiners in biometric studies, one needs to estimate the ROC curves from the ordinal scores and also account for the covariates related to the radiologists or forensic examiners. In this talk, we propose a homogeneity test for ordinal ROC curves to determine differences in accuracy among multiple rater groups while adjusting for covariates. The test relies on the derived covariance structure among the estimated covariate-specific ordinal ROC curves based on ordinal ROC regression. We conducted extensive simulation studies to evaluate the finite sample performance of the proposed test. The simulation results show that estimated ROC curves are consistent and the empirical coverage of the confidence intervals is close to the nominal level. Our proposed test is applied to a face recognition study in which participants include facial examiners, facial reviewers, super-recognizers, fingerprint examiners and students. We find that there exits difference in accuracy among five rater groups. Ad-hoc pairwise comparison tests are then conducted by establishing confidence bands of difference among ROC curves. Those pairwise tests identify statistically significant differences in ROC curves among five participant groups.