CMR endpoints for clinical trials: impact of operator experience on the accuracy of image analysis

Background

Cardiac magnetic resonance (CMR) imaging studies are increasingly being used as surrogate endpoints in clinical trials. There have been no previous studies investigating what level of training is required to provide an accurate and robust analysis of these parameters.

Methods

20 CMR studies of acute coronary syndromes were included in this study. 6 observers with three levels of training conducted the image analysis: Observers 1+4: expert SCMR level 3 operators; Observers 2+5: trainees with SCMR level 2 experience; Observers 3+6: a cardiologist and a clinical trial coordinator with no previous CMR experience. The latter level underwent a 4hr tutorial on how to use the software on 10 practice cases. All 6 observers analysed 20 studies, and re-analysed 10 studies 24 hours after. Volumes and mass were analysed using semi-automated software (Argus, Siemens), infarct size (IS) was manually planimetered using a 2SD threshold.

Results

Intra-observer variability was assessed using intraclass correlation coefficient (ICC), inter-observer variability was assessed using Bland Altman plots for agreement. Intra-observer variability was low for volumes, and was highest for mass measurements. (Representative results ICC; observer 1 EDV: 0.97, ESV: 0.99, mass: 0.95, IS: 0.99 vs observer 2 EDV: 0.97, ESV: 0.98, mass: 0.91, IS: 0.98 vs observer 3 EDV: 1.0, ESV: 1.0, mass: 0.91, IS: 0.93). When compared with the most experienced observer, inter-observer variability was highest for IS and increased with decreasing level of experience (Table 1 for representative results for 3 of the 6 observers at 3 differing levels of experience). Bland Altman plots suggested acceptable agreement. Full results for all 6 observers will be presented.

Table 1 Inter-observer variability

Conclusions

Operator experience highly influences the accuracy of calculating CMR parameters. When CMR LV volumes and infarct size are used as surrogate endpoints for clinical trials, a SCMR level 3 observer is the recommended operator to perform the analysis. Based on the results of our study, a SCMR level 2 operator will require additional experience, and a non-experienced operator significant additional training.

Funding

NIHR Cardiovascular Biomedical Research Unit.