Main points

• Reproducibility of relative cerebral blood volume (rCBV) measurement among readers with different experiences is a concern.

• The inter-reader reproducibility of rCBV measurement was poor to moderate when using the hotspot method, regardless of whether it was calculated by neuroradiologists, general radiologists, or radiology residents.

• Sampling five regions of interest and selecting the centrum semiovale for normalization improved reproducibility.

In neuroradiology practice, dynamic susceptibility contrast perfusion magnetic resonance imaging (DSC–MRI) is used extensively as an advanced method for the diagnosis, grading, and post-treatment follow-up of glioblastomas.^1,2 The DSC–MRI technique depends on a susceptibility-caused signal loss on T2*-weighted images resulting from a bolus passage of gadolinium-based contrast media. Cerebral blood volume (CBV) is the most commonly used parameter of DSC–MRI and it defines the area under the concentration–time curve.³ CBV is basically an absolute value however, it has some assumptions/conditional requirements. Therefore, to obtain a relative quantification, it is usually rated to a reference point, such as contralateral white matter, the centrum semiovale, or arterial input function.⁴ The relative CBV (rCBV) is the most robust and commonly used DSC–MRI parameter for the radiological characterization of glioblastomas.^3,5 It has been proven that the rCBV value is highly correlated with tumor grade, vascularity, and prognosis.^2,3 Moreover, rCBV has been shown to be useful in distinguishing tumor progression from its mimickers, such as pseudoprogression and radiation necrosis.^3,6,7

To calculate rCBV, radiologists draw regions of interest (ROIs) in the most hyperperfused area of a tumor and a normal-appearing reference area.⁸ In ROI analysis, determining the hyperperfused area on CBV map images and selecting the normal-appearing reference area are two important operator-dependent subjective issues. Despite its operator dependency and inter-observer variability, this is the most widely used method in clinical practice.^9,10,11,12

Radiologists with different experience levels can potentially assess DSC–MRI. However, the assessment of DSC–MRI parameters by radiologists with different levels of experience, such as radiology residents, general radiologists, and neuroradiologists, may lead to inconsistent evaluations in the diagnosis and treatment processes. It is crucial to improve a reliable and standard analysis method for rCBV measurement to eliminate incompatibility between different users. It has been recommended that DSC–MRI measurements should be reviewed by two experienced radiologists and an adjudicator should be consulted in the event of disagreement.¹³ However, this recommendation is time-consuming and not always applicable; therefore, it is impractical for clinical practice. In clinical practice, the assessment of DSC–MRI by a single radiology resident (when preparing reports), a general radiologist, or an experienced neuroradiologist is not uncommon. Residents and general radiologists do not have the opportunity to consult an experienced neuroradiologist in all DSC–MRI examinations.

To our knowledge, there is limited literature on the reproducibility aspects of rCBV in terms of sampling, normalization location, and reader experience. Therefore, we sought to fill this knowledge gap in the literature by assessing the inter-reader reproducibility of rCBV measurements in glioblastoma cases in pre-surgical settings, calculated either by radiology residents, inexperienced general radiologists, or experienced neuroradiologists. Second purpose was to investigate the effect of the number of ROI and the selection of reference areas on inter-reader reproducibility.

Methods

Ethics

For this retrospective study, institutional review board approval was acquired from the Local Medical Ethics Committee of Basaksehir Cam and Sakura City Hospital (no: 2023-18; decision date: 11/01/2023). The requirement for written informed consent was waived by the ethics committee due to the study’s retrospective design. The study adhered to the principles of the Helsinki Declaration.

Study population

In this single-center retrospective study, 30 patients with newly diagnosed glioblastoma were consecutively included between July 2021 and January 2023. The reason for including 30 cases was to meet the minimum requirement for a reliability analysis.¹⁴ This study was conducted in a tertiary academic hospital.

Eligibility criteria

The inclusion criteria were as follows: i, a definitive histopathologic diagnosis of glioblastoma according to the World Health Organization’s 2021 classification;¹⁵ ii, ≥18 years old at the time of DSC–MRI.

The following criteria determined exclusion: i, history of cranial surgical resection or biopsy, or radiotherapy, chemotherapy, corticosteroid, or anti-angiogenic treatment before imaging; ii, MRI scans with severe artifacts that impeded the diagnostic evaluation.

Magnetic resonance imaging acquisition protocol and processing

All MRI scans were performed at a 3.0 Tesla system (Ingenia, Philips Healthcare) using 32-channel phased-array head coils in the supine position. The routine DSC–MRI examination protocol included axial spin–echo T1-weighted imaging [repetition time (TR)/echo time (TE): 600/10 ms; field of view (FOV): 230 mm; slice thickness: 4 mm; matrix: 208 × 165; number of excitations (NEX): 1], a 3D axial fluid-attenuated inversion recovery sequence (TR/TE: 4,800/340 ms; inversion time: 1,650 ms; FOV: 230 mm; slice thickness: 4 mm; matrix: 272 × 243; NEX: 1), subsequent DSC–MRI data, and finally, 3D postcontrast T1-weighted imaging (TR/TE: 600/10 ms; FOV: 230 mm; slice thickness: 4 mm; matrix: 208 × 165; NEX: 1).

DSC–MRI was obtained on an axial plan with a gradient-echo echoplanar imaging technique using the following parameters: TR/TE: 1,500/30 ms; FOV: 237 × 237 mm; matrix: 128 × 128; section thickness: 3 mm; flip angle: 60°; voxel size: 2.33 × 2.39 × 4.00 mm. An intravenous bolus injection of gadolinium-based contrast agent was given at a dose of 0.1 mmol/kg and a speed of 5 mL/s, followed by a 20-mL saline flush. Before the dynamic phase, a saturation pre-bolus of contrast agent was administered as a preload to reduce contaminating T1 effects from contrast agent leakage. The DSC–MRI protocol of this study was in line with consensus recommendations.¹⁶

Dynamic susceptibility contrast perfusion magnetic resonance imaging processing and observer setting

The DSC–MRI datasets were processed using the IntelliSpace Portal (Philips). Maximum rCBV values were calculated independently by three groups of readers (six readers in total). Individual readers were denoted as R1, R2, R3, R4, R5, and R6.

Readers R1 and R2 were neuroradiologists with 5 years of experience in neuroimaging in research settings. Readers R3 and R4 were general radiologists with 4 years of experience in general radiology who were working in all divisions of radiology. Readers R5 and R6 were third-year radiology residents who had completed neuroradiology rotations at the beginning of this study. All readers were blinded to the clinical information and histopathological results of patients. All data were anonymized.

Inter-reader reproducibility was assessed using various strategies as follows: i, between individuals with the same experience levels (R1 vs. R2, R3 vs. R4, and R5 vs. R6); ii, between individuals with different experience levels (R1 vs. R3, R3 vs. R5, and R1 vs. R5). Since two readers in each group had similar experience levels and to avoid complicating the analysis, only one reader was randomly selected from each group and compared with the other readers; iii, between reader groups with different experience levels (neuroradiologists vs. general radiologists, neuroradiologists vs. residents, and general radiologists vs. residents); iv, inter-reader reproducibility of group-wise (neuroradiologists vs. general radiologists vs. residents) and reader-wise (R1 vs. R2 vs. R3 vs. R4 vs. R5 vs. R6) pooling.

Region of interest analysis

The reviewers were encouraged to place five different ROIs within the tumor that visually appeared as mostly hyperperfused on colored relative CBV map images (hotspot method). The CBV value of the first ROIs (CBV₁), the highest CBV value among the first three ROIs (CBV₃), and the highest CBV value among five ROIs (CBV₅) were recorded. Only the highest CBV value among the three and five ROIs was used (not their means). Then, the reviewers were instructed to place an ROI of the same size on the contralateral normal-appearing white matter in the same axial section as the tumor’s ROI and the contralateral normal-appearing centrum semiovale, which are known to be the most reliable reference areas.¹³ The CBV values obtained from CBV₁, CBV₃, and CBV₅ were rated to these reference areas to obtain a normalized rCBV. Statistical analyses were performed separately for each reference area.

All circular ROIs were drawn manually by the readers on CBV map images and ranged between 40 and 60 mm². In the ROI analyses, care was taken to avoid hemorrhagic, necrotic, or cystic regions, normal grey matter, and intralesional non-tumor large vessels that might affect the values. A multi-ROI analysis of a glioblastoma case is represented in Figure 1.

Statistical analysis

The statistical analysis was performed using R 4.3 (main packages: “MKinfer” and “ggpubr” for reliability analyses and inferential statistics; tool: JASP for descriptive statistics only)¹⁷ and Python 3.7 (main package: pingouin for reliability analyses) environments.¹⁸ To assess feature reliability, the mean and 95% CI values of the intraclass correlation coefficient (ICC) were calculated. The ICC was based on the type of ICC (2,1) according to Shrout and Fleiss’s convention.¹⁹ The interpretation scale for the ICC was as follows: <0.5: poor; ≤0.5 to <0.75: moderate; ≤0.75 to <0.9: good; and ≥90: excellent.¹⁴ In addition to the ICC analysis, non-parametric Bland–Altman analyses were performed to evaluate the differences in measurements and the limits of agreement, relying on median and 2.5–97.5^th percentiles, respectively. The Shapiro–Wilk test was used to determine the normality of continuous variables. Depending on the group distributions, a paired non-parametric test, the Wilcoxon signed-rank test, was used to assess pairwise statistical differences in continuous variables. Statistical results were considered significant if P <0.050. In the case of multiple comparisons, the results were considered significant if the adjusted P values were <0.050 after multiplicity correction using the Bonferroni method.

Results

Patient characteristics

In total, 30 consecutive glioblastoma cases were enrolled, 17 were male, and 13 were female. The mean age of the patients (standard deviation) was 61.1 (9.7) years (range: 38–78 years).

According to the number of ROIs and reference areas, the median rCBV values ranged between 13.7 and 20.1 for neuroradiologists, 18.1 and 22.1 for general radiologists, and 10.1 and 12.8 for residents. The median and the interquartile range (IQR) of the rCBV values of all readers, which were calculated by using the hotspot method, are presented in Supplementary Table 1.

Inter-reader reproducibility for individuals with similar experience levels

The inter-reader reproducibility of the rCBV measurements was poor to moderate (ICC range: 0.288–0.527). The highest inter-reader reproducibility of the rCBV measurements was obtained between neuroradiologists (R1 vs. R2) using one ROI and normalization with white matter (ICC: 0.527; 95% CI: 0.21–0.74) and between residents (R5 vs. R6) using one ROI and normalization with the centrum semiovale (ICC: 0.513; 95% CI: 0.20–0.73). The ICC value of general radiologists (R3 vs. R4) increased from 0.312 to 0.370, with a higher number of ROIs (from one to five) using the centrum semiovale as a reference area. The ICC value of each analysis is presented in detail in Table 1.

Inter-reader reproducibility for individuals with different experience levels

In all analyses, the inter-reader reproducibility of the rCBV measurements was poor (ICC range: 0.296–0.335). However, an increase in ICC values was observed when five ROIs were used instead of one or three ROIs, even if contralateral white matter or the centrum semiovale is used for normalization (Supplementary Table 2).

Inter-reader reproducibility for reader groups with different experience levels

In the analysis of reader groups, the inter-reader reproducibility of rCBV measurements was moderate (ICC range: 0.566–0.640) for neuroradiologists vs. general radiologists. On the other hand, inter-reader reproducibility was poor for all the other group-based analyses. The ICC values ranged between 0.350 and 0.422 for neuroradiologists vs. residents and between 0.254 and 0.334 for general radiologists vs. residents (Table 2).

Inter-reader reproducibility for group-wise and reader-wise pooling

In both the group-wise (all groups pooled) and reader-wise (all readers pooled) pooling analyses, inter-reader reproducibility was poor. The ICC ranged between 0.397 and 0.427 and between 0.296 and 0.335 for group-wise pooling and reader-wise pooling, respectively. Similar to the results of individuals with different experience levels, an increase in ICC values was observed when five ROIs were used. All the ICC values are presented in Table 3.

Analysis of the overall reader population of perfusion magnetic resonance imaging

An additional analysis, that included all the readers, groups, and pooled analyses, was performed (n = 66). The ICC values of all the analyses performed are summarized in Figure 2.

While the inter-reader reproducibility of the general radiologists (R3 vs. R4) was poor (Table 1), six of the top 10 most reproducible analyses involved neuroradiologists vs. general radiologists. Among all the analyses, the top two in terms of inter-reader reproducibility was for neuroradiologists vs. general radiologists using the centrum semiovale, with an ICC value of 0.640 (95% CI: 0.32–0.82) and 0.583 (95% CI: 0.29–0.78) for one ROI and five ROIs, respectively. The top 10 most reproducible results among all the analyses are presented in Table 4.

In this analysis, the median of the ICC value was 0.349 (IQR: 0.116) for the centrum semiovale and 0.305 (IQR: 0.107) for white matter. The inter-reader agreement was higher, with a statistically significant difference for the centrum semiovale (P < 0.001). Comparisons were performed according to the number of ROIs. The median of the ICC value was 0.335 (IQR: 0.117) for five ROIs, 0.321 (IQR: 0.128) for one ROI, and 0.316 (IQR: 0.112) for three ROIs. There was a significant difference between three and five ROIs, including all reference areas (P < 0.001). In addition, a significant difference was observed between three and five ROIs when using one of the two reference areas (P < 0.010). There was no significant difference in the use of one or three ROIs (P > 0.050). All results are summarized in Figure 3.

Figures 4 and 5 show Bland–Altman plots of the readers based on centrum semiovale normalization with the same and different experience levels, respectively. Supplementary Figures 1 and 2 show Bland–Altman plots based on white matter normalization. In all the Bland–Altman analyses, the vast majority of ROI measurements were within the upper and lower agreement limits. In the analysis based on white matter normalization, most of ROI measurements by neuroradiologists were quite close to the zero line (Supplementary Figure 1). The same condition was observed for residents using centrum semiovale normalization (Figure 4). In general, ROI measurements were far from the zero line for general radiologists (Figure 4 and Supplementary Figure 1).

Discussion

In the present study, we investigated the inter-reader reproducibility of rCBV measurements in patients with untreated glioblastomas. Generally, a moderate inter-reader agreement was observed when analyses were made between individuals with the similar experience level. In contrast, there was poor inter-reader reproducibility when the analyses were made between different experience levels. According to all the analyses performed, the inter-reader agreement of rCBV measurements when using the centrum semiovale as a reference area was significantly higher than when using contralateral white matter. In addition, the ICC values for the placement of five ROIs were significantly higher than with one or three ROIs.

In clinical radiology practice, T2*-weighted perfusion MRI is often used for gliomas. The radiology community pays relatively little attention to the reproducibility of the derived parameters, despite their importance. In this study, we focused on preoperative glioblastoma cases to make the findings more evident, i.e., to assess the reliability of obviously high perfusion values. In clinical practice, reproducible perfusion parameters are essential for the consistent target area selection of gliomas in stereotactic biopsies and for establishing consistent baseline perfusion parameter values for use in post-treatment follow-up scans. Additionally, the reproducibility of these parameters is necessary for research consistency to ensure that the results of different studies are comparable, which may increase the validity of the conclusions drawn from pooled data and meta-analyses.

In previously published reliability studies, only normal-appearing contralateral white matter^20,21 or the contralateral centrum semiovale¹³ has been generally selected as a reference area. In these studies, three observers were selected to assess DSC–MRI, including only neuroradiologists^13,21 or neuroradiologists and a resident.^20,22 We noticed that there was sparse radiological research literature on inter-reader reproducibility analyses of rCBV measurements for a population that includes neuroradiologists, general radiologists, and residents in the same study. According to our experience and knowledge, in clinical practice, general radiologists and residents (when preparing reports to present to neuroradiologists) may have to evaluate DSC–MRI, although this is not as common as with neuroradiologists. Therefore, inter-reader reproducibility of general radiologists and residents within themselves and between other reviewers is a matter of concern.

Definitive interpretations of DSC-MRI in patients with glioblastoma should be conducted and interpreted by specialized neuroradiologists. The most valuable advantage and originality of our paper is that it included two different groups in addition to neuroradiologists in the same study. For all analyses, normalization with the centrum semiovale acquired better inter-reader reproducibility than with white matter. According to our observation, the reason for this difference could be explained by the fact that the centrum semiovale is a large homogenous area that is mostly visible in more than two axial slices, and it suffers less from partial volume artifacts compared with contralateral white matter. In a recently published retrospective study, Roques et al.²⁰ evaluated the inter-observer reproducibility of rCBV measurements in 27 cases of untreated glioblastoma. In that study, three observers (two neuroradiologists and a radiology resident) calculated the maximum rCBV values independently using the hotspot method, similar to our study. However, differently, they used only contralateral white matter as a reference to normalize the CBV value. Their inter-observer reproducibility for maximum rCBV value measurements was fair [ICC: 0.46 (0.22–0.67)], but their inter-observer reproducibility was found to be poor to fair (ICC range: 0.30–0.47) when a resident was added to the analysis. Our research confirms the variability of the hotspot method, similar to the results of Roques et al.’s²⁰ study. However, their study did not include general radiologists and did not assess inter-observer reliability among radiology residents or general radiologists, which are the main differences of our study. Another advantage of our research is the use of leakage correction with a gadolinium preload to avoid the underestimation of CBV values. Furthermore, our study assessed the effect of selecting two different reference areas on inter-observer reliability.

The present study has the following limitations: a small sample size, a retrospective nature, and involvement of only a single institution. In this research, we only studied cases with untreated glioblastomas, in which DSC–MRI perfusion is anticipated to be increased. This study was conducted only on the initial DSC–MRI examinations of untreated glioblastomas and did not include the evaluation of post-treatment perfusion MRI features. Including only the baseline DSC-MRIs in our study may have contributed to reliability.

In conclusion, there is poor to moderate inter-reader reproducibility of rCBV measurements using the hotspot method, regardless of whether they are calculated by neuroradiologists, general radiologists, or radiology residents. This may indicate the need for automated methods. Selecting five ROIs and using the centrum semiovale as a reference area for normalization may increase the inter-reader reproducibility of measurements.

Conflict of interest disclosure

Burak Koçak, MD, is Section Editor in Diagnostic and Interventional Radiology. He had no involvement in the peer-review of this article and had no access to information regarding its peer-review. Other authors have nothing to disclose.