- Open Access
MRXCAT2.0: Synthesis of realistic numerical phantoms by combining left-ventricular shape learning, biophysical simulations and tissue texture generation
Journal of Cardiovascular Magnetic Resonance volume 25, Article number: 25 (2023)
Standardised performance assessment of image acquisition, reconstruction and processing methods is limited by the absence of images paired with ground truth reference values. To this end, we propose MRXCAT2.0 to generate synthetic data, covering healthy and pathological function, using a biophysical model. We exemplify the approach by generating cardiovascular magnetic resonance (CMR) images of healthy, infarcted, dilated and hypertrophic left-ventricular (LV) function.
In MRXCAT2.0, the XCAT torso phantom is coupled with a statistical shape model, describing population (patho)physiological variability, and a biophysical model, providing known and detailed functional ground truth of LV morphology and function. CMR balanced steady-state free precession images are generated using MRXCAT2.0 while realistic image appearance is ensured by assigning texturized tissue properties to the phantom labels.
Paired CMR image and ground truth data of LV function were generated with a range of LV masses (85–140 g), ejection fractions (34–51%) and peak radial and circumferential strains (0.45 to 0.95 and − 0.18 to − 0.13, respectively). These ranges cover healthy and pathological cases, including infarction, dilated and hypertrophic cardiomyopathy. The generation of the anatomy takes a few seconds and it improves on current state-of-the-art models where the pathological representation is not explicitly addressed. For the full simulation framework, the biophysical models require approximately two hours, while image generation requires a few minutes per slice.
MRXCAT2.0 offers synthesis of realistic images embedding population-based anatomical and functional variability and associated ground truth parameters to facilitate a standardized assessment of CMR acquisition, reconstruction and processing methods.
In-silico phantoms of human cardiovascular anatomy and function provide a versatile tool for the testing and validation of image acquisition, reconstruction and post-processing strategies in cardiovascular magnetic resonance (CMR) . Producing synthetic images from a phantom has the benefit that the resulting images have corresponding anatomical labels and functional ground-truth data, which are useful for the evaluation of the performance of a CMR pipeline. For example, the availability of a paired image-ground truth dataset is essential for a standardized evaluation of image processing tools, such as those for automatic left-ventricular (LV) segmentation, shape and strain analysis.
Available phantoms can be classified into three categories: voxel-based, analytical and hybrid. Voxel-based phantoms consist of labeled voxelised anatomical representations obtained from patients [2, 3]. These are realistic, but do not generalize to population statistics and pathological cases . Analytical phantoms are based on a mathematical description of tissue structures . While they are less realistic, they are more flexible in terms of definition of anatomical variations. Hybrid phantoms have been proposed to overcome the limitations of the previous two categories . Although hybrid and analytical phantoms allow for morphological variation, anatomical and functional variability is mostly limited to healthy cases and function. Veress et al.  proposed to couple a hybrid phantom to a biophysical model of the LV to simulate both healthy and infarct conditions. However, as stated by the authors, the fitting process is time consuming and it cannot account for other pathological scenarios, such as cardiomyopathy. More recently, Segars et al.  proposed a methodology to couple a full heart functional model to the XCAT phantom. While this allows to simulate realistic cardiac function, it is specific to XCAT and it cannot be rapidly deployed to general pathological cases.
In the last years, solutions based on shape models (SM) (with voxelised or mesh representations) have been proposed to address the need for expressive anatomical descriptions [9,10,11,12,13]. While these works have shown the capability of representing dominant LV anatomical features, they did not focus on the definition of a sampling strategy to generate synthetic anatomies to capture population variability, including both healthy and pathological conditions.
Given an in-silico phantom, two main methodologies for generating CMR images can be identified. In the first approach, the signal is generated using numerical solutions of the physical equations (Bloch equations). This has been applied for cardiac and brain image synthesis [14,15,16,17,18]. In [1, 19] the use of signal models for specific sequences of interest has been proposed to compute the resulting image data. In  a dataset for a virtual population with varying acquisition parameters was generated using MRXCAT  and used to pre-train a segmentation network, which was subsequently fine-tuned on real images. This approach greatly reduces the amount of in-vivo images required. However, the segmentation performance degraded when there was no fine-tuning on real data as the simulated images were not completely realistic. In [16, 21, 22] it has been shown that synthetic images can be used to augment, and eventually replace, in-vivo datasets for training of neural networks, making realistic image synthesis an important tool for CMR development.
Alternative generative approaches consist of using neural networks for conditional synthesis or style transfer [23,24,25,26,27,28,29,30]. They have been used for several imaging modalities such as ultrasound , computed tomography  and magnetic resonance imaging [23,24,25, 32]. The reader is referred to  for a recent overview of medical image synthesis.
In [26, 34] a factorised representation of images has been proposed, composed of a spatial representation of the anatomy combined with a modality description. The latter describes how tissue structures are rendered in the image. However, the network cannot be used to generate new anatomies and it requires labelled images for training, which are costly to obtain. In  unlabelled CMR images were used to learn a multi-tissue anatomical model which was fit to variable anatomies by a learned deformation model. The anatomical model was then used to condition a SPADE-GAN  to synthesise an image volume. While this approach solved both issues of the two previous factorised representation learning approaches [26, 34], the anatomical model learnt using the network does not represent conventional tissue classes and is thus not suited as anatomical ground truth. In , the XCAT phantom was used as anatomical ground truth semantic labels and MR images were synthesized using a SPADE-GAN. In , DatasetGAN, leveraging the generator features of StyleGAN , was proposed to produce a large synthetic dataset of images and to also predict pixel-wise class labels. The evaluation of this method has demonstrated that a segmentation network trained with datasets from DatasetGAN outperforms previous semi-supervised methods and is on par with the same network trained fully-supervised on a real dataset. Similarly, SemanticGAN  was developed to simultaneously generate both synthetic images and corresponding segmentation labels using StyleGAN2.
While physics-based approaches allow for better control over the parameters related to image generation with respect to style transfer approaches, they produce less realistic appearance. In  intra-organ texture for bones and organs was proposed to improve the realism of images generated with signal models. This approach, however, has not yet been applied to CMR image synthesis.
The present work proposes MRXCAT2.0 to address the two main limitations of in-silico phantoms: reduced variability and lack of realism. Realistic LV anatomy and function are generated by coupling a statistical shape description with a biophysical model. Surrounding tissue structures are generated with the XCAT model. Tissue maps of proton density (PD), longitudinal and transverse relaxation times (T1, T2) are assigned to image labels using a neural network trained to maximize the similarity of the background with the target appearance of real CMR images. Synthetic images are then generated using MRXCAT2.0 and used to assess the performance of published CMR processing methods [40, 41] against known ground truth of healthy and pathological cardiac function as a use case.
The full method of MRXCAT2.0 is schematically shown in Fig. 1. In the figure, red boxes correspond to the parts of the methods that are connected to each other via input/outputs of the black boxes. The final outputs are the synthetic CMR images paired with ground truth data (green box). The starting points are the two inputs: the selection of the (patho)physiological characteristics of anatomy and function and the parameters for the XCAT phantom (blue boxes). The (patho)physiological status is used to define the corresponding anatomy and tissue micro-structure from the statistical shape model and the appropriate physiological parameters (tissue stiffness, pressure loading, myocyte contraction) for the biophysical simulation that generates the image foreground, i.e. the LV shape and its change over the cardiac cycle. The XCAT parameters are used to define the torso anatomy and the displacement field describing the contraction of the other three cardiac chambers. This is referred to as the background of the image. The background tissue masks are warped to match the foreground and the resulting tissue maps are the input to the texturizer for the calculation of tissue properties (PD, T1, T2) and the definition of the final phantom. These properties are used as input to the signal model to generate synthetic CMR images associated with the input parameters and compliant with fundamental LV biomechanics.
Left-ventricular population shape model
The LV SM was defined using the anatomies from the Multi-Modal Whole Heart Segmentation (MMWHS) dataset [42,43,44] as built in our recent work .
A convolutional variational autoencoder (VAE)  was used to identify a low-rank representation of epicardium and endocardium coordinates (see Additional file 1 for details). The network structure is shown in Fig. 2. Each variable of the low-rank representation is associated with a normal Gaussian probability distribution, which is sampled to generate synthetic realistic endocardial and epicardial shapes from which it is possible to generate a volumetric mesh .
The expressiveness of the SM was assessed using an additional dataset, the Automated Cardiac Diagnosis Challenge (ACDC) . End-systolic images were meshed using our recently published method  and the accuracy of the reconstruction with the SM was evaluated as the average distance between corresponding endocardial and epicardial points in the original and reconstructed meshes. Additionally, a k-means clustering algorithm  was used on the latent space representation of the ACDC meshes with three target clusters to identify sampling regions of the latent space for the (patho)physiological conditions labelled in the ACDC dataset (healthy (NOR), dilated (DCM), hypertrophic (HCM)). Classification accuracy of healthy, DCM and HCM was evaluated against the clinical labels. The centres of the clusters were then used as reference anatomies for showcasing the method proposed in this work.
Cardiac functional model
The biophysical model for the LV is based on our previous work on cardiac mechanics  and material modelling [48,49,50,51]. A technical description is presented in the Supplementary material and in .
The response of the LV to the systemic pressure loading depends on the contribution of a passive and an active component. The passive component was described by the Holzapfel-Ogden model  defined as a function of tissue shear moduli and fiber orientations. In our approach, the evolution of the active contribution was simulated as in [53,54,55]. In the model, the pericardial sac was simulated by allowing for longitudinal motion of the points but constraining epicardial radial displacement. Endocardial pressure was simulated by coupling the ventricular model to a simplified lumped-parameter model of systemic circulation .
Image foreground generation
The functional model was personalized to physiological and pathological conditions of interest for the generation of the image foreground, i.e. to simulate ground-truth cardiac function. In a first step, a synthetic anatomy was sampled from the corresponding cluster (e.g. NOR, DCM, HCM). Then, material properties, fiber orientations and maximum active stress were selected to describe the target cardiac function. The LV micro-structure was defined using linear transmural laws as in .
Reference healthy values for the passive tissue response and potential propagation velocities were taken from , while values for DCM and HCM cases were obtained by defining the material coefficients between 5 and 10 times larger than in the normal case .
Anatomical details can be further modified by adding localized anatomical defects to any of the geometries thanks to the physiological parametrization associated with the shapes. The corresponding variations of wall thickness, mechanical and electrophysiological parameters were automatically adjusted, gradually transitioning from healthy to diseased tissue (see Additional file 1). In this work an elliptical scar at the free wall was considered, but any approach could be adopted here.
Image background generation
The shape and functional models described in the previous sections were used to generate time-resolved 3D LV meshes that were voxelised and sliced to produce the corresponding LV tissue masks. These were then augmented by including tissue labels for the right ventricle (RV), atria and other organs using the XCAT software .
Each two-dimensional (2D) slice generated with the XCAT phantom was warped such that the LV epicardial contours from XCAT matched those of the epicardium from the masks generated by LV deformations. The surrounding tissue was deformed accordingly by smooth interpolation. The approach can also account for breathing motion from XCAT. Details are presented in the Additional file 1. This coupling approach does not require modifications to the XCAT code (essentially, a self-contained additional step is added between anatomy and image generation) and, hence, keeps all functionalities of the software.
Tissue properties definition
A neural network was used to assign textured tissue properties to the many-tissue maps combining foreground and background. A dataset of paired many-class segmentation masks and corresponding tissue-property images (i.e. images where PD, T1, and T2 values are known for every pixel) is required to train such a network. To our knowledge, there is no large dataset of tissue-property images available (even ignoring the requirement of corresponding many-class segmentation masks). Such a paired dataset was therefore synthetically generated and then used for training.
First, a CMR generative model (CMRGenNet) based on StyleGAN2 with Adaptive Discriminator Augmentation [37, 58] was trained on the ACDC dataset. Then, using the method proposed in DatasetGAN , the CMRGenNet was augmented with an additional branch to produce many-class labels for all generated images (see Fig. 3, technical details in the Additional file 1).
The CMRGenNet was used to generate a dataset of 8640 synthetic CMR images and corresponding 12-class segmentation masks, which were then utilized to train the MultiClassNet, a UNet , to predict multi-class segmentation masks from real CMR images. The MultiClassNet was then used to perform multi-class segmentation on end-systolic (ES) and end-diastolic (ED) images from the ACDC dataset.
For these segmentations, PD, T1 and T2, maximising the similarity with the corresponding image, were computed using an analytic closed-form expression of the balanced steady-state free precession (bSSFP) sequence, used to acquire the ACDC dataset (details in the Additional file 1). The same equation is used in the MRXCAT software.
This process yielded a paired dataset of 1800 parameter maps produced directly from the segmentation masks, and the corresponding detailed texture maps produced by the optimisation. As a final step, the texturizer, TextNet, was trained to map initialized uniform parameter maps to textured parameter maps. The TextNet architecture was based on a UNet.
The final images were post-processed such that any texture was removed from the LV myocardium. This is justified by the relative uniformity of image signal in the myocardium of real CMR images and the need for removing tissue property variations at the border of the myocardium resulting from partial-voluming effects. All tissue properties were then warped according to label deformations over the cardiac cycle to preserve the consistency of the anatomical details of the images.
Synthetic CMR image generation
The resulting anatomical phantoms with corresponding texturized tissue properties were used to generate cine CMR images in MRXCAT . For the use cases presented here, 2D bSSFP acquisition parameters were: repetition time TR = 3.0 ms, echo time TE = 1.5 ms, flip angle of 60°, and a signal-to-noise ratio (SNR) of 30. Eight surface coils and a Cartesian trajectory were simulated. The signal of the image was generated using the closed-form expression of the bSSFP signal equation implemented in MRXCAT, which assumes steady-state properties. As a final note, we highlight that the tissue phantom was generated at higher resolution than the target image resolution to accommodate partial voluming effects due to the limited bandwidth of the CMR encoding process.
The paired ground-truth and images data generated in this work were used as input to the DeepStrain framework [40, 41]. DeepStrain leverages a network for segmentation (CarSON) and one for cardiac motion estimation (CarMEN). The networks were trained on the ACDC datasets, which were also used in our work to define TextNet. CarSON and CarMEN predictions were used as input to an additional network that computes the corresponding LV strain. To be used in these networks, the images of this work were intensity-normalized and resampled to an isotropic in-plane resolution of 1.25 mm and a total number of 16 slices. They were then cropped around the LV mask to obtain 128 × 128 × 16 pixel images.
Left-ventricular population shape model
Figure 4 shows the SM features encoded with selected latent variables sampled at ± 3σ of the normal probability distribution defined with the VAE. These modes are associated with an identifiable physiological interpretation: global shape scaling (Fig. 4a, c, f, j), valve plane tilting (Fig. 4e), sphericity and wall thickness (Fig. 4b, d, g, h).
The average shape reconstruction error of the ACDC dataset was 6.5 ± 1.0 mm with split errors of 6.0 ± 1.0 mm, 4.5 ± 0.6 mm, and 8.0 ± 1.5 mm for healthy, DCM and HCM anatomies, respectively. A 't-SNE' map  was used to reduce the latent space vector dimension of the ACDC dataset to 2D and allow for the visualization of the anatomical clustering in Fig. 5. Clinical labels are shown in the figure as healthy (NOR, green squares), DCM (orange circles) and HCM, (grey triangles). The anatomies corresponding to the clusters shown in Fig. 5 (black diamonds) are visualized in Fig. 6. The accuracy of the classifier evaluated on these shapes was 0.86.
Cardiac functional model
The simulation of LV function with the biophysical model provided a good representation of both physiological and pathological cases: LV mass were 90 g, 140 g, 100 g and 85 g for NOR, DCM, HCM and infarcted cases, respectively. ED and ES volumes were 145 ml/70 ml, 270 ml/180 ml, 156 ml/92 ml and 159 ml/80 ml for NOR, DCM, HCM and infarcted cases, respectively; resulting in LV ejection fraction values of 51%, 34%, 41% and 49%, respectively. The corresponding shape average peak systolic radial/longitudinal/circumferential strains (er/el/ec) were: 0.78/− 0.17/− 0.14 (NOR), 0.45/− 0.10/− 0.15 (DCM), 0.50/− 0.14/− 0.13 (HCM) and 0.95 (0.30)/− 0.18 (− 0.17)/− 0.18 (0.01) remote(scar) regions of the infarcted case.
Tissue property definition
Figure 7a shows 15 synthetic images generated with CMRGenNet and a comparison between the labels produced by CMRGenNet and our manual annotations (Fig. 7b, top and bottom rows, respectively). The CMRGenNet segmentation branch produced an average Dice score of 0.91, 0.85 for RV and LV blood pools, respectively, and 0.67 and 0.82 for RV and LV myocardium, respectively, when compared to the 10 manually annotated cases used for validation.
Figure 8 compares images and multi-class labels generated by CMRGenNet (MRI and GAN lines, respectively) with the segmentations predicted by the MultiClassNet (UNet). The MultiClassNet produced an average Dice score of 0.90, 0.86, 0.65 and 0.82 of the mask prediction for the right-ventricular (RV) blood pool, LV blood pool, RV myocardium and left-ventricular myocardium, respectively, on synthetic images generated by CMRGenNet.
Figures 9, 10, 11 and 12 show short-axis (SAx) slices of images generated with MRXCAT2.0 from the corresponding simulations overlaid with ground truth physiological strain values. Figure 13 shows the steps for the generation of the synthetic images and a comparison between the CMR images obtained without and with texturization. In the non-texturized case, the default tissue properties of MRXCAT have been considered.
Results of using the DeepStrain software are summarized in Fig. 14. On average, a Dice score of 0.82 across all cardiac phases is obtained. The lowest performance was observed for the infarcted case, in particular for the thin scar region. The average displacement error computed from the synthetic data was 1.0 ± 0.9 mm. Strain predictions showed a good agreement for circumferential strain (ec average error of 0.02 ± 0.04 across all cases). Larger errors for cases with thinner walls such as DCM and infarcted cases were seen. Additionally, a general underestimation of radial strains (− 0.24 ± 0.21) was observed, with the lowest performance obtained for the infarcted case (− 0.20 ± 0.21).
The analysis of DeepStrain performance conducted by the authors in  showed segmentation accuracy with Dice scores of 0.89 and 0.91 for ED and ES cardiac phases, respectively, which agreed with our observed accuracy of 0.82 across all phases.
A methodology to extend the current XCAT torso phantoms with LV anatomical variability derived from a population-based statistical shape model has been presented. Healthy and pathological cardiac function, also including local anatomical and tissue defects, have been simulated and personalised using a biophysical cardiac electromechanical model. Tissue properties of the phantom were assigned using an image-to-image network trained to maximize the similarity of resulting images with real bSSFP CMR data. The extended phantom was then used as input to CMR image production with realistic population settings and biophysical model parameters linked to ground truth displacement and strain values. Finally, synthetic images were used to showcase their adoption in testing CMR image processing protocols.
As observed in our previous work using proper orthogonal decomposition (POD) , amplitude variations of a single latent variable from the VAE modulated specific anatomical features of the left ventricle. The contributions of the modes computed on the ACDC dataset showed distinct cluster values (Fig. 5) for (patho)physiological conditions as healthy, DCM and HCM. Note that, as the representation was learned in an unsupervised way, the ability to discriminate classes results directly from the disentangled nature of the learned representation. The corresponding cluster centers (black diamonds in Fig. 5) determined the anatomical shapes shown in Fig. 6. The DCM cluster center featured an ES configuration with enlarged blood pool volume (180 ml) and thin muscle walls, while the HCM case showed an increased wall size and a blood pool volume similar to the healthy case (92 ml and 70 ml, respectively).
The SM showed reasonable accuracy in the reconstruction of a new dataset, with an average error between points of 6 mm. While the error was higher in comparison to our previous POD approach , the VAE method determined a higher clustering accuracy (0.86) than the linear POD approach (0.78). Additionally, the normal probability distribution associated with the latent space defined by the VAE allowed for sampling of realistic synthetic geometries spanning the population variability (from the MMWHS dataset) that could be used as input for biophysical models. The use of a shape model improved the implementation proposed by  in two ways: it allowed for fast (a few seconds) generation of anatomies and it included the possibility of representing pathological cases. While our approach works on a discrete image and not on the parametrized anatomy as the approach proposed by , it is more versatile and can be coupled with any available numerical phantom.
The selection of appropriate physiological model parameters for each of the simulations allowed to obtain realistic LV cardiac functions (Figs. 9, 10, 11, 12) and ejection fractions (EF) consistent with clinical findings. The simulated healthy cardiac function (Fig. 9) had an EF of 51% and strains within the physiological range . The DCM simulation (Fig. 10) resulted in reduced strains as compared to the healthy heart and an overall reduced deformation over the cycle and, consequently, a lower EF of 34%. Also, the HCM simulation resulted in a reduced EF (41%) and lower radial strains as compared to the healthy case (Fig. 11).
The infarcted LV simulation showed a preserved EF of 49%, with high strains of the remote tissue to compensate for the reduced mobility of the scar region (Fig. 12). Circumferential and longitudinal strains of the scar region also showed the typical bulging out of the muscle at the initial phases of systole, which was related to the reduced wall thickness and the rapid pressure increase in the ventricle cavity.
Images and multi-class maps produced by CMRGenNet (Fig. 7) demonstrated reasonable realism and accuracy in the predicted multi-tissue masks. Segmentation labels from the MultiClassNet, which were used as input for training the texturizer, were in good agreement with those generated from the segmentation branch of the CMRGenNet. The comparison of the CMR images in Fig. 13 demonstrated a substantial increase in realism in image appearance when using texturized tissue properties with respect to uniform fields for each mask. Also, contrarily to the previous approaches based on style transfer or image warping methods , our approach augmented phantom masks with tissue properties that can be used for the generation of CMR images with arbitrary sequences and parameters, allowing for higher versatility. In fact, while style transfer methods can generate very realistic images, there is no control of CMR sequence parameters, SNR, image resolution and artifacts.
In our investigation, the accuracy of the DeepStrain method in tracking LV wall displacements was 1.0 ± 0.9 mm. These errors were slightly lower than those reported in the original study (2.89 ± 1.52 mm and 1.8 ± 0.2 mm for in-vivo and synthetic images, respectively ). The higher accuracy for our synthetic images was mostly related to the high SNR values used in our simulations, and to the small dataset considered. In terms of segmentation performance, the lowest Dice score was observed for infarcted areas, where a thin myocardial wall was prescribed. Dice scores values very close to those reported in the DeepStrain publication for normal, HCM and DCM conditions.
Finally, a good agreement for circumferential strain predictions was found, but a significant underestimation of the radial component was observed. However, peak ES radial strains inferred in our analysis were in the range of those reported in  (Fig. 7a in the reference) for healthy conditions (in the range of 25%). Since for healthy conditions with EFs around 50%, radial strains between 40 and 60% are expected, we argue that radial strain underestimation could be a limitation in the DeepStrain approach. Accordingly, we believe that our synthetic images are valid.
The shape model, representing anatomical variability, was trained on a limited dataset. Although we observed that the resulting low-dimensional VAE representation was able to capture variations in a dataset different from the training one, we believe that higher reconstruction and training accuracy could be obtained by training the VAE on a larger clinical dataset. Additionally, while a simple scar model has been implemented, the approach could be extended to arbitrary shapes and property variations.
The biophysical model also simplified some aspects of cardiac function: the electro-physiological model was a reasonable approximation in absence of electrical pathology, but should be extended to more complex representations to account, for example, for the effect of fibrillation. Also, the circulation model did not account for the pulmonary path of the closed-loop response of the circulatory system. We also defined the LV microstructure using linear approaches, but 3D personalised representations could easily be implemented in the model .
The biophysical simulation was time-consuming and was responsible for more than 80% of the total computational cost. While in this work we have considered a conventional biophysical model, new approaches as the one we propose in  could be used to speed up the computations and reduce the computational time to a few minutes.
Given the paired dataset of simple and detailed texture maps produced using the proposed multi-step pipeline of CMRGenNet, MultiClassNet, and per-pixel tissue parameter optimisation, there is the potential to use a more powerful image-to-image model (e.g. ) to learn TextNet, which should allow for sharper and more realistic texturing. We also noted that the MultiClassNet did not show state-of-the-art performance in the segmentation of the LV myocardium. This aspect, however, did not represent a limitation in the method, since the multi-class masks were just used to train the texturizer and that, in the synthetic images, LV masks were defined from the biophysical model and tissue properties were manually assigned based on literature values.
We only focused on mid-ventricular 2D short-axis CMR images since the warping approach we implemented was restricted to 2D problems. Therefore, basal or apical slices could not be appropriately tracked. Further work is warranted to enable full heart simulation that could be coupled with the cardiac masks in XCAT and warped with a three-dimensional approach.
CMR images were generated from texturized tissue properties inferred from a dataset of realistic images using a simple signal model . While this approach oversimplified the physical aspects related to the generation of the signal, it showed that realistic synthetic images paired with full knowledge of the biophysical ground truth could be generated. In particular, we showed that the realism gap in simulated CMR images can be significantly reduced through the use of textured phantoms. Also, we believe that results could be further improved by using a more modern GAN-based approach, such as in  or .
We successfully generated paired CMR image and ground truth data of LV function using a statistical shape model coupled with a biophysical solver. Both healthy and pathological conditions, including infarcted, DCM and HCM, could be simulated. Therefore, this approach can be employed to generate representative image population datasets with associated ground truth values for the performance assessment of image acquisition, reconstruction and processing methods in CMR.
Availability of data and materials
The code for coupling the biomechanical simulation with the tissue phantom and the texturizer will be available upon acceptance at https://gitlab.ethz.ch/ibt-cmr-public/mrxcat-2.0 under MIT license conditions. It will include the MRXCAT2.0 code and an example case. Instructions for the download of the XCAT software will also be provided.
Automated Cardiac Diagnosis Challenge
Adaptive Discriminator Augmentation
Balanced steady state free precession
Cardiac motion estimation
Cardiovascular magnetic resonance
CMR generative model
Multimodal whole heart segmentation
Wissmann L, Santelli C, Segars WP, Kozerke S. Realistic numerical phantoms for cardiovascular magnetic resonance. J Cardiovasc Magn Reson. 2014;16(1):1–11. https://doi.org/10.1186/s12968-014-0063-3.
Zubal I, Harrell C, Smith E, Rattner Z, Gindi G, Hoffer PB. Computerized three-dimensional segmented human anatomy. Med Phys. 1994;21:299–302. https://doi.org/10.1118/1.597290.
Kramer R, Khoury H, Vieira J, Loureiro E, Lima V, Lima F, Hoff G. All about FAX: a female adult voxel phantom for Monte Carlo calculation in radiation protection dosimetry. Phys Med Biol. 2004;49(23):5203–16. https://doi.org/10.1088/0031-9155/49/23/001.
Segars W, Tsui B. The evolution of 4-D computerized phantoms for imaging research: computer models that take account of body movements promise to provide evaluation and improvement of medical imaging devices and technology. Proc IEEE Inst Electr Electron Eng. 2009;97(12):1954–68. https://doi.org/10.1109/JPROC.2009.2022417.
Pretorius PH, King MA, Tsui BM, LaCroix KJ, Xia W. A mathematical model of motion of the heart for use in generating source and attenuation maps for simulating emission imaging. Med Phys. 2004;26(11):2323–32. https://doi.org/10.1118/1.598746.
Segars W, Sturgeon S, Mendonca S, Grimes J, Tsui B. 4D XCAT phantom for multimodality imaging research. Med Phys. 2010;37(9):4902–15. https://doi.org/10.1118/1.3480985.
Veress AI, Segars WP, Weiss JA, Tsui BMW, Gullberg GT. Normal and pathological NCAT image and phantom data based on physiologically realistic left ventricle finite-element models. IEEE Trans Med Imaging. 2006;25(12):1604–16. https://doi.org/10.1109/TMI.2006.884213.
Segars WP, Veress AI, Sturgeon GM, Samei E. Incorporation of the living heart model into the 4-D XCAT phantom for cardiac imaging research. IEEE Trans Radiat Plasma Med Sci. 2019;3(1):54–60. https://doi.org/10.1109/TRPMS.2018.2823060.
Duan J, Bello G, Schlemper J, Bai W, Dawes TJW, Biffi C, de Marvao A, Doumoud G, O’Regan DP, Rueckert D. Automatic 3D bi-ventricular segmentation of cardiac images by a shape-refined multi-task deep learning approach. IEEE Trans Med Imaging. 2019;38(9):2151–64. https://doi.org/10.1109/TMI.2019.2894322.
Bai W, Shi W, de Marvao A, Dawes TJW, O’Regan DP, Cook SA, Rueckert D. A bi-ventricular cardiac atlas built from 1000 + high resolution MR images of healthy subjects and an analysis of shape and motion. Med Image Anal. 2015;26(1):133–45. https://doi.org/10.1016/j.media.2015.08.009.
Suinesiaputra A, Ablin P, Alba X, Alessandrini M, Allen J, Bai W, Cimen S, Claes P, Cowan BR, Dhooge J, Duchateau N, Ehrhardt J, Frangi AF, Gooya A, Grau V, Lekadir K, Lu A, Mukhopadhyay A, Oksuz I, Parajali N, Pennec X, Pereanez M, Pinto C, Piras P, Rohe M, Rueckert D, Saring D, Sermesant M, Siddiqi K, Tabassian M, Teresi L, Tsaftaris S, Wilms M, Young A, Zhang XA. Statistical shape modeling of the left ventricle: myocardial infarct classification challenge. IEEE J Biomed Health Informat. 2017;22(2):503–15. https://doi.org/10.1109/JBHI.2017.2652449.
Joyce T, Buoso S, Stoeck CT, Kozerke S. Rapid inference of personalised left-ventricular meshes by deformation-based differentiable mesh voxelization. Med Image Anal. 2022;79:102445. https://doi.org/10.1016/j.media.2022.102445.
Buoso S, Joyce T, Kozerke S. Personalising left-ventricular biophysical models of the heart using parametric physics-informed neural networks. Med Image Anal. 2021;71:102066. https://doi.org/10.1016/j.media.2021.102066.
Benoit-Cattin H, Collewet G, Belaroussi B, Saint-Jalmes H, Odet C. The SIMRI project: a versatile and interactive MRI simulator. J Magn Reson. 2005;173(1):97–115. https://doi.org/10.1016/j.jmr.2004.09.027.
StÅNocker T, Vahedipour K, Pflugfelder D, Shah NJ. High-performance computing MRI simulations. Magn Reson Med. 2010;64(1):186–93. https://doi.org/10.1002/mrm.22406.
Xanthis CG, Filos D, Haris K, Aletras AH. Simulator-generated training datasets as an alternative to using patient data for machine learning: an example in myocardial segmentation with MRI. Comput Methods Progr Biomed. 2021;198:105817. https://doi.org/10.1016/j.cmpb.2020.105817.
Liu F, Velikina JV, Block WF, Kijowski R, Samsonov AA. Fast realistic MRI simulations based on generalized multi-pool exchange tissue model. IEEE Trans Med Imaging. 2017;36(2):527–37. https://doi.org/10.1109/TMI.2016.2620961.
Balac S, Chupin L. Fast approximate solution of bloch equation for simulation of rf artifacts in magnetic resonance imaging. Math Comput Model. 2008;48(11):1901–13. https://doi.org/10.1016/j.mcm.2007.05.021.
Tobon-Gomez C, Sukno FM, Bijnens BH, Huguet M, Frangi AF. Realistic simulation of cardiac magnetic resonance studies modeling anatomical variability, trabeculae, and papillary muscles. Magn Reson Med. 2011;65(1):280–8. https://doi.org/10.1002/mrm.22621.
Al Khalil Y, Amirrajab S, Lorenz C, Weese J, Breeuwer M. Heterogeneous virtual population of simulated CMR images for improving the generalization of cardiac segmentation algorithms. In: International Workshop on Simulation and Synthesis in Medical Imaging. 2020; pp. 68–79.
Loecher M, Perotti LE, Ennis DB. Using synthetic data generation to train a cardiac motion tag tracking neural network. Med Image Anal. 2021;74:102223. https://doi.org/10.1016/j.media.2021.102223.
Gheorghita BA, Itu LM, Sharma P, Suciu C, Wetzl J, Geppert C, Ali MAA, Lee AM, Piechnik SK, Neubauer S, Petersen SE, Schulz-Menger J, Chitiboi T. Improving robustness of automatic cardiac function quantification from cine magnetic resonance imaging using synthetic image data. Sci Reports. 2022;12:2391. https://doi.org/10.1038/s41598-022-06315-3.
Joyce T, Kozerke S. 3D medical image synthesis by factorized representation and deformable model learning. In: International Workshop on Simulation and Synthesis in Medical Imaging. Cham: Springer; 2019. p. 110–9. https://doi.org/10.1007/978-3-030-32778-1.
Abbasi-Sureshjani S, Amirrajab S, Lorenz C, Weese J, Pluim J, Breeuwer M. 4d semantic cardiac magnetic resonance image synthesis on xcat anatomical model. arXiv. 2020. https://doi.org/10.48550/ARXIV.2002.07089.
Amirrajab S, Abbasi-Sureshjani S, Khalil YA, Lorenz C, Weese J, Pluim J, Breeuwer M. XCAT-GAN for synthesizing 3D consistent labeled cardiac MR images on anatomically variable XCAT phantoms. arXiv. 2020. https://doi.org/10.48550/ARXIV.2007.13408.
Chartsias A, Joyce T, Dharmakumar R, Tsaftaris SA. Adversarial image synthesis for unpaired multi-modal cardiac data. In: International workshop on simulation and synthesis in medical imaging. Cham: Springer; 2017. p. 3–13. https://doi.org/10.1007/978-3-319-68127-6.
Chartsias A, Joyce T, Papanastasiou G, Semple S, Williams M, Newby DE, Dharmakumar R, Tsaftaris SA. Disentangled representation learning in cardiac image analysis. Med Image Anal. 2019;58:101535. https://doi.org/10.1016/j.media.2019.101535.
Duchateau N, Sermesant M, Delingette H, Ayache N. Model-based generation of large databases of cardiac images: synthesis of pathological cine MR sequences from real healthy cases. IEEE Trans Med Imaging. 2017;37(3):755–66. https://doi.org/10.1109/TMI.2017.2714343.
Al Khalil Y, Amirrajab S, Lorenz C, Weese J, Breeuwer M. Heterogeneous virtual population of simulated CMR images for improving the generalization of cardiac segmentation algorithms. In: International workshop on simulation and synthesis in medical imaging. Cham: Springer; 2020. p. 68–79. https://doi.org/10.1007/978-3-030-59520-3.
Amirrajab S, Al Khalil Y, Lorenz C, Weese J, Pluim J, Breeuwer M. Label-informed cardiac magnetic resonance image synthesis through conditional generative adversarial networks. Comput Med Imaging Graphics. 2022;101:102123. https://doi.org/10.1016/j.compmedimag.2022.102123.
Gilbert A, Marciniak M, Rodero C, Lamata P, Samset E, McLeod K. Generating synthetic labeled data from existing anatomical models: an example with echocardiography segmentation. IEEE Trans Med Imaging. 2021. https://doi.org/10.1109/TMI.2021.3051806.
Campello VM, Martin-Isla C, Izquierdo C, Petersen SE, Ballester MAG, Lekadir K. Combining multi-sequence and synthetic images for improved segmentation of late gadolinium enhancement cardiac MRI. In: International Workshop on Statistical Atlases and Computational Models of the Heart. Cham: Springer; 2019. p. 290–9. https://doi.org/10.1007/978-3-030-39074-7_31.
Yu B, Wang Y, Wang L, Shen D, Zhou L. Medical image synthesis via deep learning. In: Deep learning in medical image analysis. Cham: Springer; 2020. p. 23–44. https://doi.org/10.1007/978-3-030-33128-3.
Chartsias A, Joyce T, Papanastasiou G, Semple S, Williams M, Newby D, Dharmakumar R, Tsaftaris S. Factorised spatial representation learning: Application in semi-supervised myocardial segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). Cham: Springer; 2018. p. 490–8.
Park T, Liu MY, Wang TC, Zhu JY. Semantic image synthesis with spatially-adaptive normalization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Springer, Cham. 2019; pp. 2337–2346.
Zhang Y, Ling H, Gao J, Yin K, Lafleche J-F, Barriuso A, Torralba A, Fidler S. DatasetGAN: efficient labeled data factory with minimal human effort. arXiv. 2021. https://doi.org/10.48550/ARXIV.2104.06490.
Karras T, Aittala M, Hellsten J, Laine S, Lehtinen J, Aila T. Training generative adversarial networks with limited data. arXiv. 2020. https://doi.org/10.48550/ARXIV.2006.06676.
Li D, Yang J, Kreis K, Torralba A, Fidler S. Semantic segmentation with generative models: semi-supervised learning and strong out-of-domain generalization. arXiv. 2021. https://doi.org/10.48550/ARXIV.2104.05833.
Abadi E, Segars WP, Sturgeon GM, Harrawood B, Kapadia A, Samei E. Modeling “textured” bones in virtual human phantoms. IEEE Trans Rad Plasma Med Sci. 2018;3(1):47–53. https://doi.org/10.1109/TRPMS.2018.2828083.
Morales MA, Izquierdo-Garcia D, Aganj I, Kalpathy-Cramer J, Rosen BR, Catana C. Implementation and validation of a three-dimensional cardiac motion estimation network. Radiol Artif Intell. 2019;1(4):e180080. https://doi.org/10.1148/ryai.2019180080.
Morales MA, van den Boomen M, Nguyen C, Kalpathy-Cramer J, Rosen BR, Stultz CM, Izquierdo-Garcia D, Catana C. Deepstrain: a deep learning workflow for the automated characterization of cardiac mechanics. Front Cardiovasc Med. 2021; 8. https://doi.org/10.3389/fcvm.2021.730316.
Zhuang X, Shen J. Multi-scale patch and multi-modality atlases for whole heart segmentation of MRI. Med Image Anal. 2016;31:77–87. https://doi.org/10.1016/j.media.2016.02.006.
Zhuang X. Challenges and methodologies of fully automatic whole heart segmentation: a review. J Healthc Eng. 2013;4(3):371–407. https://doi.org/10.1260/2040-22220.127.116.111.
Zhuang X, Rhode KS, Razavi RS, Hawkes DJ, Ourselin S. A registration-based propagation framework for automatic whole heart segmentation of cardiac MRI. IEEE Trans Med Imaging. 2010;29(9):1612–25. https://doi.org/10.1109/TMI.2010.2047112.
Kingma DP, Welling M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114. 2013. https://doi.org/10.48550/arXiv.1312.611.
Bernard O, Lalande A, Zotti C, Cervenansky F, Yang X, Heng P-A, Cetin I, Lekadir K, Camara O, Ballester MAG, et al. Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: is the problem solved? IEEE Trans Med Imaging. 2018;37(11):2514–25. https://doi.org/10.1109/TMI.2018.2837502.
Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E. Scikit-learn: machine learning in Python. J Mach Learn Res. 2011;12:2825–30. https://doi.org/10.5555/1953048.2078195.
Buoso S, Palacios R. Electro-aeromechanical modelling of integrally actuated membrane wings. J Fluids Struct. 2015;58:188–202. https://doi.org/10.1016/j.jfluidstructs.2015.08.010.
Buoso S, Palacios R. Viscoelastic effects in the aeromechanics of actuated elastomeric membrane wings. J Fluids Struct. 2016;64:40–56. https://doi.org/10.1016/j.jfluidstructs.2016.01.003.
Buoso S, Palacios R. On-demand aerodynamics of integrally actuated membranes with feedback control. AIAA J. 2017;55(2):377–88. https://doi.org/10.2514/1.J054888.
Buoso S, Dickinson B, Palacios R. Bat-inspired integrally actuated membrane wings with leading edge sensing. Bioinspira Biomimet. 2018;16(1):016013. https://doi.org/10.1088/1748-3190/aa9a7b.
Holzapfel GA, Ogden RW. Constitutive modelling of passive myocardium: a structurally based framework for material characterization. Phil Trans R Soc A Math Phys Eng Sci. 2009;367(1902):3445–75. https://doi.org/10.1098/rsta.2009.0091.
Neic A, Campos FO, Prassl AJ, Niederer SA, Bishop MJ, Vigmond EJ, Plank G. Efficient computation of electrograms and ECGs in human whole heart simulations using a reaction-eikonal model. J Comput Phys. 2017;346:191–211. https://doi.org/10.1016/j.jcp.2017.06.020.
Augustin CM, Gsell MAF, Karabelas E, Willemen E, Prinzen F, Lumens J, Vigmond EJ, Plank G. Validation of a 3D-0D closed-loop model of the heart and circulation—modeling the experimental assessment of diastolic and systolic ventricular properties. 2009.08802.
Niederer SA, Plank G, Chinchapatnam P, Ginks M, Lamata P, Rhode KS, Rinaldi CA, Razavi R, Smith NP. Length-dependent tension in the failing heart and the efficacy of cardiac resynchronization therapy. Cardiovasc Res. 2011;89(2):336–43. https://doi.org/10.1093/cvr/cvq318.
Peirlinck M, Sack KL, De Backer P, Morais P, Segers P, Franz T, De Beule M. Kinematic boundary conditions substantially impact in silico ventricular function. Int J Numer Methods Biomed Eng. 2019;35(1):3151. https://doi.org/10.1002/cnm.3151.
Miller R, Kerfoot E, Mauger C, Ismail TF, Young AA, Nordsletten DA. An implementation of patient-specific biventricular mechanics simulations with a deep learning and computational pipeline. Front Physiol. 2021;12:1398. https://doi.org/10.3389/fphys.2021.716597.
Karras T, Laine S, Aittala M, Hellsten J, Lehtinen J, Aila T. Analyzing and Improving the Image Quality of StyleGAN. arXiv. 2019. https://doi.org/10.48550/ARXIV.1912.04958.
Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image segmentation. arXiv. 2015. https://doi.org/10.48550/ARXIV.1505.04597.
Van der Maaten L, Hinton G. Visualizing data using t-SNE. J Mach Learn Res. 2008; 9(11).
Maceira A, Prasad S, Khan M, Pennell D. Normalized left ventricular systolic and diastolic function by steady state free precession cardiovascular magnetic resonance. J Cardiovasc Magn Reson. 2006;8(3):417–26. https://doi.org/10.1080/10976640600572889.
Alessandrini M, De Craene M, Bernard O, Giffard-Roisin S, Allain P, Waechter-Stehle I, Weese J, Saloux E, Delingette H, Sermesant M, D’hooge J. A pipeline for the generation of realistic 3D synthetic echocardiographic sequences: methodology and open-access database. IEEE Trans Med Imaging. 2015;34(7):1436–51. https://doi.org/10.1109/TMI.2015.2396632.
Stimm J, Buoso S, Berberoglu E, Kozerke S, Genet M, Stoeck CT. A 3D personalized cardiac myocyte aggregate orientation model using MRI data-driven low-rank basis functions. Med Image Anal. 2021;71:102064. https://doi.org/10.1016/j.media.2021.102064.
Isola P, Zhu J-Y, Zhou T, Efros AA. Image-to-image translation with conditional adversarial networks. arXiv. 2016. https://doi.org/10.48550/ARXIV.1611.07004.
This work is supported by the Swiss National Science Foundation (SNF) [Grants CR23I3-166485, 325230-197702], and by PHRT SWISSHEART Failure Network of the ETH Domain.
Open access funding provided by Swiss Federal Institute of Technology Zurich. Swiss National Science Foundation (SNF) [Grants CR23I3-166485, 325230-197702], PHRT SWISSHEART Failure Network of the ETH Domain.
Ethics approval and consent to participate
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Buoso, S., Joyce, T., Schulthess, N. et al. MRXCAT2.0: Synthesis of realistic numerical phantoms by combining left-ventricular shape learning, biophysical simulations and tissue texture generation. J Cardiovasc Magn Reson 25, 25 (2023). https://doi.org/10.1186/s12968-023-00934-z
- Cardiac magnetic resonance
- Biomechanical modeling
- Image synthesis
- Population variability