Purpose: Image registration is the process of aligning images, and it is a fundamental task in medical image analysis. While many tasks in the field of image analysis, such as image segmentation, are handled almost entirely with deep learning and exceed the accuracy of conventional algorithms, currently available deformable image registration methods are often still conventional. Deep learning methods for medical image registration have recently reached the accuracy of conventional algorithms. However, they are often based on a weakly supervised learning scheme using multilabel image segmentations during training. The creation of such detailed annotations is very time-consuming.

Approach: We propose a weakly supervised learning scheme for deformable image registration. By calculating the loss function based on only bounding box labels, we are able to train an image registration network for large displacement deformations without using densely labeled images. We evaluate our model on interpatient three-dimensional abdominal CT and MRI images.

Results: The results show an improvement of ∼10  %   (for CT images) and 20% (for MRI images) in comparison to the unsupervised method. When taking into account the reduced annotation effort, the performance also exceeds the performance of weakly supervised training using detailed image segmentations.

Conclusion: We show that the performance of image registration methods can be enhanced with little annotation effort using our proposed method.

Purpose: In stable coronary artery disease (CAD), reduction in mortality and/or myocardial infarction with revascularization over medical therapy has not been reliably achieved. Coronary arteries are usually extracted to perform stenosis detection. As such, developing accurate segmentation of vascular structures and quantification of coronary arterial stenosis in invasive coronary angiograms (ICA) is necessary.

Approach: A multi-input and multiscale (MIMS) U-Net with a two-stage recurrent training strategy was proposed for the automatic vessel segmentation. The proposed model generated a refined prediction map with the following two training stages: (i) stage I coarsely segmented the major coronary arteries from preprocessed single-channel ICAs and generated the probability map of arteries; and (ii) during the stage II, a three-channel image consisting of the original preprocessed image, a generated probability map, and an edge-enhanced image generated from the preprocessed image was fed to the proposed MIMS U-Net to produce the final segmentation result. After segmentation, an arterial stenosis detection algorithm was developed to extract vascular centerlines and calculate arterial diameters to evaluate stenotic level.

Results: Experimental results demonstrated that the proposed method achieved an average Dice similarity coefficient of 0.8329, an average sensitivity of 0.8281, and an average specificity of 0.9979 in our dataset with 294 ICAs obtained from 73 patients. Moreover, our stenosis detection algorithm achieved a true positive rate of 0.6668 and a positive predictive value of 0.7043.

Conclusions: Our proposed approach has great promise for clinical use and could help physicians improve diagnosis and therapeutic decisions for CAD.

Purpose: We propose a one-step tissue characterization method for spectral photon-counting computed tomography (SPCCT) using eigentissue decomposition (ETD), tailored for highly accurate human tissue characterization in radiotherapy.

Methods: The approach combines a Poisson likelihood, a spatial prior, and a quantitative prior constraining eigentissue fractions based on expected values for tabulated tissues. There are two regularization parameters: α for the quantitative prior, and β for the spatial prior. The approach is validated in a realistic simulation environment for SPCCT. The impact of α and β is evaluated on a virtual phantom. The framework is tested on a virtual patient and compared with two sinogram-based two-step methods [using respectively filtered backprojection (FBP) and an iterative method for the second step] and a post-reconstruction approach with the same quantitative prior. All methods use ETD.

Results: Optimal performance with respect to bias or RMSE is achieved with different combinations of α and β on the cylindrical phantom. Evaluated in tissues of the virtual patient, the one-step framework outperforms two-step and post-reconstruction approaches to quantify proton-stopping power (SPR). The mean absolute bias on the SPR is 0.6% (two-step FBP), 0.6% (two-step iterative), 0.6% (post-reconstruction), and 0.2% (one-step optimized for low bias). Following the same order, the RMSE on the SPR is 13.3%, 2.5%, 3.2%, and 1.5%.

Conclusions: Accurate and precise characterization with ETD can be achieved with noisy SPCCT data without the need to rely on post-reconstruction methods. The one-step framework is more accurate and precise than two-step methods for human tissue characterization.

Purpose: Industrial microcomputed tomography (micro-CT) scanners are suitable for nondestructive testing (NDT) of metal, 3D-printed medical components. Typically, these scanners are equipped with high-energy sources that require heavy shielding and costly infrastructure to operate safely, making routine NDT of medical components prohibitively expensive. Alternatively, fixed-current, low-cost x-ray units could be implemented to perform CT-based NDT of 3D-printed medical parts in a subset of cases, if there is sufficient x-ray transmission for the CT reconstruction. A lack of signal—caused by areas of high attenuation in two-dimensional-projection images of metal objects—leads to artifacts that can make an image-based NDT unreliable. We present the implementation of a dual-exposure technique devised to extend the dynamic range (DR) of a commercially available CT scanner equipped with a low-cost low-energy (80 kV) x-ray unit, increasing the signal-to-noise ratio of highly attenuated areas for NDT of 3D-printed medical components.

Approach: Our high-dynamic-range CT (HDR-CT) technique adequately combines projection images acquired at two exposure levels by modifying the integration times of each protocol. We evaluate the performance and limitations of this HDR-CT technique by imaging a series of titanium-alloy test-samples. One of the test-samples was a resolution and conspicuity phantom designed to assess the improvements in void visualization of the proposed methodology. The other test-samples were four porous cylinders, 17 × 40 mm, with 60%, 70%, 80%, and 90% nominal internal porosities.

Results: Our HDR-CT technique adequately combines projection images acquired at two exposure levels by modifying the integration times of each protocol. Our results demonstrate that the 12-bit native DR of the CT scanner was increased to effective values of between 14 and 16 bits.

Conclusions: The HDR-CT reconstructions showed improved contrast-to-noise and void conspicuity, when compared with conventional CT scans. This extension of DR has the potential to improve defect visualization during NDT of medium-size, titanium-alloy, 3D-printed medical components.

Purpose: Radiomics of magnetic resonance images (MRIs) in rectal cancer can non-invasively characterize tumor heterogeneity with potential to discover new imaging biomarkers. However, for radiomics to be reliable, the imaging features measured must be stable and reproducible. The aim of this study is to quantify the repeatability and reproducibility of MRI-based radiomic features in rectal cancer.

Approach: An MRI radiomics phantom was used to measure the longitudinal repeatability of radiomic features and the impact of post-processing changes related to image resolution and noise. Repeatability measurements in rectal cancers were also quantified in a cohort of 10 patients with test–retest imaging among two observers.

Results: We found that many radiomic features, particularly from texture classes, were highly sensitive to changes in image resolution and noise. About 49% of features had coefficient of variations ≤10 % in longitudinal phantom measurements. About 75% of radiomic features in in vivo test–retest measurements had an intraclass correlation coefficient of ≥0.8. We saw excellent interobserver agreement with mean Dice similarity coefficient of 0.95 ± 0.04 for test and retest scans.

Conclusions: The results of this study show that even when using a consistent imaging protocol many radiomic features were unstable. Therefore, caution must be taken when selecting features for potential imaging biomarkers.

Purpose: Modern medical imaging enables clinicians to effectively diagnose, monitor, and treat diseases. However, clinical decision-making often relies on combined evaluation of either longitudinal or disparate image sets, necessitating coregistration of multiple acquisitions. Promising coregistration techniques have been proposed; however, available methods predominantly rely on time-consuming manual alignments or nontrivial feature extraction with limited clinical applicability. Addressing these issues, we present a fully automated, robust, nonrigid registration method, allowing for coregistering of multimodal tomographic vascular image datasets using luminal annotation as the sole alignment feature.

Approach: Registration is carried out by the use of the registration metrics defined exclusively for lumens shapes. The framework is primarily broken down into two sequential parts: longitudinal and rotational registration. Both techniques are inherently nonrigid in nature to compensate for motion and acquisition artifacts in tomographic images.

Results: Performance was evaluated across multimodal intravascular datasets, as well as in longitudinal cases assessing pre-/postinterventional coronary images. Low registration error in both datasets highlights method utility, with longitudinal registration errors—evaluated throughout the paired tomographic sequences—of 0.29 ± 0.14 mm (<2 longitudinal image frames) and 0.18 ± 0.16 mm (<1 frame) for multimodal and interventional datasets, respectively. Angular registration for the interventional dataset rendered errors of 7.7 ° ± 6.7 ° , and 29.1 ° ± 23.2 ° for the multimodal set.

Conclusions: Satisfactory results across datasets, along with additional attributes such as the ability to avoid longitudinal over-fitting and correct nonlinear catheter rotation during nonrigid rotational registration, highlight the potential wide-ranging applicability of our presented coregistration method.

Purpose: Retinopathy screening via digital imaging is promising for early detection and timely treatment, and tracking retinopathic abnormality over time can help to reveal the risk of disease progression. We developed an innovative physician-oriented artificial intelligence-facilitating diagnosis aid system for retinal diseases for screening multiple retinopathies and monitoring the regions of potential abnormality over time.

Approach: Our dataset contains 4908 fundus images from 304 eyes with image-level annotations, including diabetic retinopathy, age-related macular degeneration, cellophane maculopathy, pathological myopia, and healthy control (HC). The screening model utilized a VGG-based feature extractor and multiple-binary convolutional neural network-based classifiers. Images in time series were aligned via affine transforms estimated through speeded-up robust features. Heatmaps of retinopathy were generated from the feature extractor using gradient-weighted class activation mapping++, and individual candidate retinopathy sites were identified from the heatmaps using clustering algorithm. Nested cross-validation with a train-to-test split of 80% to 20% was used to evaluate the performance of the screening model.

Results: Our screening model achieved 99% accuracy, 93% sensitivity, and 97% specificity in discriminating between patients with retinopathy and HCs. For discriminating between types of retinopathy, our model achieved an averaged performance of 80% accuracy, 78% sensitivity, 94% specificity, 79% F1-score, and Cohen’s kappa coefficient of 0.70. Moreover, visualization results were also shown to provide reasonable candidate sites of retinopathy.

Conclusions: Our results demonstrated the capability of the proposed model for extracting diagnostic information of the abnormality and lesion locations, which allows clinicians to focus on patient-centered treatment and untangles the pathological plausibility hidden in deep learning models.

Purpose: Vascular changes are observed from initial stages of breast cancer, and monitoring of vessel structures helps in early detection of malignancies. In recent years, thermal imaging is being evaluated as a low-cost imaging modality to visualize and analyze early vascularity. However, visual inspection of thermal vascularity is challenging and subjective. Therefore, there is a need for automated techniques to assist physicians in visualization and interpretation of vascularity by marking the vessel structures and by providing quantified qualitative parameters that helps in malignancy classification

Approach: In the literature, there are very few approaches for vascular analysis and classification of breast thermal images using interpretable vascular features. One major challenge is the automated detection of breast vascularity due to diffused vessel boundaries. We first propose a deep learning-based semantic segmentation approach that generates heatmaps of vessel structures from two-dimensional breast thermal images for quantitative assessment of breast vascularity. Second, we extract interpretable vascular parameters and propose a classifier to predict likelihood of breast cancer purely from the extracted vascular parameters.

Results: The results of the cancer classifier were validated using an independent clinical dataset consisting of 258 participants. The results were encouraging as the proposed approach segmented vessels well and gave a good classification performance with area under receiver operating characteristic curve of 0.85 with the proposed vascularity parameters.

Conclusions: The detected vasculature and its associated high classification performance show the utility of the proposed approach in interpretation of breast vascularity.

Purpose: Deep brain stimulation (DBS) is an interventional treatment for some neurological and neurodegenerative diseases. For example, in Parkinson’s disease, DBS electrodes are positioned at particular locations within the basal ganglia to alleviate the patient’s motor symptoms. These interventions depend greatly on a preoperative planning stage in which potential targets and electrode trajectories are identified in a preoperative MRI. Due to the small size and low contrast of targets such as the subthalamic nucleus (STN), their segmentation is a difficult task. Machine learning provides a potential avenue for development, but it has difficulty in segmenting such small structures in volumetric images due to additional problems such as segmentation class imbalance.

Approach: We present a two-stage separable learning workflow for STN segmentation consisting of a localization step that detects the STN and crops the image to a small region and a segmentation step that delineates the structure within that region. The goal of this decoupling is to improve accuracy and efficiency and to provide an intermediate representation that can be easily corrected by a clinical user. This correction capability was then studied through a human–computer interaction experiment with seven novice participants and one expert neurosurgeon.

Results: Our two-step segmentation significantly outperforms the comparative registration-based method currently used in clinic and approaches the fundamental limit on variability due to the image resolution. In addition, the human–computer interaction experiment shows that the additional interaction mechanism allowed by separating STN segmentation into two steps significantly improves the users’ ability to correct errors and further improves performance.

Conclusions: Our method shows that separable learning not only is feasible for fully automatic STN segmentation but also leads to improved interactivity that can ease its translation into clinical use.

Purpose: Deep learning (DL) applications strongly depend on the training dataset and convolutional neural network architecture; however, it is unclear how to objectively select such parameters. We investigate the classification performance of different DL models and training schemes for the anatomic classification of cone-beam computed tomography (CBCT) projections.

Approach: CBCT scans from 1055 patients were collected and manually classified into five anatomic classes and used to develop DL models to predict the anatomic class from single x-ray projections. VGG-16, Xception, and Inception v3 architectures were trained with 75% of the data, and the remaining 25% was used for testing and evaluation. To study the dependence of the classification performance on dataset size, training data was downsampled to various dataset sizes. Gradient-weighted class activation maps (grad-CAM) were generated using the model with highest classification performance, to identify regions with strong influence on CNN decisions.

Results: The highest precision and recall values were achieved with VGG-16. One of the best performing combinations was the VGG-16 trained with 90 deg projections (mean class precision = 0.87). The training dataset size could be reduced to ∼50 % of its initial size, without compromising the classification performance. For correctly classified cases, Grad-CAM were more heavily weighted for anatomically relevant regions.

Conclusions: It was possible to determine those dependencies with a higher influence on the classification performance of DL models for the studied task. Grad-CAM enabled the identification of possible sources of class confusion.

Purpose: Radiomics have become invaluable for non-invasive cancer patient risk prediction, and the community now turns to exogenous assessment, e.g., from genomics, for interpretability of these agnostic analyses. Yet, some opportunities for clinically interpretable modeling of positron emission tomography (PET) imaging data remain unexplored, that could facilitate insightful characterization at voxel level.

Approach: Here, we present a novel deformable tubular representation of the distribution of tracer uptake within a volume of interest, and derive interpretable prognostic summaries from it. This data-adaptive strategy yields a 3D-coherent and smooth model fit, and a profile curve describing tracer uptake as a function of voxel location within the volume. Local trends in uptake rates are assessed at each voxel via the calculation of gradients derived from this curve. Intratumoral heterogeneity can also be assessed directly from it.

Results: We illustrate the added value of this approach over previous strategies, in terms of volume rendering and coherence of the structural representation of the data. We further demonstrate consistency of the implementation via simulations, and prognostic potential of heterogeneity and statistical summaries of the uptake gradients derived from the model on a clinical cohort of 158 sarcoma patients imaged with F18-fluorodeoxyglucose–PET, in multivariate prognostic models of patient survival.

Conclusions: The proposed approach captures uptake characteristics consistently at any location, and yields a description of variations in uptake that holds prognostic value complementarily to structural heterogeneity. This creates opportunities for monitoring of local areas of greater interest within a tumor, e.g., to assess therapeutic response in avid locations.

Purpose: Internal fixation of pelvic fractures is a challenging task requiring the placement of instrumentation within complex three-dimensional bone corridors, typically guided by fluoroscopy. We report a system for two- and three-dimensional guidance using a drill-mounted video camera and fiducial markers with evaluation in first preclinical studies.

Approach: The system uses a camera affixed to a surgical drill and multimodality (optical and radio-opaque) markers for real-time trajectory visualization in fluoroscopy and/or CT. Improvements to a previously reported prototype include hardware components (mount, camera, and fiducials) and software (including a system for detecting marker perturbation) to address practical requirements necessary for translation to clinical studies. Phantom and cadaver experiments were performed to quantify the accuracy of video-fluoroscopy and video-CT registration, the ability to detect marker perturbation, and the conformance in placing guidewires along realistic pelvic trajectories. The performance was evaluated in terms of geometric accuracy and conformance within bone corridors.

Results: The studies demonstrated successful guidewire delivery in a cadaver, with a median entry point error of 1.00 mm (1.56 mm IQR) and median angular error of 1.94 deg (1.23 deg IQR). Such accuracy was sufficient to guide K-wire placement through five of the six trajectories investigated with a strong level of conformance within bone corridors. The sixth case demonstrated a cortical breach due to extrema in the registration error. The system was able to detect marker perturbations and alert the user to potential registration issues. Feasible workflows were identified for orthopedic-trauma scenarios involving emergent cases (with no preoperative imaging) or cases with preoperative CT.

Conclusions: A prototype system for guidewire placement was developed providing guidance that is potentially compatible with orthopedic-trauma workflow. First preclinical (cadaver) studies demonstrated accurate guidance of K-wire placement in pelvic bone corridors and the ability to automatically detect perturbations that degrade registration accuracy. The preclinical prototype demonstrated performance and utility supporting translation to clinical studies.

Purpose: The most frequently used model for simulating multireader multicase (MRMC) data that emulates confidence-of-disease ratings from diagnostic imaging studies has been the Roe and Metz (RM) model, proposed by Roe and Metz in 1997 and later generalized by Hillis (2012), Abbey et al. (2013), and Gallas and Hillis (2014). A problem with these models is that it has been difficult to set model parameters such that the simulated data are similar to MRMC data encountered in practice. To remedy this situation, Hillis (2018) mapped parameters from the RM model to Obuchowski–Rockette (OR) model parameters that describe the distribution of the empirical AUC outcomes computed from the RM model simulated data. We continue that work by providing the reverse mapping, i.e., by deriving an algorithm that expresses RM parameters as functions of the OR empirical AUC distribution parameters.

Approach: We solve for the corresponding RM parameters in terms of the OR parameters using numerical methods.

Results: An algorithm is developed that results in, at most, one solution of RM parameter values that correspond to inputted OR parameter values. The algorithm can be implemented using an R software function. Examples are provided that illustrate the use of the algorithm. A simulation study validates the algorithm.

Conclusions: The resulting algorithm makes it possible to easily determine RM model parameter values such that simulated data emulate a specific real-data study. Thus, MRMC analysis methods can be empirically tested using simulated data similar to that encountered in practice.

Purpose: Automated breast ultrasound (ABUS) presents three-dimensional (3D) representations of the breast in the form of stacks of coronal and transverse plane images. ABUS is especially useful for the assessment of dense breasts. Here, we present the first eye tracking data showing how radiologists search and evaluate ABUS cases.

Approach: Twelve readers evaluated single-breast cases in 20-min sessions. Positive findings were present in 56% of the evaluated cases. Eye position and the currently visible coronal and transverse slice were tracked, allowing for reconstruction of 3D “scanpaths.”

Results: Individual readers had consistent search strategies. Most readers had strategies that involved examination of all available images. Overall accuracy was 0.74 (sensitivity = 0.66 and specificity = 0.84). The 20 false negative errors across all readers can be classified using Kundel’s (1978) taxonomy: 17 are “decision” errors (readers found the target but misclassified it as normal or benign). There was one recognition error and two “search” errors. This is an unusually high proportion of decision errors. Readers spent essentially the same proportion of time viewing coronal and transverse images, regardless of whether the case was positive or negative, correct or incorrect. Readers tended to use a “scanner” strategy when viewing coronal images and a “driller” strategy when viewing transverse images.

Conclusions: These results suggest that ABUS errors are more likely to be errors of interpretation than of search. Further research could determine if readers’ exploration of all images is useful or if, in some negative cases, search of transverse images is redundant following a search of coronal images.

Purpose: Validation of artificial intelligence (AI) algorithms in digital pathology with a reference standard is necessary before widespread clinical use, but few examples focus on creating a reference standard based on pathologist annotations. This work assesses the results of a pilot study that collects density estimates of stromal tumor-infiltrating lymphocytes (sTILs) in breast cancer biopsy specimens. This work will inform the creation of a validation dataset for the evaluation of AI algorithms fit for a regulatory purpose.

Approach: Collaborators and crowdsourced pathologists contributed glass slides, digital images, and annotations. Here, “annotations” refer to any marks, segmentations, measurements, or labels a pathologist adds to a report, image, region of interest (ROI), or biological feature. Pathologists estimated sTILs density in 640 ROIs from hematoxylin and eosin stained slides of 64 patients via two modalities: an optical light microscope and two digital image viewing platforms.

Results: The pilot study generated 7373 sTILs density estimates from 29 pathologists. Analysis of annotations found the variability of density estimates per ROI increases with the mean; the root mean square differences were 4.46, 14.25, and 26.25 as the mean density ranged from 0% to 10%, 11% to 40%, and 41% to 100%, respectively. The pilot study informs three areas of improvement for future work: technical workflows, annotation platforms, and agreement analysis methods. Upgrades to the workflows and platforms will improve operability and increase annotation speed and consistency.

Conclusions: Exploratory data analysis demonstrates the need to develop new statistical approaches for agreement. The pilot study dataset and analysis methods are publicly available to allow community feedback. The development and results of the validation dataset will be publicly available to serve as an instructive tool that can be replicated by developers and researchers.