1.1.

OCT Image Classification

This method focuses on generating 2-D OCT image (i.e., B-scans) classifiers when OCT images and their labels are provided. Traditional machine learning is often used in OCT image classification. These methods first extract B-scan features and subsequently design classifiers.^4–7 Srinivasan et al.⁵ extracted histograms of oriented gradients features of B-scans and then classified them using a support vector machine (SVM) classifier. Sun et al.^6,7 performed feature extraction of B-scans using dictionary learning, sparse coding, and spatial pyramid matching and recognized them using an SVM classifier. Another useful technique is deep learning, especially CNN classifier models. Several works on macular OCT image classification using CNN models have been conducted.^8–13 The attention mechanism in deep learning is similar to the attention mechanism of human vision in that it focuses attention on important points among a large number of information, selecting key information and ignoring other unimportant information. In OCT image classification, attention could focus on lesion part, which usually occupies only a very small part of the OCT scan, and it has been explored and introduced for macular OCT classification applications.^14,15

1.2.

OCT Volume Classification

This method focuses on generating OCT volume (i.e., a series of B-scans) classifiers when OCT volumes and their volume-level labels are known. A voting inference strategy is often applied to OCT volume classification. For an OCT volume, the voting strategy involves initially obtaining all of its B-scan classifications, and then it yields a volume-level classification based on the results. Majority voting has been used in several studies.^5–7,12 Rasti et al.¹⁶ presented a multiscale CNN ensemble structure and suggested a voting strategy to obtain volume-level diagnosis. Qiu et al.¹⁷ proposed a B-scan classifier using a relabeling technique and suggested another voting strategy to classify OCT volume. Another important technique is true volume-level OCT data classification.^18–29 For an OCT volume, it first obtains a global feature representation of the volume and then designs classifiers to recognize it. Venhuizen et al.^20,21 obtained the global representation of an OCT volume using clustering and bag-of-word models and classified it using a random forest classifier. Fang et al.²³ extracted the global feature of an OCT volume based on the combination of principal component analysis network (PCANet)²² and composite kernels and recognized it using an extreme learning machine. Rasti et al.²⁴ obtained the global feature of an OCT volume using a wavelet-based convolutional neural network and classified it using a random forest classifier. Apostolopoulos et al.²⁵ directly tiled all of the B-scans in a volume vertically in a 2-D plane to obtain the global feature of an OCT volume and obtained its classification using a 2-D CNN classifier. De Fauw et al.²⁶ obtained a tissue-segmentation map of an OCT volume as a global feature and classified it using a deep learning architecture. Santos et al.²⁷ extracted the global feature of an OCT volume from the perspective of a C-scan using semivariogram and semimadogram functions and recognized it using an SVM. Seebock et al.²⁸ obtained the feature representation of a retinal OCT volume using deep denoising autoencoders to segment anomalous regions and employed clustering to classify it. Sun et al.²⁹ proposed multiple instances of a learning-based SVM to perform volumetric classification using features extracted from the histogram obtained from oriented gradient and principal component analysis.

In this report, we propose a method to extract the global feature of an OCT volume for OCT volume classification. Specifically, for an OCT volume, we extract its B-scan feature vectors and stack them together to generate a 2-D feature map that contains the global features of the OCT volume. The 2-D feature map is then used to recognize the OCT volume using a volume-level OCT volume classifier. We fine-tune a pretrained CNN classifier as a B-scan feature extractor based on transfer learning and train a volume-level OCT volume classifier using 2-D feature maps. Therefore, OCT volume classification is successfully implemented through two image classifiers. We design some volume-level classifiers for 2-D feature maps. In particular, we propose a classifier, i.e., convolutional neutral network with attention mechanism. To the best of our knowledge, this is the first algorithm to introduce the 2-D feature map of an OCT volume for classification.³⁰ The main contributions of this report are described as follows:

The remainder of this report is organized as follows: Sec. 2 describes our proposed methods. Experimental results on clinical OCT datasets and discussions are presented in Sec. 3, and Sec. 4 summarizes the main conclusions.

2.1.

CNN Model for B-Scan Feature Extraction

2.1.1.

Pretrained ResNet-50

The CNN model has been widely used in computer vision (especially in image classification) since the ImageNet³¹ competition in 2012. In recent years, variants of CNN architectures such as AlexNet,³² VGG,³³ GoogLeNet,³⁴ and ResNet³⁵ have been developed. In principle, all of these CNN models can be used to train our models based on transfer learning to adapt specific datasets such as an OCT image dataset. ResNet is a representative deep network. In this study, we take ResNet-50 as an example to show how to obtain a CNN model as a B-scan feature extractor. The architecture of ResNet-50 is shown in Fig. 2. The input of ResNet-50 is an image of size $224\times 224$ with RGB channels, followed by convolutional building blocks: one Conv1, three Conv2, four Conv3, six Conv4, and three Conv5, and a feature vector of size 2048 is obtained using the AvgPool module on the output of the last Conv5 block as the input of a fully connected (FC) layer. “FC 1000” stands for an FC layer with 1000 class outputs.

Fig. 2

Network architecture of ResNet-50.

The most straightforward approach for obtaining a CNN model for B-scan feature extraction is to utilize the pretrained ResNet-50 as a feature extractor. Specifically, we fix all of the pretrained weights in all of the Conv1 to Conv5 blocks and ignore the FC layer. For each B-scan of an OCT volume, we take the average pooling of the output of the Conv5 block as the B-scan feature vector. The advantage of this method is that no training of the ResNet-50 is required; therefore, it is efficient. Yet, because it does not exploit the advantages of the characteristics of OCT images, the extracted B-scan feature vector is often inadequate.

2.2.

2D Feature Map

For a given OCT volume $X_i$, we extract its B-scan feature vectors using a B-scan feature extractor and obtain its 2-D feature map by concatenating all of its B-scan feature vectors row-by-row. Suppose each B-scan feature vector is of size $L$, where $L$ is far less than the B-scan’s dimension. In particular, if we use the finetuned ResNet-50 as the B-scan feature extractor, then we have $L=2048$. Evidently, the 2-D feature map of the volume $X_i$ is a 2-D matrix of size $(|X_i|,L)$, where $|X_i|$ is the number of B-scans of $X_i$. This global feature representation possesses the following advantages:

In the public Duke dataset,³⁶ each volume consists of 100 B-scans and each B-scan is an image of size $1000\times 512$. Therefore, using its 2-D feature map representation, its dimensions could be reduced from $1000\times 100\times 512=\mathrm{51,200,000}$ to $100\times 2048=\mathrm{204,800}$.

The 2-D feature map of the OCT volume has data correlation between the rows, as illustrated in Sec. 3.5.3.

Since image classifiers could be used on 2-D feature maps, this representation transforms OCT volume classification into image classification.

2.3.

Volume-Level CNN Classifier with/without Attention Mechanism

Various classifiers could be designed to classify 2-D feature maps. Although the 2-D feature map is different from natural image and OCT B-scan image, it also has data correlation. Therefore, a CNN image classifier could be designed to deal with it. Furthermore, convolution operations extract informative features by blending cross-channel and spatial information together, while the convolutional block attention module (CBAM)³⁷ could be used to emphasize meaningful features along those two principal dimensions: channel and spatial axes. Specifically, given an intermediate feature map, CBAM sequentially infers attention maps along two separate dimensions: channel and spatial; then the attention maps are multiplied to the input feature map for adaptive feature refinement. Based on these ideas, we propose a volume-level classifier with attention mechanism, denoted by CNN_CBAM, as is shown in Fig. 3. This architecture consists of $n$ convolution blocks, a CBAM module, an FC layer and a softmax classifier, where two to the power of $n$ is less than the row number of the 2-D feature map. Each convolution block is a standard CNN module, including a 2-D convolution layer, batch normalization,³⁸ Leaky ReLU, and max pooling. Each convolution block has a $3\times 3$ kernel with four channels. In the case of $n=4$, for an input of a 2-D feature map with a size of $100\times 2048$, the outputs of the four convolution blocks are $4\times 50\times 1024$, $4\times 25\times 512$, $4\times 12\times 256$, and $4\times 6\times 128$ sequentially; then the output $4\times 6\times 128$ is refined in features by CBAM and transferred into a one-dimensional (1-D) vector of size 3072 as the input of the FC layer. We note that CNN_CBAM becomes CNN volume-level classifier when the CBAM module is ignored.

Fig. 3

CNN classifier structure.

For any OCT volume in the volume-level training dataset, we first obtain its 2-D feature map using the finetuned ResNet-50 as the feature extractor and assign its label as the same as the label of the OCT volume. Then, we train the volume-level CNN or CNN_CBAM classifier model using all of the 2-D feature maps and their labels.

For any OCT volume in the volume-level test dataset, as shown in the flowchart in Fig. 1, we extract all of its B-scan feature vectors using the finetuned ResNet-50 model as a feature extractor and stack them together to obtain its 2-D feature map. Finally, we input the 2-D feature map in the trained CNN or CNN_CBAM volume-level classifier to recognize the OCT volume.

3.1.

Experimental Environments

Our experiments were performed on a machine with an IntelCorei7-7700K 4.20 GHz CPU, 32 GB RAM, NVIDIA Titan X GPU, 12 GB RAM, and Windows 10 operating system. We use PyTorch as the deep learning framework, powered by the cuDNN Toolbox, and compiled the code in Python 3.6.

3.2.

Datasets

One test dataset is a publicly available two-class dataset released by the VIP Lab of Duke University,³⁶ in which the OCT volume data were acquired with a Bioptigen SD-OCT system. This dataset consists of 269 intermediate AMD volumes and 115 normal volumes, with each volume having $\sim 100$ B-scans and each B-scan being of size $1000\times 512$. The other dataset is from Tsinghua University. The OCT volume data were obtained with Spectralis (Heidelberg Engineering, Heidelberg, Germany) in Beijing Hospital. It consists of 448 AMD volumes and 462 DME volumes. Each volume has 25 B-scans, and each B-scan is of size $1000\times 512$.

3.3.

Experimental Settings

For all B-scans in OCT volumes, we standardize them with the mean value of 0.45 and the standard deviation of 0.23, as ResNet-50 does usually. During the training stage, we fix weight parameters in the Conv1 to Conv3 blocks and train weight parameters in the Conv4 and Conv5 models and in the FC layer, alternately using B-scan training samples. To prevent the model from overfitting, data augmentation techniques are used. Specifically, we adopt two data augmentation strategies on each B-scan in the training set: cropping and horizontally flipping. Cropping can increase the diversity of data samples when the proper cropping parameter is selected, and horizontally flipping can preserve the generalization of the left and right eye samples. In our experiments, we do data augmentation online during the training stage as follows: (1) make a crop of random size (0.7 to 1.0) of the original B-scan size and a random aspect ratio (3/4 to 4/3) of the original aspect ratio and then resize it to $224\times 224$; (2) horizontally flip the cropped B-scan randomly with a 50% probability, i.e., randomly generate a number from 0 to 1 and flip the image if the number is $<0.5$.

Softmax cross-entropy loss and momentum-based stochastic gradient descent (SGD) are utilized to train the model. The learning rate is set to $5\times {10}^{-4}$, and the momentum factor is set to 0.95. At each iteration, the minibatch size is set to 64 B-scans. The number of epochs is set to 3. As a volume-level classifier, the linear support vector machine (LSVM) with a $L_2$ penalty and squared hinge loss is used. We transfer the 2-D feature map of a volume into a 1-D feature vector with lexicographical order for the input of LSVM. The weight parameters of the CNN classifier as shown in Fig. 3 are initially trained using 2-D feature maps with a size of $100\times 2048$ for the Duke dataset and $25\times 2048$ for our private dataset. Softmax cross-entropy loss and momentum-based SGD are used to train the model. The learning rate is set to $1\times {10}^{-3}$, and the momentum factor is set to 0.9. The number of epochs is set to 150, and the minibatch size is set to 64 2-D feature maps at each iteration. For the CNN_CBAM classifier, the code of CBAM is available at the Github repository: https://github.com/Jongchan/attention-module.

3.4.

Evaluation Metrics

We utilize accuracy (ACC), sensitivity (SE), and specificity (SP) to evaluate the performance of our proposed methods. They are defined as

We perform one fivefold cross-validation for both datasets. Each class of volumes is randomly divided into five approximately equal subsets at the volume level. The experiments are repeated five times, and for each experiment, four subsets are used as the training set and the remaining set is used as the test set. We report the mean and standard deviation of the metrics ACC, SE, and SP for each method.

3.5.

Experiments on Duke Dataset

In our experiments, all AMD volumes are randomly divided into five approximately equal parts, denoted as AMD_1, …, AMD_5. Similarly, normal volumes are divided into five parts as NOR_1, …, NOR_5. By doing so, we split the Duke dataset into five parts, D_i; here D_i consists of all of the OCT volumes in AMD_i and NOR_i, $i=\mathrm{1,2},\mathrm{3,4},5$. For each fold experiment, four parts of D_1, …, D_5 are used as the training set to finetune ResNet-50 and to train the volume-level classifier, and the remaining set is used as the test set to evaluate the volume-level classifier.

3.5.2.

Comparison with state-of-the-art methods

We compare our methods with several methods, such as that proposed by Santos et al.²⁷ and the voting method of Qiu et al.,¹⁷ on the Duke dataset. Santos et al. obtained classification results with an SVM classifier using fivefold cross-validation with 100 repetitions. Qiu et al. obtained classification results using a voting strategy using fivefold cross-validation with five repetitions.

In voting inference methods, we choose FTA-ResNet50 as the B-scan classifier. For any test OCT volume $X_i$, we perform classification on all of its B-scan $X_{ij}$ to get the class label $Y_{ij}$. Let $\epsilon$ be a threshold, according to a voting inference strategy, we obtain the volume-level label $Y_i$ of $X_i$ by computing $P_i^L$ and comparing it with $\epsilon$, where $P_i^L$ stands for the percentage of the B-scans labeled as $L$ in $X_i$, $L=\mathrm{AMD}$, NOR for the Duke dataset. The voting strategy is as follows: if $P_i^{\mathrm{AMD}}$ is larger than $\epsilon$, then the classification result $Y_i$ of $X_i$ is AMD; otherwise $Y_i$ is NOR. How to determine optimal thresholds is important. Some empirical thresholds are given in Refs. 16 and 17. To compare our methods with the voting inference methods using an optimal threshold, we perform tests to demonstrate the relationship between accuracy, sensitivity, specificity, and threshold $\epsilon$, using fivefold cross-validation for $\epsilon =0.1,0.2,\dots ,0.9$. The voting classification results are shown in Fig. 4, where the solid lines represent the average values and the shaded part of the corresponding color is the confidence interval. It is evident that ACC achieves the best results when $\epsilon =0.6$, and we choose the best ACC to compare with our method.

Fig. 4

Voting results: the relationship between accuracy, sensitivity, specificity, and threshold.

Performance comparisons of our proposed method FTA-ResNet50+CNN_CBAM with other methods are given in Table 2. Table 2 shows that the performance of our proposed method is much better than the others. We note that the voting strategy uses only information that is available at each B-scan from a 3-D OCT volume for classification, whereas our proposed volume-level classifiers can integrate information from B-scans. This maybe the reason that our proposed methods outperform the common voting strategy.

Table 2

Performance comparisons with state-of-the-arts on two classes of the Duke dataset (%).

Methods	ACC	SE	SP
Voting strategy	$87.76\pm 7.09$	$90.74\pm 13.25$	$80.87\pm 18.77$
Santos et al.27	$95.20\pm 2.30$	$94.20\pm 3.10$	$97.50\pm 3.20$
Qiu et al.17	$95.88\pm 0.19$	$98.22\pm 0.72$	$90.43\pm 2.36$
FTA-ResNet50+CNN_CBAM	$\mathrm{98.17}\pm \mathrm{0.64}$	$\mathrm{99.26}\pm \mathrm{0.91}$	$\mathrm{95.65}\pm \mathrm{2.75}$

The method proposed by Sun et al.²⁹ achieves a classification accuracy of 94.4% on the Duke dataset, wherein a different train/test set separation is used.

3.5.3.

Visualization

In this section, we intuitively demonstrate the reasonableness of the proposed method in reasoning. Taking the 20th volume of an AMD set and the 65th volume of an NOR set and denoting them as AMD20 and NOR65 respectively, in the original Duke dataset as representative samples, these two volumes belong to a training set in an experiment in which D_3 is the test set. We utilized the finetuned ResNet-50 as the B-scan feature extractor.

Visualization of B-scan feature vectors

We select a representative AMD B-scan image in AMD20 and an NOR B-scan image in NOR65, as shown in Fig. 5, and visualize their feature vectors in Fig. 6. Figure 6 shows that feature vectors of the AMD and NOR B-scan images have different patterns.

Fig. 5

Representative B-scan examples: (a) 50th B-scan in AMD20 and (b) 50th B-scan in NOR65.

Fig. 6

Visualizations of B-scan feature vectors: 50th B-scan (red) in AMD20 and 50th B-scan (blue) in NOR65.

Visualization of 2-D feature maps

For AMD20 and NOR65, we visualize their 2-D feature maps with a size of $100\times 2048$, as shown in Fig. 7. In the two maps, their feature values are between 0 and 5, and the color bars show the quantized colors for visualization. Figure 8 visualizes the feature vectors of AMD20 and NOR65 from 35 to 65 B-scans, respectively. Overall, the texture in NOR65 is more uniform than that in AMD20, and NOR65 possesses larger feature values than AMD20 does. The 2-D feature map has data correlation between rows, reflecting the correlation of B-scan features in 3D, and 2-D feature maps of AMD volume and NOR volume are intuitively distinguishable.

Fig. 7

Visualization example of 2-D feature maps (a) AMD20 and (b) NOR65.

Fig. 8

Visualization of the feature vectors of AMD20 and NOR65 from 35 to 65 B-scans.

3.6.

Experiments on Tsinghua Dataset

Our private dataset is obtained using FAST scan mode (see Fig. 9), in which every volume has 25 B-scans. These OCT data were collected from patients in clinics, and no volunteers were recruited, so only AMD and DME volumes are included. Two representative examples are shown in Fig. 10. Our observations demonstrate that distinguishing a AMD B-scan from a normal B-scan in the Duke dataset is more difficult than classifying AMD and DME B-scans in the private dataset.

Fig. 9

FAST scan mode.

Fig. 10

Examples of B-scan images from Tsinghua dataset (a) AMD and (b) DME.

We note that, in general, a dataset often consists of AMD and NOR volumes or DME and NOR volumes. In these cases, NOR volumes are often considered negative samples and AMD or DME volumes as positive samples, and accuracy (ACC), sensitivity (SE), and specificity (SP) are calculated with the equation in Sec. 3.4. For our private dataset, to calculate ACC, SE, and SP using the same equation as before, without loss of generality, we take AMD as a negative and DME as a positive sample.

We test to validate the effectiveness of our proposed algorithms FTA-ResNet50 +SVM, FTA-ResNet50 +CNN, and FTA-ResNet50+CNN_CBAM on classification of AMD and DME volumes. Every volume has 25 B-scans, so the 2-D feature map of the OCT volume is of size $25\times 2048$. For this application, we let $n=3$ in Fig. 3, i.e., three convolutional blocks are used in our CNN and CNN_CBAM classifiers. We partition the dataset into five parts with roughly the same amount, conduct fivefold cross-validation, and evaluate our proposed methods. Our experiments show that, for the finetuned ResNet50 feature extractors, the classification accuracies (ACC) achieve $100\pm 0.00\%$, $99.78\pm 0.27\%$, and $99.89\pm 0.22\%$ for SVM, CNN, and CNN_CBAM volume-level classifiers, respectively. This reveals the effectiveness of the proposed methods on the one hand and implies that it is more distinguishable between AMD and DME on the other hand. To be able to better discriminate the classification performances of SVM, CNN, and CNN_CBAM volume-level classifiers, we redesign our experiment with a different training/test set partition. We randomly select 40% of the AMD and DME volumes as the training set separately, and the remaining 60% of the AMD and DME volumes as the test set. This procedure is repeated five times. The classification results of methods are shown in Table 3.

Table 3

Classification performance of our methods on Tsinghua dataset (%), where 40% AMD and DME volumes are training set, AMD is negative, and DME is positive.

Table 3 also shows the effectiveness of our proposed methods. For the private dataset, the volume-level classifier SVM is the best of all, while CNN_CBAM is slightly better than CNN. We conduct further experiments to show the number $n$ of convolutional blocks in the CNN and CNN_CBAM classifiers. Comparing with $n=3$, the experimental results are a little better for $n=2$ and a little bit worse for $n=4$. So $n=3$ is a feasible number for the private dataset.

3.7.

Discussion

This report focuses on volume-level classification in which only the label of the OCT volume is known. Our classification scheme was first proposed in Ref. 30. Here, we did an in-depth study. The proposed classification architecture is general, consisting of three modules: B-scan feature extractor, 2-D feature map generation, and volume-level classifiers. The finetuned ResNet-50 is selected as the B-scan feature extractor and the retraining scheme of ResNet-50 is provided. In our finetuning strategy of ResNet-50, the label of an OCT volume is assigned to each B-scan of the volume. An OCT volume with the label AMD often includes many normal B-scans, which would lead to many noisy labels. This kind of disadvantage was pointed out first in Ref. 17, and a relabeling technique was proposed to overcome it. Hence, the finetuning strategy suggested in this paper should be improved by combining it with the relabeling technique or integrating it with attention techniques. Apart from ResNet-50, other classical CNN models could also be considered the backbone networks for B-scan feature extraction. When 2-D feature maps of OCT volumes are generated, how to design classifiers to classify them is another key point. In this aspect, we adopt traditional LSVM and propose CNN with/without attention mechanism as volume-level classifiers. Our experiments show that all of these volume-level classifiers are very successful. Our proposed OCT volume classification methods do not need any OCT denoising or retinal flattening image preprocessing, they outperform the state-of-the-art methods greatly on the publicly available Duke dataset, and they are also very effective on the private dataset. Of course, extending this private dataset to include the NOR data is encouraged and will be a future effort.

In our experiments, finetuning ResNet-50 requires $\sim 7.1\mathrm{G}$ GPU RAM, whereas training of the SVM and CNN classifiers requires $\sim 0.6\mathrm{G}$ CPU RAM and 1.8G GPU RAM, respectively. Hence, our classification scheme also saves memory resources, so it is highly suitable for large OCT volume datasets. Therefore, the proposed scheme is very promising in assisting ophthalmologist to screen macular diseases from OCT volume.

For given datasets, our methods were used to recognize macular diseases such as AMD and DME. In principle, the proposed 2-D feature map representation is not limited to OCT volume; it may be adapted to any other 3-D medical data such as volumetric magnetic resonance imaging and/or computed tomography (CT) data.