3.1.

Data Collection Protocol

Cereal crop GS is categorized by means of pre-defined scales. Each scale assigns a value to a recognizable crop stage. The most frequently adopted scale is Zadoks.⁷ The principal GSs of the Zadoks scale are listed in Table 1.

Table 1

Zadoks GS scale metric, including principal and minor growth stages.7

Ground truth was determined in the field by an agricultural scientist, or operator, who had sufficient knowledge of cereal GS metrics. GS was determined manually by comparing the plants to the objective visual features defined in the scale.

Images were recorded with a DJI Osmo+ camera.²⁰ The DJI Osmo+ includes a camera, gimbal, and a supporting mobile device handle. The recording was captured with 1080 pixel quality and at $30\text{frames}/\sec$. At each visit, the operator walked the field, along the tramlines for 3 to 6 min recording a video file of crops. Two camera poses were used: vertically downward looking at the field and at a 45-deg declination from the horizon. The camera was held parallel to the sowing rows of the field at a height of 2 m above the ground. In the post-processing stage, the video frames were extracted as image files for training and testing the network. A series of images were extracted and indexed sequentially. To ensure that no two images were the same, frames were extracted with a minimum of 120 ms between each.

The data collected included the ground truth GS, crop cultivar, seed rate, sowing date, date of capture, field global positioning system (GPS), brightness level, and wind speed. The data were captured over 3 years of growing seasons from 2017 to 2019.

3.2.

Wheat Dataset

The seen field-day wheat training dataset consists of 21 GS classes where each class includes 2000 images for training, 600 images for validation, and 1400 images for test purposes. These 21 classes include four classes in the seedling stage, four classes in the tillering stage, two classes in stem elongation, two classes in ear emergence, one in anthesis, three in milk development, two in dough development, and three in ripening. There are overall 84,000 wheat images in this dataset. The wheat training images are from five distinct fields in Ireland and include two different cultivars in Costello and JB Diego. The brightness range in the wheat training dataset varies between 73.0 and 156.2 (AV). There are five different seed rates in the wheat training data. The wind speed at wheat data capture time varied between 6 and $27\mathrm{km}/\mathrm{h}$. Figure 1 shows sample of wheat images from these two cultivars and various GSs.

Fig. 1

Samples of wheat image from two cultivars of Costello and JB Diego. Image samples of GS14, GS22, GS76, GS23, and GS69 are from downward set of data. Image samples of GS30, GS55, and GS88 are from 45-deg-angled set of data.

The unseen field-day dataset, which is separate from the training data, consists of 13 GS classes with 2000 images per class. These classes include two from the seedling stage, three from tillering, one from stem elongation, two from ear emergence, two from milk development, one from dough development, and two from ripening. Each class of test data includes 1000 downward and 1000 images of 45-deg-angled looking. Overall, 26,000 images of unseen field-day wheat data are in the test dataset. The data include brightness variation from 75.9 to 168.2 (AV) and three different seed rates. The wind speed at the time of capture wheat test data varied between 16 and $28\mathrm{km}/\mathrm{h}$. The unseen field-day data are only used for testing, not training.

Details of the wheat dataset are provided in Table 2 and the seen and unseen field-day split is listed in Table 4(a). Information the about fields and their GPS can be found in Table 5.

Table 2

Wheat training and test dataset.

3.3.

Barley Dataset

The barley seen field-day dataset includes 20 GS classes where each class includes 2000 images for training, 600 images for validation, and 1400 images for test. These 20 classes include four classes in the seedling stage, four classes in the tillering stage, two in stem elongation, one in booting, two in ear emergence, one in milk development, three in the dough development, and three in ripening. There are 80,000 barley images overall in this dataset. The barley training, validation, and test images are from four fields and include two different cultivars of Cassia and Infinity. The brightness range in the barley training data varies between 76.4 and 159.6 (AV). There are four different seed rates in the barley training data. The wind speed at the time of capturing the barley training data varied between 9 and $27\mathrm{km}/\mathrm{h}$. Figure 2 shows sample barley images from these two cultivars at various GSs.

Fig. 2

Samples of barley image from two cultivars of Infinity and Cassia. Image samples are from downward set of data.

The unseen field-day barley dataset, which is separate from the seen field-day data. It includes 13 GS classes with 2000 images per class. These classes include two from seedling, four from tillering, two from stem elongation, two from ear emergence, one from dough development, and two from ripening. Each class of test data includes 1000 downward and 1000 images of 45-deg-angled looking. Overall, 26,000 images of unseen field-day barley data are in the test dataset, which was collected from three distinct fields in Ireland. These images include brightness variation from 81.2 to 157.8 (AV) and three different seed rates. The wind speed at the time of capturing barley test data varied between 14 and $23\mathrm{km}/\mathrm{h}$.

The unseen field-day data are only used for testing and not training. Details of the barley dataset are provided in Table 3 and the seen/unseen field-day split is listed in Table 4(b). Information about the fields and their GPS can be found in Table 5.

Table 3

Barley training and test dataset.

Table 4

Seen and unseen field-day test split by GS (a) wheat dataset and (b) barley dataset.

Table 5

Fields and their GPS coordinates.

4.1.

SVM Classifier

GS classification of wheat and barley crops was investigated using feature extraction and an SVM classifier.²¹ Blurry images were detected using a Laplacian Kernel and were removed from dataset below a threshold of 120.²² Data were pre-processed by brightness correction.²³

The best results using the SVM classifier were obtained by training on a mix of downward and 45-deg-angled looking images in each class of data. Excess Green index features²⁴ were extracted from images. Data dimensionality was reduced by employing principal component analysis. The SVM classifier was equipped with a radial basis function kernel,²⁵ and regularization parameters of $C=1.0$ and $\gamma =0.1$. A five-fold cross validation scheme was applied and 1400 images per class were utilized for testing purposes. Moreover, for each crop (wheat/barley) 13 classes of unseen field-day test data were used for principal GS classification.

GS classification using the SVM classifier with input pre-processing and a mix of downward and 45-deg-angled looking images in each class, resulted in 63.8% and 59.8% accuracy for wheat and barley, respectively. Principal GS classification using the same classifier on unseen field-day test data resulted in 26.4% and 29.3% accuracy rates for wheat and barley, respectively. Table 6 presents a summary of the experimental results obtained using the SVM classifier.

Table 6

The result of GS classification and principal GS classification of unseen field-day data using SVM classifier.

4.2.

ConvNet with Learning from Scratch

Two ConvNet models were trained from scratch for GS image classification and principal GS classification of wheat and barley crops.

The first ConvNet includes five-trainable layers including three Conv layers and two dense layers. The Conv layers (Conv2D, Conv2D-1, Conv2D-2) have 32, 64, and 64 filters respectively and the filter size was set to $3\times 3$ and the dense-layers have 1024 and 21/20 neurons for wheat/barley, respectively.

The second ConvNet is almost identical to the first ConvNet apart from two layers of batch normalization that are added to the network after the max-pooling layer of the first and the third trainable layers. The following paragraphs are summarized and the equations were removed.

For all ConvNet experiments, an image size of $256\times 256$ was employed. An image size of $125\times 125$ was tested but the results were not satisfactory. The image pixel values were rescaled to $[\mathrm{0,1}]$ interval. The training data were pre-processed using the hue, saturation, value color space, employing the brightness correction function.

Since data augmentation has proven effective in training deep learning algorithms.²⁶ Three different data augmentation schemes were applied to the input of the network. The data were augmented with various in-range brightness values.²⁷ To this end, while reading the images into the training data-generator, the brightness range was set to produce either darker images (setting uniform distribution values $<1.0$) or brighter images (by setting uniform distribution values over 1.0). See Eq. (1) for the transform equation and Table 7 for the brightness parameter setting. In this work, the brightness range is set to $[0.7,1.3]$ for data brightness augmentation

Table 7

Description and range of values of the parameters used for brightness, rotation, and zoom augmentation, where u(a,b) denotes a uniform distribution.

The network was made robust,²⁸ to 90-deg rotation ranges by data rotation augmentation.²⁷ This method randomly rotates the image clockwise by the given angle. The affine transformation for rotation can be found in Eq. (2). The rotation range parameter setting employed in this work is listed in Table 7.

Zoom augmentation²⁷ was exerted on the training data by employing a scale range of $[0.7,1.3]$. The affine transform for zoom augmentation complies with Eq. (2) and the parameter settings of horizontal and vertical scale were obtained from Table 7. This function randomly produces images that are zoomed in for values $<1.0$ (interpolates original image pixel values) and zoomed out for values greater than 1.0 (add new pixel values around the original image).

The input for each class of data includes 50% downward and 50% angled images. The test data were classified in the principal GS range without any pre-processing.

An extensive series of experiments were carried out to find the best performing ConvNet with learning from scratch model and input format for the GS classification and principal GS classification task. The results demonstrate an improvement when employing the ConvNet with batch normalization layers. Moreover, including input pre-processing and data augmentation was proven to play an important role for both the classification and principal GS classification tasks using this network, see Table 8.

Table 8

Overall performance of the ConvNet trained from scratch for GS classification on test data and principal GS classification on unseen field-day test data for wheat and barley crops. There are three different experiments: whether (a) the network includes batch normalization layers, (b) the input data is a mix of downward and 45-deg-angled looking images, and (c) training includes pre-processing and data augmentation

The best average results for barley and wheat using the ConvNet learned from scratch including batch normalization, input pre-processing, and data augmentation is 95.9% GS classification accuracy, and principal GS classification accuracy of 75.3%.

4.3.

ConvNet with Transfer Learning

The ConvNet with transfer learning approach seeks to transfer knowledge from a source task to a target task. The network’s pre-trained parameters from the source task are re-purposed for a target task within a similar or related domain. The concept of transfer learning relaxes the urgent requirement of having the ConvNet trained on a large independent and identically distributed dataset.

The following paragraphs are summarized and the description of experiments are removed.

The Visual Geometry Group ConvNets has provided a reliable base for numerous image recognition systems since its introduction in 2014.²⁹ In this work, Visual Geometry Group-19 was employed as a basis for transfer learning, with 19 weight layers of 16 Conv and three fully connected (FC) layers. The network is pre-trained on the ImageNet dataset and the knowledge can be transferred at any layer of the network for the new classification task. To acquire the best architecture for setting trainable and non-trainable layers for the classification problem at hand, five different experiments were tested. The experiment models of E1 to E5 are listed in Table 9, including non-trainable and trainable layers and parameters.

Table 9

Series of experiments to find the best-performing transfer learning method, the table includes the description and number of trainable parameters.

The input for each class of data includes 50% downward and 50% angled images. Similar to the data preparation of ConvNet with learning from scratch (presented in Sec. 4.2), pre-processing and data augmentation were applied to the training data of this network. The training data are augmented for brightness²⁷ in range $[0.7,1.3]$, rotation in the range 90 deg²⁸ and zoom augmentation with scale in the range $[0.7,1.3]$.²⁷ The test data were classified in the principal GS range without any pre-processing or data augmentation.

Among the aforementioned experiments, Experiment E4 achieved the best GS classification accuracy, with 15 non-trainable Conv-layers, including the last Conv-layer and four FC-layers of 1024, 512, 256, 21/20 nodes as trainable layers. The input setting for this practice includes data pre-processing, data augmentation and a mix of downward and 45-deg-angled images in each class of data.

The result from Experiments E1 to E5 were investigated to choose the best transfer learning model, Fig. 3. The accuracies for experiment E1 are 76.2% and 73.4% for wheat and barley respectively, these increased to 99.1% and 99.7% in experiment E4. Moving deeper by training the network with another trainable ConvNet layer, accuracy drops slightly to 98.1% and 98.3% for wheat and barley, respectively. This is a costly drop in accuracy as the number of trainable parameters doubles from $\sim 3.5$ to $\sim 6$ million.

Fig. 3

GS classification accuracy rate, including network’s number of trainable parameters for each experiment. Model trained on (a) wheat and (b) barley data.

It is shown that experiment E4 resulted in the best performance with a reasonable number of trainable parameters. Hence it was used as the base model for training and testing the GS classification.

The results obtained for the various methods of GS classification and principal GS classification employing experiment E4 are presented in Table 10.

Table 10

Overall performance of ConvNet with transfer learning experiment E4, for GS classification and principal GS classification of wheat and barley. There are three different experiments: whether (a) the input data is a mix of downward and 45-deg-angled looking images, (b) training data includes pre-processing, and (c) training data includes data augmentation.

The first two rows of Table 10 present the results of training with single mode data (either angled or downward looking images) with no pre-processing or data augmentation. The result of GS classification using these inputs is fairly good with 93.6% and 92.4% accuracy for wheat and barley, respectively. However, principal GS classification using the aforementioned trained network does not yield good results for unseen field-day data; generating principal GS classification accuracy of 73.1% and 70.4% for wheat and barley, respectively.

The next series of experiments involved training the transfer learning network using pre-processed data with brightness correction. The network classifies GSs with almost the same accuracy rates as the previous experiment. However principal GS classification improved noticeably reaching 77.6% and 75.3% accuracy for wheat and barley, respectively.

Including input data augmentation as well as pre-processing brings about higher GS classification and principal GS classification accuracies. The results show 95.3% and 93.1% accuracy for wheat and barley GS classification. Moreover the principal GS classification progressed positively and achieved 83.8% and 85.2% accuracy for wheat and barley, respectively.

Finally, including downward and 45-deg-angled images in each class of data for training was considered. The input data were pre-processed and augmented as well. A significant improvement was noticed in both classification accuracy rates and the number of images from each class correctly classified in their corresponding principal GSs. Employing this input setting to transfer learning, the network achieved 97.3% and 97.5% GS classification accuracy rates and principal GS classification accuracies of 93.5% and 92.2% were achieved for wheat and barley, respectively, see Table 10.

For both GS classification and principal GS classification tasks, the network architecture of experiment E4 trained on a mix of downward and 45-deg-angled looking images in each class, including pre-processing and data augmentation, achieved the best results. The confusion matrices for principal GS classification of wheat and barley unseen field-days data are presented in Figs. 4(a) and 4(b), respectively.

Fig. 4

(a) Wheat and (b) barley, principal GS classification employing ConvNet with transfer learning experiment E4, presented in confusion matrix of actual GSs on left and the classified principal GS range at top.