A novel multi-scale based deep convolutional neural network for detecting COVID-19 from X-rays

1.1. Motivation and contribution

In recent years, tremendous progress has been made in measurement science by applying deep neural networks (DNN) techniques to computer vision applications such as salient object detection [13], [14], facial expression recognition [15], [16], and deception detection [17], thus DNN models have become the defacto-standard nowadays. DNN has specialized in learning-rich images with high-level discriminatory semantic characteristics automatically, eliminating the need for hand-crafted descriptors. These breakthroughs have revealed that deeper models can improve results [16]. Thus, it is viable to train a DNN model to obtain promising performance in COVID-19 screening and monitoring. Moreover, technological advancement has enabled the manufacturing of low-cost portable computing devices for consumers. Cellular devices have advanced in terms of technical capabilities and processing power, and they have become a source of information, interaction, and sharing. They are now almost indispensable in our daily lives. Internet of Things (IoT) with cellular devices have permitted a far wider range of uses, not only for entertainment but also for the treatment and monitoring of health requirements, environmental surveillance, home automation, and many more [18]. Therefore, the motivation for this study is twofold. Firstly, there is a lack of resources and screening tools for identifying and monitoring COVID-19 patients, and secondly, DNN has a great potential for fetching features and accurately classifying images without any manual intervention. This work introduces a framework that includes a novel DNN enabled IoT service to intelligently measure the severity of COVID-19 lung infections by analyzing CXR images. The proposed DNN module consists of multi-scale sampling filters that allow extracting more reliable and noise-invariant features at different image patches. We have circumvent the shortcomings of the existing DNN models and achieve superior performance by carefully designing the proposed DNN model-based multi-scale sampling. All the experiments are implemented on five databases, namely COVIDx (D1) [19], COVID-19 Radiography (D2) [20], COVID-XRay-5K (D3) [21], COVID-19-CXR (D4) [22], and COVID-chestxray (D5) [23]. The proposed framework is compared with fourteen existing approaches by utilizing four well-known classification metrics viz., F1-score, recall, precision, and accuracy. Empirical evidence manifests that the proposed method outranks all the fourteen existing approaches. The integration of the proposed algorithm with an IoT framework results in an efficient and precise real-time online service for COVID-19 diagnosis. The contributions of this study can be summarized as follows:

• •

  A detection and monitoring tool for the diagnosis of COVID-19 patients is introduced. This framework is instrumented with an IoT system that helps to oversee both potential and real cases. Thus, the newly developed equipment can be employed to observe patients efficiently, especially in a remote location.

• •

  A novel DNN framework is designed for distinguishing non-COVID-19 from COVID-19 classes using CXR images. The use of X-ray simplifies the implementation of the proposed method in real-world scenarios. When compared to other testing procedures, X-rays are less expensive and take less time.

• •

  The proposed DNN consists of multi-scale filters. The strength of multi-scale sampling filters to fetch robust and noise invariant facets with distinguishing power is exploited. We hypothesize that by integrating multi-scale feature extraction, we can learn more resilient convolutional filters since the scale of features varies substantially among distinct ground objects captured from several sensing devices. Moreover, our proposed DNN is simple as it has fewer layers and learning parameters.

• •

  We give insights into the theoretical enhancement made to the DNN model and document their empowering effect through experiments. The experimental results illustrate that the computation cost is considerably lower compared with some related approaches. This confirms that our approach is more computationally efficient.

The organization of the current study is structured as follows: A concise summary of a few notable previous approaches related to COVID-19 classification is put forward in Section 2. Section 3 describes the proposed work in depth. The obtained outcomes of the proposed model along with other baseline approaches are reported in Section 4. At last, Section 5 concludes the current study.

3.1. The IoT framework

Social distancing is a non-pharmaceutical method of prevention. When we are forced to stay locked up in our homes, the IoT revolution plays an important role in modern healthcare systems in terms of professional, social, and economic prospects. As a result, in the context of the current pandemic, IoT-enabled applications can be used to reduce the potential spread of COVID-19 through early and remote diagnosis. As a result, the present study introduces an end-to-end IoT framework to help virtually the patients in remote locations in the event of a pandemic. The challenges associated with each layer of the proposed framework are addressed, and design guidelines for dealing with them are discussed. This sub-section describes the developed IoT-based framework for observing and recognizing COVID-19 cases. This framework can also be used to track how well-reported patients respond to treatment and learn more about the COVID-19 disease. The proposed IoT framework is shown in Fig. 1, consisting of six steps labeled from 1–6. The doctor can upload a COVID-19 X-ray image or a group of images to an internet application from this screen. This method will extract information from images and classify them as non-COVID-19 or COVID-19. The proposed method extracts features from an image using the DNN model, followed by a softmax classifier that uses the extracted features as inputs to classify COVID-19. This method makes use of the LINDA, which is available as a web service. It consists of a processing flow that can (i) extract, (ii) train, (iii) predict, and (iv) store the statistics and results obtained for COVID-19 recognition. All computational processing for this IoT system is done in the cloud. The server is housed at the Instituto Federal de Educaço, Ciência, e Tecnologia do Ceará. LINDA has recently gained popularity, and it has been used to develop not only medical IoT services such as stroke classification based on cerebral vascular accident images, melanocytic lesion classification based on skin images [40], [41], but also machine condition monitoring [42].

When patients exhibit COVID-19-related symptoms, smartphones, computers, and other electronic devices are permitted to transmit information and X-ray images to LINDA. A user can perform a variety of operations in LINDA, including defining the number of classes, configuring the extraction and classifier characteristics, and changing the extractors and classifiers used. LINDA also includes a graphical dashboard with metrics for evaluating the performance of the extractor and classifier. The IoT system supports the Python programming language, the PostgreSQL database, the TensorFlow, and Keras frameworks. The flow of the developed LINDA-based IoT system is depicted in Fig. 2: the first phase entails integrating the five X-ray image databases, as well as the feature extraction and classification procedures.

The data flow of the LINDA system is depicted in detail in Fig. 1. The information flow of the LINDA system will begin by sending an image from a device, as shown in Fig. 1 by the number 1. A security hash code will also be sent to the system. The system will then call the prediction API, which will select the algorithms to use based on the secure hash. The required models will be loaded into memory. If the system settings have not been completed and some changes are required, the web application (number 2) will be used to upload and categorize images. The proposed DNN is deployed on this platform, and the algorithm was trained on five databases. Section 3.2 contains a detailed description of DNN. To learn more about LINDA, interested readers should visit [43]. The proposed method has three advantages:

• •

  There is no need for face-to-face communication between physicians and subjects, which reduces medical staff exposure to infection.

• •

  The proposed application diagnoses the X-ray image in less than a second, allowing faster response in case of positive cases.

• •

  By virtue of enjoying a short development cycle, the proposed IoT-based service can be easily upgraded at a minimal cost without disrupting the service.

3.2. The proposed DNN architecture

In this study, we developed a multi-scale DNN system for extracting and recognizing COVID-19 features from CXR images. DNN automatically learns the various features from X-ray images using different scales, and these facets are learned by training the network over several iterations. In previous research, researchers discovered that convolutional sampling on fixed scales frequently limits a DNN’s ability to find local invariant patterns, whereas multi-scale sampling allows a DNN to find more reliable and noise-invariant features at different image patches. To address this scope in the context of the current study, a variable filter size (7 × 7, 5 × 5, 3 × 3) is used at various convolutional layers with strides of 1 × 1. Before training the network, pre-processing is performed on X-ray images as shown in Fig. 3. Fig. 4 displays the architecture of the proposed DNN. The DNN extracts robust and geometrically invariant patterns from different patches of X-ray. The input to the DNN is gray-scale images. The proposed DNN consists of 5 blocks. The first block contains 3 convolutional layers with different filter banks and ${Block}_2$ consisting of pooling layers, which are stacked with ${Block}_1$ convolutional layers as shown in Fig. 4. The features obtained from the ${Block}_2$ are concatenated into a single feature vector. Later, three convolution operations are applied with various filter banks (i.e. ${Block}_3$) on concatenated vector and combined all the features obtained from ${Block}_3$, then, they are stacked with a single convolutional layer consisting of 1 × 1 filter bank followed by max-pooling operation with filter size 4 × 4. Finally, we find ${Block}_5$ which is composed of two fully connected (FC) layers and a softmax layer of sizes 256, 512, and 2. Technically, the first four blocks (${Block}_1,{Block}_2,{Block}_3,{Block}_4$) are considered for feature extraction and the last block (${Block}_5$) is employed for classifying COVID-19 using X-rays i.e. final mapping to the output. Table 2 reports the comprehensive description of each layer and its parameters.

Furthermore, filters 7 × 7, 5 × 5, and 3 × 3 are utilized to capture the enriched contextual information. Moreover, the 1 × 1 filter is used as an identity function.

Smaller window sizes (i.e., 2 × 2) are used for the pooling layer in the proposed method as the highest information loss occurs due to the pooling layers. A max-pooling scheme is considered in this work. The formal description of the model is defined mathematically by Eq. (1). The filter initialization values $f_{i,k}^i$ is selected at random from the distribution defined by the filter size, input, and output number of the specific layer’s feature maps where a uniform distribution with upper and lower bounds of $\pm k$ is defined by $U(\pm k)$. The mathematical formulation for uniform distribution of filter initialization is shown in Eq. (2).

The total number parameters are the sum of the parameters of each layer, where the number of parameters of each convolution layer is $(f\times f\times (w_p+b))\times f_o$, where $f$ is a filter bank size, $p$ is the input number of feature maps, and $b$ is a bias i.e., 1 and $f_o$ is the output number of feature maps. The representation of the $i$th convolutional layer, $L_i$ is shown in Eq. (3). Where $\Delta (.)$ denotes an activation function of a layer. Rectified linear unit (ReLU) is an activation function adopted in this work, where $f_m\in (1,f_o)$, $f_o$ defines the number of facets maps exist in the layer $L_i$, $e$ is the number of input facets maps in previous layer i.e., $L_e^{i-1}$. $a,b$ denotes the coordinates of the feature maps, and $\star$ indicates convolution operation. Eq. (3) is further elaborated as shown in Eq. (4).

The filter bank size of the layer $L^i$ is $(2\times m+1)\times (2\times m+1)$. In the proposed model, the “same” padding is applied to keep the size of the feature map constant. Max-pooling operation is performed on the output of convolutional layer $L_{f_m}^i$ with filter bank of size 2 × 2. From each feature map, max-pooling measures the utmost importance of each patch to highlight the primary feature represented within the patch. Max-pooling also reduces the number of parameters to make the model simple, In addition, it provides feature maps that are invariant to translation, rotation, and scale. As shown in Fig. 4, the input images of 128 × 128 pixels are down-sampled by the max-pooling layer, resulting in filter maps of various sizes after each layer of convolution in the ${Block}_2$, later concatenated, with $C_1$ being the output obtained by the max-pooling layer. Further, ${Block}_3$ is stacked with $C_1$, containing variable sizes of filter banks for convolution operation. Followed by concatenation operation, $C_2$ is applied on outputs of ${Block}_3$, which is called ${Block}_4$. The convolution operation with filter bank of 1 × 1 is applied on output feature maps of ${Block}_4$ followed by a max-pooling operation, then all the feature maps are flattened to a single vector of size 16384 × 1, where $W_i$ and $b^i$ are the weight and bias of the $i$th FC layer. The output of the second FC layer, $F{C}_2$, is further fed into the softmax layer. The softmax layer consists of two neurons and produces a probability vector, $\hat{Z}=[\widehat{z_c},{\hat{z}}_{nc}]$, where, $\widehat{z_c}$ is the prediction score of COVID-19 class and ${\hat{z}}_{nc}$ of non-COVID-19 class. The $j$th probability value is obtained by Eq. (5).

4.1. Experimental setting

This sub-section describes the resources used for experiments. For the training and testing of the proposed and existing models, the Keras framework and Anaconda Python 3.6 package are considered in this study. The specifications of the working system are NVIDIA Quadro P5000 graphics processor, 256-bit memory interface, 16 GB GPU RAM, Cuda core-2560, GDDR5X memory, and 288.5 GB/s bandwidth.

4.2. Experimental data

In this sub-section, we discuss about the databases and evaluation procedure of the proposed DNN model for diagnosing the COVID-19 is described. All the experiments are performed on the five publicly available databases, namely D1 [19], D2 [20], D3 [21], D4 [22], and D5 [23]. The statistical information of these databases is reported in Table 3.

When dealing with a database containing a small number of images, overfitting or excessive variance of ML algorithms is common. The overfitting problem is addressed in this study by considering horizontal flip, random rotation by 10 degrees, and zoom range 0.4 as image augmentation strategies. Moreover, to maintain the consistency of the proposed model, all the X-ray images of five databases are resized into 128 × 128 and each database is divided into three groups: train, validation, and test sets. For all of the investigations, a k(=10)-fold cross-validation methodology is adopted to assess the performance of the proposed method. In other words, out of ten subsets, eight are employed for training, one is used for validation, and the remaining one is utilized for testing. The underfitting and overfitting problems may solve by considering the 10-fold cross-validation technique. Table 3 reports the number of X-ray images employed in the train, validation, and test in the ratio of 0.8, 0.1, and 0.1, respectively. The upper and lower part of Table 3 denotes the number of samples before and after image augmentation. Moreover, four well-known evaluation metrics, namely accuracy, precision, recall, and F1-score are employed for evaluation of the proposed DNN and comparative methods. The detailed description of evaluation metrics is beyond the scope of this study.

4.3. Results

In this sub-section, the empirical results of five databases are discussed. The proposed DNN is evaluated on five databases, namely D1, D2, D3, D4, and D5. Fig. 5 depicts the training procedure for five datasets. As seen in Fig. 5, accuracy improved rapidly during training until it reached an average of 10 to 20 iterations, and then progressively increased. After multiple iterations, the performance of the training and validation sets appeared to be smooth and did not grow any further. Similarly, training and validation losses decreased until it reached 10 to 20 epochs. The training loss assesses how well the model fits the training data, whereas the validation loss assesses how well the model fits new data. We discovered that the proposed DNN can achieve nearly 100% accuracy in training and the best results in validation. The training and validation losses of the proposed DNN are 0.1003, 0.1291 for D1, 0.0019, 0.0167 for D2, 0.0186, 0.01426 for D3, 0.0096, 0.0201 for D4, 0.0099, and 0.0213 for D5. As a result, it is clear that the proposed DNN structure offers considerable benefits in terms of COVID-19 identification.

To illustrate the robustness of the proposed DNN structure on five databases, metrics such as precision, recall, F1-score are measured, which is noted in Table 4. Furthermore, the accuracy of the proposed method is compared with the fourteen existing works on the five databases, and the results are noted in Table 4.

We can summarize the following:

• •

  Table 4 clearly indicates that the proposed DNN framework obtains an average detection F1-score, recall, and precision of 96% on D1, 100% on the D2, 99% on D3, 99% on D4, and 100% on D5 databases respectively. This indicates that the proposed model learns well on X-ray images and it is able to distinguish the features belonging to COVID-19 and non-COVID-19.

• •

  It is observed from Table 4 that the detection accuracy of the proposed method is 96.01% on the D1, 99.61% on D2, 99.22% on D3, 98.83% on D4, and 100% on D5 databases respectively, which is far better than the accuracies obtained by the existing methods. Besides, the error rate incurred by the proposed method on the testing set is 0.1391, 0.0057, 0.0996, 0.0178, and 0.0124 on D1, D2, D3, D4, and D5 databases, which is impressive enough in comparison with the existing methods.

4.4. Robustness of the proposed DNN

In this sub-section, we conduct experiments to find the robustness of the proposed DNN. In basic terms, feature engineering is the process of converting chest x-ray images into desirable features using the proposed DNN in order to improve model accuracy. To compare the performance of the proposed DNN with some state-of-the-art approaches, accuracy, precision, recall, and F1-score are used, and the results are presented in Table 4. The proposed DNN outperforms all state-of-the-art techniques, as reported in Table 4. It means the proposed DNN does the feature engineering task well in comparison to other state-of-the-art methods. Nowadays, the values of these evaluation measures are no longer sufficient to demonstrate how good a DL model is. The t-SNE plots of the feature vectors obtained by the proposed DNN along with state-of-the-art approaches on five different databases are shown in Figs. 7, 8, 9, 10, and 11 in order to illustrate the effectiveness of the proposed DNN model over some of the existing methods. The features in a 2-D plot are depicted using t-distributed stochastic neighbor embedding, a dimensional reduction technique that allows us to perceive a high-dimensional database in a low-dimensional environment. Because such embedding incorporates categorization information, it visualizes the learned proposed network’s most recent embeddings. It is clear from Figs. 7, 8, 9, 10, and 11 that the proposed DNN is the only method, which can extract distinguishable features for five databases separately and forms well-separated clusters when we are mapping them from higher-dimensional to a two-dimensional plane. The t-SNE plots of a few of the existing approaches are also well separated with a small margin on some of the databases. Thus, the performances of these methods are not always consistent. This experiment shows how efficient the proposed method is. Furthermore, to show the proposed method’s efficient feature learning capacity, the probability vector created by the softmax layer is compared to a few existing DL techniques. Initially, the five best state-of-the-art methods are selected based on their accuracies [19], [22], [25], [28], [39]. We pick four CXR images (two are from COVID-19 patients and two are from a healthy person) randomly from the test sets. Then, the probability score of each method along with the proposed approach to these four images is estimated and displayed in Fig. 12. The label ‘0’ and ‘1’ in the graphs of Fig. 12 denote the probability of covid-19 infected patients and healthy persons, respectively. The probability score ranges between 0 to 1 and the ideal probability score of label ‘0’ and label ‘1’ for covid-19 infected patients is 1 and 0, respectively. Similarly, it is vice versa for healthy people. It can be observed from Fig. 12 that the proposed method predicts scores nearer to ideal values for both covid-19 infected patients and healthy persons. On the other hand, probability scores obtained by the existing DNN based approaches for healthy persons and infected patients are relatively far from the ideal values.

In addition to this, we observe that the proposed DNN is evaluated using a 10-fold cross-validation procedure, in which the original dataset is randomly divided into ten equal-size subsamples. Only one subsample of the ten is preserved as test data for the algorithm, while the other nine are used for training and validation. The folds are then used to repeat the cross-validation process ten times, with each of the ten subsamples serving as validation data exactly once. The performance of the proposed DNN can then be estimated by averaging (or otherwise combining) the evaluation metrics acquired from the 10 folds. However, the proposed DNN’s overall accuracy differs from the individual accuracy of each fold. To identify the variance in the obtained accuracies on each database, the standard deviation of accuracies acquired in ten different folds is assessed, as illustrated in Fig. 13. It is clear from Fig. 13 that the obtained low standard deviation on a database implies that 10-fold accuracies tend to be extremely close to the averaged accuracy of the proposed DNN.

4.5. Ablation study

To develop a DNN model from scratch for a particular problem is not an easy task especially when we are facing the data scarcity problem with a database containing few images. An ablation study is required to understand the contribution of each module of the proposed DNN. Thus, an ablation study is conducted to finalize the architecture of the proposed model in such a way that it performs well with a test set. In accordance with our experiments, the proposed DNN gives an accuracy of 96.01%, 99.61%, 99.22%, 98.83%, and 100% for the task of COVID-19 classification of the X-ray images on D1, D2, D3, D4, and D5 databases respectively. In this sub-section, the effect of the performance of the proposed DNN is assessed by varying the model parameters. In the first experiment, the behavior of the DNN model for different activation functions is shown in Fig. 14. Fig. 14 indicates that ReLU activation gives a better performance than others. In the second experiment, the effect of the batch normalization layer on the performance of the proposed DNN is evaluated. The results of the second experiment are presented in the upper part of Table 7. In the third experiment, the effect of change in pooling layers on the performance of the proposed DNN is assessed. The results of this study are presented in the lower part of Table 7. The proposed DNN is divided into five blocks, namely ${Block}_1,{Block}_2,{Block}_3,{Block}_4$, and ${Block}_5$. We have experimented i.e., the fourth experiment, to quantify the block-wise performance of the proposed DNN. The results of the fourth experiment are shown in Table 8.

From Table 8, we can observe that although increasing layers containing blocks the proposed model gives better performance. Moreover, we can observe from the last row of Table 8 is that overfitting occurred after increasing more than 5 blocks. In addition, as seen in Fig. 15, there is a huge gap between training and validation losses when increasing more than 5 blocks. So, the model is overfitting.

Moreover, optimization plays a crucial role in the DL model for updating the weights to reduce the losses of the DL model, also called hyper-parameter tuning. There are many optimization techniques available for parameter hyper-space search. Some of the widely used optimization techniques for DL approaches are stochastic gradient descent (SGD), adaptive gradient descent (Adagrad), root mean square propagation (RMSprop), SGD with momentum, and adaptive moment estimation (Adam). In the fifth experiment, we have tested the aforementioned optimization techniques. The performance analysis of various optimization techniques is shown in Fig. 16. It can be observed from Fig. 16 that Adam optimization gave a better performance for COVID-19 classification from X-ray images. However, the proposed model obtained a satisfactory performance for other optimization techniques. Variations of different scales are examined by the sixth experiment. Table 9, shows the performance of the proposed DNN using different filter scales. To measure the importance of the multi-scale approach, conducted a seventh experiment with and without multi-scale blocks of the proposed DNN. The measurement of the proposed DNN performance with and without multi-scale is shown in Fig. 17.

It is clear from Fig. 17 that the performance of the proposed DNN without multi-scale is not satisfactory. Thus, we can conclude that multi-scale features provide significant features to distinguish COVID-19 from non-COVID-19.

It can be observed from Table 9 that the combination of $3\times 3,5\times 5,7\times 7$ scales gave better performance than other scales. In addition, an experiment is conducted on varying learning rates. Fig. 18, shows the performance varying while changing learning rates. From Fig. 18, it is observed that the learning rate at 0.00001 obtained higher classification accuracy.