Deep learning–based drone detection with SWIR cameras

2.1 YOLO Algorithm

The YOLO algorithm considers the locations of bounding boxes and the class probability as a regression problem to guess the classes and locations of objects by looking at them once on video streams or images. It measures the locations of bounding boxes that cover labeled objects, after dividing the input images by an S ☓ S grid. Anchor boxes that overlap the bounding boxesthat are separated by the grid guess the objects' class probabilities, probabilities of the objects' presence, locations of object centers, and width and height of the bounding boxes to measure the confidence score. As results, the YOLO network outputs information on the bounding boxes and confidence score. Figure 1 shows the entire process.

Figure 1:

Cross-stage partial network structure

In Figure 1, the bounding boxes and class probability are estimated by dividing the input images by 7 × 7 (S = 7), and the locations and classes of the objects are finally recognized by combining them. The confidence score relates to whether objects are really present in the bounding boxes and how much the concerned class is reflected. It can be calculated as shown in Equation (1).

where IoU (Intersection over Union) is the value resulting from dividing the size of the intersection of the correct-answers box and the prediction box by the size of the union, referring to the degree of overlap. In other words, if there is no intersection, IoU is 0, and if they completely overlap, it is 1.

2.2 YOLO v4

YOLO v4 [10] is the latest YOLO-algorithm version released in 2020. Figure 2 shows the object-detection process for YOLO v4.

Figure 2:

Object-detection process of YOLO v4

Data input through a Convolutional Neural Network (CNN) using input images are divided into various channels through the backbone–neck module. Objects are detected through the basic YOLO algorithm for a grid divided in this manner. The performance of YOLO v4 has been improved from the previous method, which used YOLO v3 [11] as the head, a CSP (crossstage partial) network [12] as the backbone, and SPP (spatial pyramid pooling) [13] as the neck with a PAN (path-aggregation network) [14]. Therefore, to improve the performance of the preceding research, this study constructs an image analyzer with YOLO v4.

2.3 CSP

CSP is an optimization method that reduces the computation by 20% by removing the values of repeated gradients generated from the CNN learning process. Figure 3 shows the differences between networks where the CSP method is applied and not applied.

Figure 3:

CSP network structure

In Figure 3, while (a) shows the images going through the dense block using all of the output of the first layer, (b) shows only some passing through the partial dense block, with the remnant conveyed to the transition layer before being combined.

Through this process, the values of repeatedly computed gradients can be removed when the error back-propagation algorithm is applied to the CNN.

2.4 SPP

SPP is a method to improve accuracy, breaking away from using only fixed-sized images, which was pointed out as a limitation of the YOLO algorithm. Figure 4 shows the SPP network structure, with examples from specific networks.

Figure 4:

SPP network structure

In Figure 4, feature maps for as many filters as went through the convolution layer go through the SPP network. Additionally, they are converted to fixed-sized output data. As the CNN-based YOLO algorithm could only use fixed-sized images as input data, the learning-data categories became narrower, or the original images had to be modified or cut. This was attributed to the fact that each neuron had a fully connected structure in the CNN structure. SPPs enable the use of various-sized images for input; thus, they simultaneously break away from this structure and the fixed-sized output.

2.5 PAN

PAN, as a segmentation model, resolves imbalances in feature information through changes in the network structure because low-level features have less influence on the results than high-level features do on the final output layer in the previous CNN-layer structure. This method is referred to as bottom-up path augmentation.

3.1 Deep learning–based image analyzer

The deep learning–based image analyzer detects the locations of drones after receiving input thermal images from the SWIR cameras; it transmits the results and event information to the integrated controller.

The deep-learning model uses the YOLO v4 algorithm described in Chapter 2. The thermal images used for input are divided based on frames and converted to one image per frame. The proposed system divides images based on 30 frames per second.

Figure 6 shows examples of the drone locations on the divided images. (a) shows the image of a drone flying on an overcast day with thick clouds. (b) is a shot of a drone flying on a fine, slightly cloudy day. The blue boxes in each image are labels attached to the images to indicate the locations of the drones that should be detected.

Figure 6:

Images of drones applied to the deep-learning model

The locations of the drones marked in Figure 6 are converted to the same type of vertex coordinate as shown in Figure 7. The numbers in Figure 7 represent the object’s class number, x-coordinate, y-coordinate, width, and height, respectively.

Figure 7:

Example of converted input data

Each value in Figure 7 shows the class numbers of the objects, x- and y-coordinates of the bounding boxes, width, and height in sequence. Each vertex coordinate is considered a bounding box and saved as a text file. Information on the bounding boxes of each image is used as learning data for the deep-learning model (YOLO v4). When the learning is completed, according to the learning algorithm described in Chapter 2, the drone locations are detected on the input images before the results are sent to the integrated controller.

Figure 8 shows an example of how the drones detected through the deep learning–based image analyzer are displayed on the monitor of the integrated controller. The coordinates of the detected drone in Figure 5 are output by the image analyzer with a data type that matches the input data in Figure 7. The data are produced in the order of class numbers, x- and y-coordinates of the detected bounding boxes, width, and height. These data are sent to the integrated controller to display red bounding boxes. The names of the concerned objects are attached to the upper right of the bounding boxes on the monitor images of the integrated controller.

Figure 8:

Examples of output images of detected drones

3.2 Integrated Controller

The integrated controller controls the pan-tilt and SWIR cameras, displays the images input from the SWIR cameras, receives event information, e.g., the locations and movements of recognized drones, from the image analyzer, and displays the information by overlaying it on the images from the SWIR cameras already displayed. The detailed hardware configuration of the integrated controller is shown in Table 1.

Table 1:

Detailed hardware configuration of the integrated controller

The integrated controller does not simply detect drones; it also displays their routes by tracking them. It allows users to easily see the detected drones by tracking their routes, even when they are misrecognized.

4.1 Experimental Environment

This study upgraded the YOLO algorithm version from v3 to v4 by developing an experiment from preceding research [15]. It simultaneously quadrupled the volume of drone images as learning data from 5,734 to 22,921. The increased learning data were shot directly by SWIR cameras. Thirty image frames per second were obtained by flying small drones over 1.5-2 km distances.

A program called Darklabel was used to establish these learning data [16]. The learning data were established by attaching labels, x-coordinates, y-coordinates, width, and height values to all the images obtained through this program. Out of 22,921 images of established learning data, 21,437 images included drones and 1,484 images did not. Seventy percent of the data were established for training and 30% were used for testing. Consequently, 16,121 images were used for training, with 6,800 used for testing.

4.2 Parameters and Deep Learning–Model Structure

The batch size and subdivision values were all set to 64, as hyperparameters used for training, with the epoch number at 61,000. The computing resources used for training are as shown in Table 2 and are the same as those in the preceding study [15].

Table 2:

Computing resources used for training

The YOLO v4–darknet53–conv.137 deep-learning model was used for training. The network structure is as shown in Figure 9.

Figure 9:

YOLO v4–darknet53–conv.137 model structure (architecture)

4.3 Performance Test and Discussion

The deep-learning model was trained and tested using the model shown in Figure 9. Table 3 shows the test results expressed in a confusion matrix.

Table 3:

Confusion matrix of the deep learning–based image analyzer

In Table 3, images including drones are marked as positive (P), with those not including drones marked as negative (N). Thus, TP (true positive) refers to the frequency of accurate predictions of the presence of drones; false positive (FP) is the frequency of predictions of the presence of drones, when no drones are present. Furthermore, FN (false negative) refers to the frequency of predictions of the absence of drones, when drones are actually present, and TN (true negative) is the frequency of accurate predictions of the absence of drones. Table 4 shows the results of the performance test using Table 3.

Table 4:

Performance test of the deep learning–based image analyzer

The precision in Table 4 is the measurement of the accuracy of the predictions of the presence of drones; the precision of the proposed system is 98.17%. This means that its prediction results are mostly accurate. Recall is a measurement of how accurately a system predicts images including drones; the recall of the proposed system is 94.65%. This means that it does not detect about five percent of drones; hence, it has considerable room for improvement.

The F1 measurement (F1 score) is the harmonic mean of the precision and recall; the F1 measurement of the proposed system is 96.38%. The accuracy is a measure of how accurately it predicts all classes or labels, and the accuracy of the proposed system is 93.3%, with room for improvement. The false-alarm rate is the rate of incorrect predictions about images without drones, predicting that they are present. The false-alarm rate of the proposed system was 28.25%, which is very high. The specificity is the rate at which the system accurately predicts images without drones, predicting that drones are not present. The specificity of the system was 71.75%.

In summary, our system still has considerable room for improvement in terms of the false-alarm rate and specificity. It is believed that the proposed system recognizes clouds or other noise as drones because the drones are relatively small in the remotely shot images. It is deemed that additional studies on removing such noise should be conducted in the future.

Furthermore, the proposed system not only detects drones, but also tracks their locations. However, its real-time tracking function was not reflected in the test. In other words, consecutive images were not used as test data; therefore, its location-tracking function was not tested.

4.4 Analysis in Comparison with Preceding Research

This study was a further development of the methodology of a previous study and contains four differences, which are summarized in Table 5.

Table 5:

Differences between preceding research and this study

This study updated the composition of the deep learning–based image analyzer of the preceding research [15] from the YOLO v3 algorithm to YOLO v4, and quadrupled the volume of learning data from 5,734 images in the previous study to 22,921, to enhance the detection accuracy of the image analyzer.

Moreover, the deep-learning model was transplanted to its integrated controller, the embedded hardware, to raise its actual usability. In the previous study, drone detection could only be carried out when shot images were input. However, as the system of this study uses real-time input data shot by SWIR cameras, if the integrated controller can be installed in a given environment, drone detection can be carried out anywhere.

The mAP (mean average precision), the typical performance metric in the image-processing area, was used to compare our system's performance with that of the preceding research. With the preceding research at 98.4% and that of this system at 92.9%, the preceding research was about 5.5% higher. However, this difference seems to be due to randomly sampling the evaluation data from the training data, as mentioned in the previous study. Based on this, in this study, the volume of learning data was quadrupled, and new learning data were made to study the deep-learning model, avoiding the possibility of overfitting, as was pointed out in the preceding research, whose learning data had high similarity.

Author Contributions

Conceptualization, H. M. Park, J. H. Kim, J. K. Kim, and S. D Lee; Hardware, J. K. Kim and S. D. Lee; Software, H. M. Park, G. S. Park, and Y. R. Kim; Validation, H. M. Park, J. H. Kim, J. K. Kim, and S. D Lee; Data Management G. S. Park and Y. R. Kim; Writing-Original Draft Preparation, H. M. Park; Writing-Review & Editing, H. M. Park, J. H. Kim, J. K. Kim, and S. D Lee.