[ISMT2021] Object detection in high-resolution sonar images

2.1 Object detection in sonar images

Sonar images show the seabed and underwater objects using sound waves. They appear in various forms depending on the method by which they are created; moreover, side-scan and forward-looking sonar are typically used. Because side-scan sonar is primarily used to acquire information from seabed landforms or underwater objects, a wide range of information can be acquired; consequently, the sonar image sizes are large. Figure 1 shows an equipment used for side-scan sonar.

Figure 1:

Equipment for side-scan sonar

Forward-looking sonar is used to avoid forward obstacles in water. Therefore, it acquires information from specific regions, and the objects are included in most of the sonar images. Figure 2 shows an equipment used for forward-looking sonar.

Figure 2:

Equipment for forward-looking sonar

Depending on the equipment used, as shown in Figures 1 and 2, the sonar images created may differ. The image property changes for different images, and the object detection method must be changed to accommodate the changing property.

The aim of object detection using sonar images is to determine locations and recognize objects. Such an object detection approach is used in fields such as seabed detection, fish identification, and tracking [6]-[7]. Conventionally, this type of object detection using sonar images is performed manually by professionals owing to difficulties in analyzing sonar images. Recently, to overcome this problem, three-dimensional sonar images were used, and object detection was performed using deep learning [8]-[10]. In this study, the difficulty in object detection was alleviated using deep learning. Labeled data are required when applying deep learning methods. However, sonar images have a low resolution and are unfamiliar; hence, training data are difficult to build. Therefore, the effort required to build training data must be reduced. Various existing data generation methods have been investigated via these efforts [11]-[12]. However, the previously investigated data generation methods are complicated and unsuitable for object detection, e.g., the faster-RCNN [13] and YOLO [4]. Therefore, a simple data augmentation method for data generation is proposed herein.

Object detection in sonar images is performed on board ships or divers underwater. Hence, the closer the object detection is to real time, the higher is the efficiency. Therefore, to perform object detection in real time, investigations using various sonar images have been performed [14]-[16]. In this study, to perform object detection as closely as possible to real time, YOLO, which shows high performance in real time, was used.

2.2 YOLO

Among the various object detection techniques, YOLO was used in this study. YOLO demonstrates good performance in terms of execution time as it is a one-stage detector that performs two steps simultaneously; furthermore, it is not a two-stage detector that requires the separate prediction and classification of object candidate regions [5].

Figure 3 shows the object detection method using YOLO [5]. YOLO is based on a convolutional neural network; it is a method in which an object is detected via non-max suppression through the network after the size of the input image is changed.

Figure 3:

YOLO system [5]

Figure 4 shows an example of object detection using a basic YOLO network.

Figure 4:

Example of YOLO object detection

The input image is partitioned into S × S grid cells. If the center of a specific object corresponds to the center of the grid cell, then the corresponding grid cell performs object detection [5]. Each grid cell comprises multiple bounding boxes and probabilities of each class, and each bounding box contains the information of each class and a confidence score (CS). The information in a bounding box includes four pieces of information: x-coordinate, y-coordinate, width w, and height h. The CS represents the confidence level of an object in the corresponding bounding box, and is calculated using Equation (1).

Intersection over union (IOU) refers to the cross-overlapping union. The CS in each cell has an IOU value between the value of the predicted box and the correct answer if an object exists, and 0 if it does not exist. Figure 5 shows a diagram for calculating the IOU [17].

Figure 5:

IOU diagram

To calculate the class CS for a bounding box, the conditional probability is multiplied by the CS, as shown in Equation (2).

Subsequently, the size of the tensor is expressed as S × S × (B × 5 + C), where S is the number of grid cells, B the number of bounding boxes, and C the number of classes.

3.1 Object detection system

The sonar-imaging equipment used in this study was a side-scan sonar and forward-looking sonar. Therefore, object detection was performed differently in this study. In addition, the object detection algorithm uses YOLO for real-time searches and achieving high performance, as described in Section 2. Figure 6 illustrates the overall process of the object-detection system.

Figure 6:

Process for object detection system

Object detection is performed in the deep seabed using side-scan sonar. By contrast, object detection near the surface of water or detecting objects floating in water is performed using forward-looking sonar. Detailed descriptions are provided next.

3.2 Segmentation

YOLO resizes an input image based on its nature. This causes small objects to disappear. To overcome this problem, image segmentation was proposed. Segmentation is a method for performing object detection by partitioning a large raw image into several smaller input images. Therefore, it is applied to side-scan sonar data and not forward-looking sonar data. Figure 7 shows an example of image segmentation using side-scan sonar.

Figure 7:

Example of segmentation

When segmentation is applied, the segments overlap because an object may lie on the boundary dividing the segments. Additionally, YOLO is performed in parallel on the segmented images to reduce the execution time. Subsequently, the final object detection result is obtained via a merging process (refer to Section 3.4).

3.3 Multi-YOLO

Because forward-looking sonar does not apply image segmentation, object detection is performed using a single YOLO model. By contrast, side-scan sonar performs object detection by configuring multiple YOLO models. The learned YOLO model is the same for both forward-looking and side-scan sonar. Figure 8 shows an example of multi-YOLO object detection.

Figure 8:

Example of multi-YOLO object detection

3.4 Merging

In image segmentation, because multiple inputs are used, multiple outputs are yielded. Therefore, the results should be merged back into one image similar to the input image. If the object detection results overlap during merging, the largest resulting value is selected as the final value, as in the nonmax suppression method.

Figure 9 shows an example of the merging process. In terms of the object detection result, the red and blue boxes represent the tire and drum, respectively.

Figure 9:

Example of merging

3.5 Labeling

Basic raw data were required to perform data augmentation. Labeling [18] was used to produce raw data. A rectangle was used to assign the position of an object to the raw datum, and the class of the object was selected subsequently. By performing labeling, 2,000 basic side-scan sonar images and 4.000 basic forward-looking sonar images were produced. Based on the prepared raw data, data augmentation was performed (Section 3.6).

3.6 Data augmentation

To use YOLO, high-quality data are required for training. Data augmentation is a method of increasing the amount of data in situations where such data are insufficient [19]. The primary methods of image data augmentation include adding Gaussian noise, color inversion, blur, contrast, inversion, segmentation, and cropping [20]. In this study, data augmentation was performed using Imageaug [21]. Furthermore, methods that are not suitable for sonar images, such as color inversion and blur, were excluded from the study.

Figure 10 shows the data augmentation method. Data augmentation was performed twice to build the first augmented dataset, followed by the second augmented dataset. For the first augmented dataset, horizontal and vertical inversion, 90° rotation, and distortion were applied to the raw data. For the second augmented dataset, the raw data were rotated by 180° and 270°. Ad-ditionally, distortion was added to the data of horizontal and vertical inversions and 90° rotations among the first augmented data.

Figure 10:

Data augmentation method

4.1 Experimental data

Sidescan sonar images were captured in an actual ocean. Figure 11 shows an example of a side-scan sonar image.

Figure 11:

Example of side-scan sonar image

In this study, the side-scan sonar image included six classes of objects (rock, pier, shelter, sandbar, tire, and drum). The side-scan sonar images were large, i.e., they measured 2250 × 898. Moreover, they varied significantly in terms of size from large objects such as piers to small objects such as tires.

Unlike side-scan sonar, forward-looking sonar uses sonar images captured in an artificial water tank rather than an actual ocean. Figure 12 shows an example of forward-looking sonar. The forward-looking sonar included six classes of objects (tires, divers, shelters, ladders, and drums). The forward-looking sonar images measured 334 × 225 pixels and were smaller than the side-scan sonar images, as they were captured in an artificial water tank. The size difference between the objects was insignificant.

Figure 12:

Example of forward-looking sonar image

In this study, 2,000 raw side-scan sonar image data points were augmented to 20,000 data points. Meanwhile, 4,000 raw forward-looking sonar image data points were augmented to 40,000 data points. Here, 10% of the images were used as the test data. Table 1 lists the statistics for the training and testing data.

Table 1:

Statistics of training and test data

4.2 Experimental method

Based on the trained YOLO model, the performance was evaluated using data that were not used for training. The mean average precision (mAP) was used to evaluate the performance. The mAP is a performance evaluation method used in many object detection tasks, and its precision and recall are measured based on the classification shown in Table 2.

Table 2:

Confusion matrix for classification

The equations for precision and recall are shown in Equation (3).

In object detection, true positives (TPs) and false negatives (FPs) are determined based on the IOU values described in Section 2. Figure 13 shows an example of a precision–recall graph based on the precision and recall formulas shown in Equation (3). Average precision (AP) corresponds to the yellow area, and the mAP is the average AP in each class.

Figure 13:

Example of precision–recall graph

4.3 Determining segment size

In image segmentation, the number of segments affects the performance. Therefore, this number was determined experimentally. Figure 14 shows the results of segmentation performances based on the number of equal horizontal and vertical segments (x-axis). For instance, 5 × 3 on the x-axis has a corresponding value on the y-axis, which shows the result of partitioning the width into five equal segments and the height into three equal sections. The last row shows the result of segmentation based on the input size of YOLO.

Figure 14:

Graph showing splitting performance

Figure 14 shows that a better performance was achieved when segmentation was performed on both the horizontal and vertical directions compared with when segmentation was performed on only one of the directions. Moreover, a high value was obtained when segmentation was performed based on the input size of YOLO. An analysis of these results show that the standard for segmentation can be adjusted based on the basic YOLO input size. Based on the experimental results, segmentation was performed by partitioning the width and height into five and three segments, respectively.

4.4 Side-scan sonar results

An experiment was conducted on 2,000 basic images, but the results were unsatisfactory, as shown in Table 3. This low performance was speculated to be caused by insufficient data. Therefore, to expand the data, the first data augmentation was performed by applying the data augmentation method described in Section 3. The first data augmentation step significantly improved the performance. Subsequently, a second data augmentation step was performed. Comparing the results of the first and second data augmentations, the performance of small objects such as tires and drums was lower than that of relatively larger objects such as bedrock, piers, and shelters. It is speculated that this occurred because the objects of interest were smaller than the sonar images for input. Hence, an experiment was performed using image segmentation. Consequently, the detection performance for small objects increased significantly.

Table 3:

Performance of object detection based on side-scan sonar

4.5 Forward-looking sonar results

Table 4 presents the results from 2,000 images, including three objects (tires, divers, and shelters).

Table 4:

Performance of object detection under forward-looking sonar

The number of object classes in the front-looking sonar was less than that in the side-scan sonar, and hence different from the actual environment. Therefore, after an experiment was performed on the basic data, an additional experiment was conducted using 2,000 additional images with three objects (ladder, frame, and drum). To improve the low performance of the basic experiment, the first and second data augmentations were performed in the same manner as the side-scan sonar. Unlike side-scan sonar, because the input images were not large and the objects to be detected were not small compared with the input images, segmentation was not applied in the forward-looking sonar.

4.6 Analysis results

Figure 15 shows the results of a correctly identified tire. Meanwhile, Figure 16 shows the result of incorrectly classifying a frame as a tire.

Figure 15:

Correct result of tire

Figure 16:

Error type 1: (frame incorrectly classified as tire)

Figure 17 shows error type 2, where the correct object is a drum. However, in real object detection, both the drum and shadows that appear behind it are recognized as tires. As shown in Table 4, an error occurs when the location of an object is identified but incorrectly classified, or when noise is recognized as an object. It is speculated that this problem occurs because outlines and shadows naturally occupy most of the sonar images, unlike in optical images. Even when the objects are different, if the outlines and shades are similar at certain angles, the object cannot be recognized or classified correctly. Solving this problem will likely improve the performance.

Figure 17:

Error type 2: (drum shadow incorrectly classified as tire)

4.7 Discussion

Table 5 presents a comparison between this study and other studies pertaining to sonar object recognition.

Table 5:

Comparison between this study and other studies

As shown in Table 5, the performance differs depending on the experimental environment. Nos. 1 and 3 show data released by the Naval Surface Warfare Center (NSWC). The data provided by the NSWC comprises four classes, and the objects are large. In addition, in No. 1, data are augmented via active learning. No. 2 shows data that have been self-constructed in various shapes. No. 4 shows the result of this study. A comparison of the results shows that a simpler object image yielded a higher performance. Moreover, although the same method was not employed in Nos. 1 and 3, it was assumed that they benefitted from data augmentation, based on the experimental results of this study and that the same data were used. Finally, it was inferred that the performance deteriorated as the number of classes increased.

Table 6:

Confusion matrix for objects

Author Contributions

Conceptualization, M. S. Choi and J. H. Kim; Methodology, M. S. Choi; Validation, M. S. Choi and J. H. Kim; Formal Anal-ysis, M. S. Choi; Investigation, Y. S. Oh and S. S. Park; Resources, M. S. Choi; Writing—Original Draft Preparation, M. S. Choi; Writing—Review & Editing, J. H. Kim; Project Administration, J. H. Kim; Funding Acquisition, J. H. Kim.