Description Region Expansion-based relationship-oriented dense image captioning model

2.1 Encoders in Image Captioning

The image captioning model is divided into two steps, namely, image feature vector extraction and sentence generation from the feature vectors. In image captioning, information is generally acquired immediately before the classifier in the form of the image feature vector of the image without using class information from object recognition. Therefore, despite the application of image analysis models, the use of object information from object recognition models in most image captioning models varies with the characteristics of the image captioning model. In such image captioning models with an encoder-decoder structure, the expressive power of the sentence is dependent on the performance of the encoder. Therefore, a precise image captioning model is required to classify the various objects and an accurate object recognition model should be used for the encoder.

Early studies on deep learning have focused on object recognition and resulted in the development of several excellent object recognition models. In particular, the VGGNet [12], ResNet [13], Inception v3, and Inception v4 [14][15] models achieved excellent performance in the ImageNet Large Scale Visual Recognition Competition (IRSVRC), and have been applied as encoders for image captioning. The problem of gradient loss due to the deepening of the CNN layer was solved through the use of the residual block in ResNet, which has been widely applied in image captioning models. The application of object detection models such as RCNN and Yolo, which have relatively complex structures, was limited compared to existing object recognition models because of their low object classification and computational efficiency. Image captioning models that use object detection models capable of detecting even small and diverse objects are being studied. An object detection model such as Yolo [16][17] or Faster R-CNN [18] is typically used as an encoder. Faster-RCNN has been widely used as an encoder because it can offer more precise detection than Yolo and has a relatively easily modified network. However, Faster-RCNN has a disadvantage in that its detection speed is relatively slow compared to Yolo.

2.2 Dense Image Captioning

Unlike conventional image captioning, multiple captions are acquired from a single image in dense image captioning. Therefore, it is necessary to provide information on the various objects to the decoder in the encoder stage. However, because only a single feature vector over the entire image is input to the decoder in existing methods, the trained model can only generate limited captions. To solve this problem, approaches for generating and inputting feature vectors through the separation of object units in the image have been studied.

Johnson et al. [2] applied an object detection model that output the location information of the various objects that existed in the image together so that their feature vectors could be input. In their FCLN model, a feature vector for each object region was extracted by estimating the relative position in the image feature vector based on the positions of the bounding boxes generated by Faster-RCNN. The caption for each object was generated by performing individual captioning on each extract feature vector.

Subsequently, Yang et al. [9] and Li et al. [10] separately analyzed the contextual features in the entire image to obtain the background and context information lost during feature vector extraction. Although this approach could capture the interaction of an object with the surrounding space, it still had the limitation of outputting only local information. This is because only extremely local information was provided to the decoder when only a single object was acquired in the process of extending the feature vector. Therefore, to solve this problem, information on complex object clusters should be provided.

2.3 Evaluation Metrics for Image Captioning

The performance of image captioning technology is commonly evaluated using performance indicators for natural language processing machine translation. The most commonly used performance indicators are the Bi-Lingual Evaluation (BLEU) [20] and Metric for Evaluation of Translation with Explicit Ordering (METEOR) [21].

3.1 Overview of the Proposed Method

Although existing dense image captioning methods can generate captions for all the objects in an image and present detailed information such as the object properties, they can only output very local information. To solve this problem, we propose a DRE-based, relational-oriented dense image captioning model in which clustered objects that integrate high-relevance surrounding objects are extracted. Generally, a sentence is output in dense image captioning through a decoder by cropping the feature vector of the image based on the bounding box of the object generated by the object detection model. In this process, only the minimized object area can be obtained owing to bounding box regression in the object detection model. The surrounding background and related objects are removed during feature vector extraction based on the minimized object area. To solve this problem, the description area is expanded to the union between the detected object areas in the proposed model, and a feature vector that includes the surrounding objects is output. Figure 1 shows the overall system architecture of the proposed model.

Figure 1:

The structure of the proposed image captioning model

The detection network uses a pre-trained Yolo v3 [15] object detection algorithm for region of interest (ROI) prediction. The input image is first converted to 3 × 416 × 416 and input to Darknet53, the backbone of Yolo v3, for feature vector extraction. Darknet53 consists of 52 convolution layers and one average pooling layer. The size of the feature map is adjusted by setting the stride of the convolution layer without using a max-pooling layer. The 3 × 416 × 416 input image passes through DarkNet and is converted into a 1024 × 13 × 13 feature vector. (x₁, y₁, x₂, y₂) bounding boxes are finally output by the subsequent detection layer. Each ROI contains eight predicted quantities, namely, the image index, bounding box information (x₁, y₁, x₂, y₂) , objectness score, confidence score, and classified class. The bounding box information contains a total of four values (x₁, y₁, x₂, y₂) where (x₁, y₁) are the coordinates of the upper left corner of the predicted ROI, and (x₂, y₂) are the coordinates of the lower right corner. The objectness score is the probability that an object exists in the predicted ROI, and the confidence score quantifies the classification probability. The proposed model integrates the surrounding objects using only the coordinates and objectness of the ROIs.

3.2 Detection Network

Because the size of the ROI obtained from the detection network is tailored to the size of the target object, it is difficult to obtain the relationship between different objects with only a single ROI. It is therefore necessary to expand the ROI by combining adjacent bounding boxes. To achieve this, we propose DRE, which expands the ROI by identifying objects with relationships to one another based on their IOUs.

To describe an object in a sentence, the area of interest must include the object and its surrounding environment. Therefore, a margin of 10 % is added to each bounding box obtained by preprocessing to expand the size of the box. Because some bounding boxes obtained during this process may be too small to be expressed as sentences, bounding boxes below a certain size are deleted. Figure 2 illustrates the DRE process. The remaining bounding boxes are first sorted in descending order of their objectness score. The DRE process proceeds in the following sequence, where each step is graphically represented in Figure 2:

Figure 2:

The sequence of Description Region Expansion

• 1. After creating a new set of bounding boxes, the bounding box with the highest score is moved.
• 2. The IOU between the moved and remaining bounding boxes is calculated to understand the relationship between objects.
• 3. Objects with an IOU of 0.05 or higher calculated in the previous process are recognized as related objects and moved to the set of bounding boxes.
• 4. Steps 2 – 4 are repeated until there are no more boxes to be moved.

After the above process is completed, the size of the new bounding box for each set is defined using minimum value of the upper left corner and the maximum value of the lower right corner of the set. That is, the entire set is converted into a bounding box. Therefore, D bounding boxes are reduced to D′ bounding boxes. After the completion of the above process, each set becomes a large bounding box through the combination of bounding boxes included in the newly created set.

3.3 Bilinear Interpolation

The size and ratio of the bounding box output from DRE is adjusted to 3 × 416 × 416 to fit the input size of the detection network. Therefore, it is necessary to adjust the coordinates of the bounding box to match the ratio of the image feature map bounding box.

The image feature map uses the output from the DarkNet53 backbone of the detection network. While the size of the feature map is 1024 × 13 × 13 , the extracted bounding box has the same ratio as the image, which has a size of 416 × 416. A 13 × 13 grid is therefore created, and the coordinates of the bounding box and mapped part cropped and extracted. The size of the feature map is then transformed into 1024 × 7 × 7 using bilinear interpolation.

3.4 Recognition Network

The recognition network is composed of fully connected layers. It receives a 1024 × 7 × 7 feature map as input. The input feature map is flattened into a one-dimensional vector and passes through a fully connected layer, which has a dimension of 512, and then through batch normalization before it is input to the language model.

3.5 Language Model

The language model receives the feature vector and embedded words as the input from the recognition network. The structure of the input x_t and output y_t of the language model are as follows:

The feature vector in Equation (4) is the output of the recognition network and is input only once at time t = −1. S_t in Equation (5) denotes an embedded word, and N the number of embedded words in the sentence. S₀ is the <start> token indicating the beginning of the sentence, and S_N is the <end> token indicating the end of the sentence. In Equation (6), the input x_t is repeatedly input to the LSTM at each time point t to generate the output vector y_t and hidden vector ℎ_t as ℎ_t, y_t = f(ℎ_t−1, x_t) where f represents the LSTM. The size of each embedding vector and the hidden layer is 512. The loss function L(S) of the language model is expressed as

where S_t is the input at each time point, and p_t is the probability distribution for all the words. L(S) is calculated as the sum of the negative log likelihood at each time point. The parameters of the LSTM should be set to minimize L(S).

4.1 Datasets and Preprocessing

The Visual Genome (VG) dataset was used in this study to train and verify the proposed model. The version of the VG dataset used contains a total of 108,000 images and 4.1 million sentences. Each sentence has a relation property that indicates whether the sentence describes one object or a relationship between objects. Here, to investigate the expression of relationships between the target objects in a sentence, only the cases where relational information exists were extracted from the 4.1 million sentences in the dataset. A total of 997937 sentences were used for training after the removal of short sentences consisting of four or fewer words. To optimize the dataset, we converted words with a frequency of five or less into tokens during the preprocessing process. In addition, the beginning and end of each sentence can be learned by adding a <start> token at the beginning of the sentence and an <end> token at the end. The total number of words, including the <unk>, <pad>, <start>, and <end> tokens, was 6655, which was used as the dimension of the decoder embedding and final output layers. The output dimension of the embedding layer for vectorizing sentence information was set to 512. This is the same as the dimension of the last fully connected layer constituting the LSTM and recognition layers.

4.2 Results and Discussion

The ROI is expanded to realize dense image capturing for describing the relationship between objects. The BLEU-1 to BLEU-4 scores, which compare successive identities with sentences in the dataset, were used to evaluate the performance of the proposed model. Table 1 shows the BLEU-1 to BLEU-4 scores of the model. The proposed model achieved a lower BLEU-4 score than the compared models, which seems to imply that it could not generate accurate and rich sentences by focusing only on image information. However, it achieved the highest Meteor score of 0.463 while CAG-NET and FCNL achieved similar scores.

Table 1:

The BLEU-1 to BLEU-4 and METEOR scores achieved by various models on the dataset

Figure 3 shows the sentences and their domains generated by the proposed model. Figure 3(a) shows an image of a cat sitting on a desk containing other objects, such as monitors and books. However, information on the objects other than the cat does not appear in the sentence because of the disappearance of the image feature vectors.

Figure 3:

Samples of captions generated by the model

Figure 3(b) shows a scene of several horses in a field. A total of three description regions were generated, and the sentences describing the horses standing on the field or grazing on grass are well expressed. However, the detection network did not detect the object that looks like a dog on the far right because of its small size. A sentence for this object was therefore not generated.

Figure 3(c) shows a scene of a train on a track. No other objects except for the train are visible in the picture, and only one description region was generated. This sentence also accurately describes the train on the track.

Figure 3(d) shows an image of a parked bicycle and a cat. A total of two description regions were detected. The sentence for the left region accurately expresses information about the bicycle. For the right region, it was confirmed that the output sentence describes the cat in the back and not the bicycle.

Author Contributions

Conceptualization, J. H. Seong; Methodology, Y. J. Shin; Software, Y. J. Shin and S. B. Jeong; Data curation S. B. Jeong; Writing-Original Draft Preparation, Y. J. Shin; Writing-Review & Editing, J. H. Seong and D. H. Seo; Supervision, D. H. Seo.