Toward Attribute-Controlled Fashion Image Captioning

Fashion image captioning has attracted significant attention in the field of computer vision for fashion images. Yang et al. [59] introduce a semantic-guided fashion image captioning model, which utilizes spatial attention mechanism [57], semantic attribute visual features, and a reinforcement learning-based approach to effectively generate descriptive fashion captions with the FACAD. Nguyen et al. [43] experimented using channel and spatial attention features [7] for fashion image captioning. Moratelli et al. propose a transformer-based fashion captioning model that integrates external textual memory through k-nearest neighbor searches, which retrieves fashion contexture information for captioning. These pioneer methods produce static image captions for fashion, lacking the ability to dynamically adjust the generated caption based on individual preferences. In contrast, our proposed method incorporates text-based semantic attribute information to provide additional controllability to the fashion captioning model, which generates preferable fashion captions. Different from our previous work [5] that directly fuse semantic attribute and visual features for controllable fashion image captioning, in this article, we further explore and develop various multimodal feature fusion strategies that effectively integrate the visual and semantic attribute features in the decoding stage for better attribute-controlled fashion image caption generation.

Image captioning is a cross-modal task combining computer vision [4, 18, 24, 55] and natural language processing [48, 53]. Many methods [26, 44, 50, 54, 57] have been proposed since theemergence of deep learning in recent years. Vinyals et al. [50] proposed to use the deep Convolutional Neural Network (CNN) encoder to extract the visual features of the image and utilized the RNN network as the language decoder to generate caption sequences. Subsequently, the spatial attention mechanism [57] was further investigated for its application in image captioning tasks. More recently, the transformer network [48] with a Multi-Head Attention (MHA) mechanism has been introduced to benefit image caption generation. Huang et al. [26] proposed attention on attention architecture, which utilized the gated attention mechanism to measure the relevance between the attention weight and the queries. Cornia et al. [11] developed a meshed-memory transformer to exploit both low-level and high-level contributions of the visual feature for caption generation. Yan et al. [58] integrated a task-adaptive attention module into the transformer-based model, enabling it to identify task-specific clues and reduce misleading information from improper key-value pairs. Li et al. [33] developed a long short-term graph, which effectively captures short-term spatial relationships and long-term transformation dependencies of visual features for image captioning. Guo et al. [22] introduce the normalized attention module, allowing the transformer network to incorporate the geometry structure of the input objects through geometry-aware self-attention. Pan et al. [44] developed the unified X-linear attention block for image captioning. Luo et al. [41] proposed a dual-level collaborative transformer for image captioning to realize the complementary advantages of the region and spatial image features. Moreover, many existing methods [27, 61] incorporated semantic visual features, which allows the captioning model to generate the most relevant attribute words. Jiang et al. [27] built a guiding network to learn an extra guiding vector for caption sentence generation. Deng et al. [13] proposed a syntax-guided hierarchical attention network that incorporates visual features with semantic and syntactic information to improve the interpretability of the captioning model. Li et al. [32] devised the transformer-based framework, which exploits the visual and semantic information simultaneously. Yao et al. [60] proposed to utilize the Graph Convolutional Network (GCN) to integrate semantic and spatial relationships between objects. Yang et al. [21] proposed to integrate semantic priors into the model by exploiting a graph-based representation of both images and sentences. Dong et al. [15] introduced dual-GCN to model the object relationships over a single image and model the relationships over similar images in the dataset for image captioning.

Controllable image captioning generates image descriptions following designated control signals. Deshpande et al. [14] proposed to predict a meaningful summary of the image using the part-of-speech tag and subsequently guiding the caption generation based on that summary. Chen et al. [8] utilized abstract scene graph structure to represent user intention at the fine-grained level and control what and how detailed the generated description should be. Chen et al. [6] developed a controllable image captioning model that uses human-like verb-specific semantic roles as the control signal, considering both event-compatible and sample-suitable requirements. Shuster et al. [47] specified 215 different text-based personality traits and were able to generate the visual caption based on the different traits. Differently, our controllable fashion image captioning method controls both text-based fashion attributes and styles, which allows users to inject the preferred semantic meaning into the fashion description.

We utilize semantic attributes to control and guide the process of caption generation for fashion images. By incorporating semantic attributes, such as fashion attributes and fashion style, we expect that the model can interpret fashion image content with semantic information to impart a higher level of control and allow for a more tailored fashion description generation. Let \(\mathcal{A}=\{a_{1},a_{2},...,a_{n}\}\) be the input semantic attributes, where \(a_{i}\) denotes one text word describing the attribute (e.g., stripe, fuzzy, sleeve, comfort) and \(n\) represents the total number of attributes which can be flexibly determined depending on personal preference. We use an attribute controlling encoder to encode the semantic attributes \(\mathcal{A}\) to feature space as \(\hat{\mathbf{A}}\in\mathbb{R}^{n\times d}\), where \(d\) is the feature dimension. Specifically, the set of semantic attributes \(\mathcal{A}\) are first encoded as word embeddings \(\mathbf{T}=[\mathbf{t}_{1};\mathbf{t}_{2};...;\mathbf{t}_{n}]\in\mathbb{R}^{n \times d}\) and then fed to a single layer transformer encoder block with the MHA module, which can be formulated aswhere \(\mathbf{T}^{q},\mathbf{T}^{k}\), and \(\mathbf{T}^{v}\) are the query, key, and value matrix projected from \(\mathbf{T}\), respectively. \(\mathbf{W}_{o}\) denotes the output projection matrix that aggregates the information from \(h\) parallel attention heads. \(\top\) represents the transpose operation. To ease the presentation, we neglect the Feed-Forward Network and Layer normalization modules in the equation. \(\hat{\mathbf{A}}\) will be injected into the fashion caption decoder and serve as the control signal.

The fashion image encoder is utilized to extract rich and meaningful visual information from fashion images. For the input fashion image \(\mathcal{I}\), we first use a transformer-based image encoder upon ResNet backbone [24] to extract its visual features. Furthermore, we introduce an FEM that attends and enhances the visual representation over channel dimension. Applying an attention mechanism in a channel-wise manner can be viewed as a process of selecting critical visual semantic representations, which further improve the feature representations and contribute to better fashion caption prediction.

Specifically, we use the pre-trained CNN-based network to extract visual features and reshape to \(\mathbf{X}\in\mathbb{R}^{N\times D}\), where \(N=H\times W\) are spatial size, \(H\) and \(W\) is the height and width of the visual features, and \(D\) denotes the channel dimension. The visual features \(\mathbf{X}\) are then fed into the Transformer encoder to obtain the attended visual representations \(\mathbf{X}^{a}\)

In this way, the visual encoder can capture long-range contextual information over spatial dimensions. In addition, the encoder further enhances the visual representation by capturing the long-range contextual information over channel dimensions. To achieve this, we introduce an FEM to aggregate the context from the channel dimension \(D\). Concretely, the FEM first reshapes the extracted visual features from the backbone network with the dimension of \(H\times W\times D\) to \(\mathbf{\bar{X}}\in\mathbb{R}^{D\times N}\). Subsequently, it calculates the channel attention score based on \(\mathbf{\bar{X}}\) using the \(softmax\) function, followed by conducting matrix multiplication to weight the visual features. We present the following steps to capture the channel-level dependencies:

Here, \(\mathbf{E}\in\mathbb{R}^{D\times D}\) is a channel attention weight. \(\mathbf{X}^{e}\) is reshaped to the same dimension as \(\mathbf{X}^{a}\) and they are concatenated to obtain the final visual representations \(\hat{\mathbf{X}}=\mathbf{X}^{a}\oplus\mathbf{X}^{e}\in\mathbb{R}^{N\times D}\), where \(\oplus\) denotes the concatenation operator. \(\hat{\mathbf{X}}\) will be input into the fashion caption decoder for fashion description generation.

The main goal of our decoder is to predict the fashion caption based on the visual features \(\hat{\mathbf{X}}\) and the semantic attribute features \(\hat{\mathbf{A}}\). One possible solution is to apply the transformer decoder [48]. Suppose we are given \(t\) caption words, which are either predicted at inference or given at training. We can encode them as word embeddings \(\mathbf{Y}=[\mathbf{y}_{1};\mathbf{y}_{2};\dots;\mathbf{y}_{t}]\in\mathbb{R}^{ T\times d}\) and apply an MHA module to obtain the sentence representations \(\mathbf{U}\in\mathbb{R}^{T\times d}\), i.e., \(\mathbf{U}={\textrm{MHA}}(\mathbf{Y}^{q},\mathbf{Y}^{k},\mathbf{Y}^{v})\). The MHA module captures the long-term dependencies among the caption words. Moreover, we observe that the modeling local structure of the words is also beneficial for our fashion image captioning task (e.g., as shown in Table 3). We propose incorporating a localLSTM with MHA to model both local structures and global dependencies information over the caption words.

Specifically, we segment the long sequence of word embeddings, denoted as \(\mathbf{Y}\), into multiple shorter sequences based on a predefined window size of \(M=3\) (as illustrated in Figure 3). These short sequences of word embeddings are then processed by the shared Long Short-Term Memory (LSTM) network (LocalLSTM) independently and output a sequence of hidden representations \(\mathbf{H}=[\mathbf{h}_{1};\mathbf{h}_{2};\dots;\mathbf{h}_{t}]\in\mathbb{R}^{ T\times d}\) with the information of local structures, that is,

Then, we use a masked MHA to further capture the long-term global dependencies and obtain the sentence representations as

Next, we predict the fashion caption words based on sentence features \(\mathbf{U}\), visual features \(\hat{\mathbf{X}}\), and semantic attribute features \(\hat{\mathbf{A}}\). Nevertheless, our previous work [5] combined visual features with semantic attribute features through direct fusion or concatenation, which makes it challenging for the decoder to balance both representations for fashion caption generation. Hence, it is essential to investigate and develop an effective feature fusion strategy that seamlessly integrates multimodal features. We explored three feature fusion strategies to combine the visual and semantic attribute features. The evaluation result is shown in Table 5. The three feature fusion strategies include (1) Direct Concatenation (DC) of both features and using Cross-MHA with sentence features \(\mathbf{U}\) to compute fashion caption representations \(\mathbf{C}_{dc}\); (2) Cross Fusion that utilizes two Cross-MHA modules conditioned on visual features \(\hat{\mathbf{X}}\) and semantic attribute features \(\hat{\mathbf{A}}\) to compute the fashion caption representations \(\mathbf{C}_{cf}\); (3) Utilizing a gating mechanism to balance the contribution of multimedia representation and contribute to better fashion caption generation \(\mathbf{C}_{amf}\), which is named Adaptive Multimodal Fusion Module (AMFM) and shown in Figure 4.where [;] and \(\delta\) denote concatenation operation and Sigmoid activation function. \(\mathbf{W}_{cf}\) and \(\mathbf{W}_{g}\) is the projection matrix. The fashion caption representations \(\mathbf{C}\) will be fed into a linear projection layer and a softmax layer for the prediction of caption words. During the inference, we can vary the input semantic attributes to compute the different fashion caption representations and obtain different caption words.

In the following experiments, we used the FACAD170K dataset to conduct the training and testing for fashion image captioning. We further evaluate the effectiveness of the proposed method on the conventional MSCOCO [36] and Flickr30K [62] image captioning dataset. It consists of more than 120,000 images, and each image is annotated with five GT captions. We follow the Karpathy dataset split [30], where 113k, 5k, and 5k images are used for training, testing, and validation, respectively. The vocabulary size for the MSCOCO caption dataset is 9,487. The Flickr30K dataset consists of 31,000 images and is annotated with 5 captions per image. Similarly, we used splits from Karpathy, where 29k, 1k, and 1k images are used for training, validation, and testing, respectively. The vocabulary size is 7,000.

We adopt the commonly used automatic evaluation metrics to evaluate the quality of image captions, which includes BLEU\(@\)N (B\(@\)N, N = 1,2,3,4) [45], METEOR (M) [3], ROUGE-L (R) [35], CIDER (C) [49], and SPICE (S) [1]. The BLEU evaluation metric is employed to evaluate the precision accuracy between the candidate and reference sentences, with “N” denoting the n-gram precision between these sentences. METEOR evaluates the probabilities of uni-gram precision and recall, while ROUGE-L measures similarity by calculating the longest common subsequence between two sentences. Both of these two matrices account for sentence fluency that involves the penalty factor. CIDER further measures the semantic representation of the sentence by computing the cosine similarity with term frequency-inverse document frequency values. SPICE measures the effectiveness of the captions in recovering objects, attributes, and relations. In all cases, higher metric scores indicate greater accuracy in the generated captions.

We fine-tune the conv5 layers to extract the fashion image features similar to the existing method [59]. The dimensions of image features, semantic attribute features, and caption features are 2,048, 300, and 512, respectively. The local window size is set to 3. The number of layers for encoder and decoder is set to 2 and 4, with 8 attention heads for the best captioning performance. We use the Adam optimizer to train the proposed model with a batch size of 16. The base learning rate is set to \(1\times 10^{-4}\). The model is trained using the XE loss. We use the beam search strategy, and the beam size is set to 5. At the inference stage, the input semantic attributes are optional, and we assume that given attributes should reasonably match the image if they are the input of the model.

Table 2 compares our proposed method to the existing fashion image captioning method and the conventional image captioning methods trained on the FACAD170K dataset. The column Semantic Attribute (SA) represents whether the model uses semantic attributes for captioning. For all the baselines that are used for comparison, we re-implement their method based on their original paper. We report the results using ResNet-101 as the backbone network. The methods compared include SAT [57], AFIC [59], ETA [32], MVT [63], ASET [66], EDK [56], and MFFT [67]. AFIC [59] is the method proposed for fashion image captioning. Other methods are originally proposed for the conventional image captioning, and we use them as the baselines for the fashion image captioning task. We modify \(\text{MVT}_{fa}\), \(\text{ASET}_{fa}\), \(\text{EDK}_{fa}\) that takes semantic attributes as additional input to train the model and use it for comparison. \(\text{ACC}_{visual}\) is our proposed method that only takes the image as input to generate fashion caption without any semantic attributes, while \(\text{ACC}_{visual+fa}\) uses image and up to 5 semantic attributes for fashion captioning. In Table 2, we can observe that when we only use the image as input, our framework achieves the best performance over all the metrics, and we believe that it’s due to feature enhancement and localLSTM module. Furthermore, our proposed model can significantly improve the performance when it takes paired semantic attributes and images as input. The semantic attributes are beneficial for fashion image captioning tasks, in which we can generate better fashion image captions by incorporating the semantic attributes.

In Table 3, we investigate the effectiveness of various proposed modules in fashion image captioning. \(Baseline\) uses the full transformer-based encoder–decoder architecture and takes fashion images as input only for captioning. \(Baseline+LocalLSTM\) includes the localLSTM module to model the local structure of word embeddings during fashion caption training, while \(Baseline+LocalLSTM+FEM\) further involves the FEM to incorporate more informative visual features from fashion images. We can observe that both LocalLSTM and FEM benefit the fashion image captioning model in generating more accurate fashion captioning, in which \(Baseline+LocalLSTM+FEM\) model improves C and B@4 score of 8.4% and 1%, respectively.

Table 4 measures the effectiveness of the given semantic attributes that benefit the fashion image captioning model. However, it is hard to evaluate the performance of generated controlled fashion captions based on the user’s preferable fashion attributes or fashion styles. Hence, we include semantic attributes multi-label image classifier to model the semantic attributes representations \(Visual+f_{rep}\) similar to the work [66] and assume that the modeled attributes semantic features represent the user’s preferable fashion attributes. Furthermore, we also evaluate the effectiveness of using the paired semantic attributes as input for fashion image captioning. We test the effectiveness of utilizing 1, 3, and 5 (\(Visual+5f_{a}\)) semantic attributes. We can see that the performance of using modeled attributes semantic features \(Visual+f_{rep}\) is close to the \(Visual\) model, meaning that the model performs well when using reasonably matched fashion semantic attributes. In addition, we can observe that the evaluation result improves when the number of given semantic attributes increases. This further proves that the semantic attributes are beneficial for fashion image captioning tasks.

Furthermore, we investigated the three multimodal feature fusion strategies (in Section 3.3) for the fashion captioning model and showed the evaluation result in Table 5. We compared the strategies in Equations (10), (12), and (13) and concluded that the proposed AMFM aids in the fuse visual and attributes features for ACC. The proposed model with AMFM outperforms another two strategies in all evaluation matrices.

Figure 6 shows the examples of fashion image captions generated by our ACC model and the existing methods [59, 63] as well as the GT captions. Our method lets user describe the fashion image according to their preferences. For instance, in example (a), the fashion expert or user can inject the fashion attributes of “logo” and “crewneck” into the model to generate a sentence that focuses more on describing the clothing detail of “iconic logo on this comfy crewneck sweatshirt,” or they may inject the fashion style like “cool” to control the fashion caption generation that describes the fashion image with more stylish feelings of cool as “a cool black sweatshirt keep you comfy and chic while kicking back at home.” Similar to the examples shown in (b), (c), (d), (e), and (f), we can generate charming and engaging fashion captioning by changing the preferred fashion attribute and style words. Especially in (e), the generated caption based on image-only input tends to describe the sneaker with “Italian” style (e.g., “handcrafted italian sneake”), but the user can change the style more toward “French” as “minimal detailing brings maximum versatility to a black french fashion sneaker kicked up a notch” based on their preferences. Besides controlled fashion caption, \(\text{ACC}_{visual}\) and \(\text{ACC}_{visual+f_{a}}\) models generate a more accurate caption that better describes fashion items. The fashion caption generated by existing methods is aligned with the logic of language but less accurate with respect to GT caption. We can see that the word predicted (bold words) in the captions generated by \(\text{ACC}_{visual}\) is more accurate as compared to the GT caption.

Besides fashion image captioning evaluation, we further test the effectiveness of the introduced modules in the captioner on the MSCOCO captioning dataset. Since our proposed ACC model is an end-to-end captioning method, we followed the settings in recent end-to-end transformer-based captioning works [38, 65]. We evaluated the captioning performance by including the LocalLSTM and FEM in the baseline transformer encoder–decoder framework. The model is trained using XE loss. For a fair comparison, the semantic attribute is not included for captioning evaluation on MSCOCO captioning dataset. We use Swin Transformer [39] (less parameter) as a backbone network instead of the Vision Transformer [16] backbone used in [38] to ease the training process. The ablative evaluation is shown in Table 6. We can observe that the introduced model performs better as compared to the baseline model in 1% and 1.7% in B@4 and C scores. The improved results are aligned with the result shown in Table 3 and further prove the effectiveness of the introduced components.

We report the performance comparisons between our proposed method with the state-of-art models trained using self-critical loss (SCST) [46] on the MSCOCO captioning test set (Karpathy test split) in Table 7. The models compared include Up-Down [2], GCN-LSTM [60], RFNet [28], ORT [25], AoANet [26], ETA [32], MVT [63], ASET [66], BCAN [29], CPTR [38], S2TC [65], MFFT [67], and IRM [69]. Up-down explored using bottom-up region features for image captioning with soft attention. GCN-LSTM exploits the pairwise relationship information between region features for image captioning. RFNet and BCAN learn to collaboratively enhance the attention over regional visual features and word features to improve the captioning performance. OTR, AoANet, and MVT utilize transformer-based design for image captioning. OTR introduces geometric attention to the captioner that additionally injects the spatial relationship over regional visual features. AoANet improves the MHA mechanisms to select the relevant features for captioning. MVT incorporates multiple visual features with various backbones as visual inputs. ETA and ASET inject semantic and visual information into transformer-based architecture to boost the captioning performance. CPTR and S2TC studied end-to-end image captioning methods with grid or token-based visual features. MFFT proposed to fuse multi-features, such as spatial and semantic information, to assist the caption generation. IRM introduced a retrieval-based method that retrieves the semantic-related information from sentences for the description generation. Our method achieves superior performance in one of the most important metrics “C” for image captioning tasks that evaluate the sentence accuracy and the semantic precision of the predicted sentences. Furthermore, it can be observed that our proposed method achieves competitive performance in most of the evaluation metrics as compared to other methods. The performance emphasizes the effectiveness of our approach.

We further validate the performance of the proposed method on the Flick30K captioning test set (Karpathy test split) in Table 8. We compared with the recent methods, namely DAN [19], STP [20], IPSG [68], IGCA [51], ASET [66], and IRM [69], which have reported their performance on the Flick30K captioning dataset. DAN introduced a deliberate residual attention network that generates preliminary captions and then refined them for better captions. STP explored a ruminant decoding framework to refine the initial caption generated by the base decoder to produce a more comprehensive and polished result. IPSG proposed to integrate part of speech information into the encoder-decoder framework. IGCA captured global information on the target words, thereby improving the predictions of target words in image captioning. Table 8 shows that our proposed method outperforms other approaches across most evaluation metrics. These improvements highlight the effectiveness of the proposed method.