ComicBERT: A Transformer Model and Pre-training Strategy for Contextual Understanding in Comics

Despite the growing interest in digital comic processing, foundational models tailored for this medium still need to be explored. Existing methods employ multimodal sequential models with cloze-style tasks, but they fall short of achieving human-like understanding. Addressing this gap, we introduce a novel transformer-based architecture, Comicsformer, and a comprehensive framework, ComicBERT, designed to process and understand the complex interplay of visual and textual elements in comics. Our approach utilizes a self-supervised objective, Masked Comic Modeling, inspired by BERT’s [6] masked language modeling objective, to train the foundation model. To fine-tune and validate our models, we adopt existing cloze-style tasks and propose new tasks - such as scene-cloze, which better capture the narrative and contextual intricacies unique to comics. Preliminary experiments indicate that these tasks enhance the model’s predictive accuracy and may provide new tools for comic creators, aiding in character dialogue generation and panel sequencing. Ultimately, ComicBERT aims to serve as a universal comic processor.

1. 1.

   https://github.com/gsoykan/comicbert.

2. 2.

   https://numpy.org/doc/stable/reference/generated/numpy.lexsort.html.

3. 3.

   https://docs.opencv.org/4.x/d9/d61/tutorial_py_morphological_ops.html.

This project is supported by Koç University & İş Bank AI Center (KUIS AI). We would like to thank KUIS AI for their support.

Authors and Affiliations

1. Computer Engineering Department, Koç University, Istanbul, Turkey

   Gürkan Soykan, Deniz Yuret & Tevfik Metin Sezgin

2. KUIS AI Center, Istanbul, Turkey

   Gürkan Soykan, Deniz Yuret & Tevfik Metin Sezgin

Corresponding author

Correspondence to Gürkan Soykan .

Editors and Affiliations

1. Nantes Université, Nantes, France

   Harold Mouchère

2. Wuhan University of Technology, Wuhan, China

   Anna Zhu

Disclosure of Interests

The authors have no competing interests to declare that are relevant to the content of this article.

A Data Preparation Details

1.1 A.1 Handling Characters and Associating with Speech Bubbles

First, we performed pairing for bodies and faces. The algorithm used for pairing works as follows: we calculate the intersection rate of faces within bodies. If it is higher than 20%, they are added to the candidate list. From the candidates, we select the one with the highest intersection rate. However, in some cases, there can be multiple faces within a body bounding box. In such situations, the y-coordinate of the faces is used to choose the one higher up in the body bounding box. If there is no intersection, faces or bodies alone can represent the entire character instance.

After extracting character instances, finding their associations with the existing speech bubbles is necessary. The Multi-Task Learning (MTL) model provides relation scores between all speech bubbles and all faces and bodies.

We filtered out the ones with relation scores below 0.2. Then, we successfully attempted to assign relations to character instances based on these scores.

Since we had already determined which panel each component belonged to, we leveraged this information. Therefore, we filtered the relations for speech bubbles, faces, and bodies based on whether they belong to the same panel. As a result, the possibility of a speech bubble being assigned to a character outside of its panel was eliminated.

Next, we checked the relation scores for the components that make up each character (i.e., face and body). We selected the highest score associated with the character instance’s chosen speech bubble.

While associating each speech bubble with a character, initially, each character should be assigned the highest-scoring speech bubble that corresponds to them. Then, any remaining speech bubbles may belong to the same character as their second or third speech bubble.

At this point, we employed the same algorithm again. However, if a speech bubble has already been assigned to a character, we added a penalty of up to 0.35 in subsequent calls for that character’s possible next assignments. The purpose of this penalty is to prevent the model from making incorrect associations, as the likelihood of one of two characters owning both speech bubbles in the same panel is lower than the possibility of them both sharing a single speech bubble.

1.2 A.2 Z-Order Calculation

To simulate the Z-order, we performed the following steps. First, we obtained the union box of the bounding boxes and then partitioned this union box into an \(n \times n\) grid. The center points of the bounding boxes are then snapped to the grid points. Next, each bounding box is considered as a point with integer values for the x and y coordinates on the grid. Then these points are sorted first based on the column values and then based on the row values^{Footnote 2} to calculate the Z-order. The value of \(n\) is 4, which applies to both panels and speech bubbles.

1.3 A.3 Component Association with Panels

Every detected component is assigned to a panel using the following logic: their intersection rate with each panel is calculated, along with their distance based on the center coordinates. Then, the panel with the maximum intersection rate is chosen for assignment. In cases where a bounding box has the same intersection rate with multiple panels, the one with the minimum center distance is selected. If the intersection rate is below 0.2, then that bounding box is left unassigned and is not used for our experiments.

A sample comic page is used to demonstrate the finalized inference process of the MTL model for the data preparation step in the Comicsformer model. The panels are z-ordered, and speech bubbles within a panel are displayed with their order indicated at the bottom left. Characters’ components are colored with the same color, and speech bubble associations are shown with lines originating from their centers.

Full size image

1.4 A.4 OCR and Text Extraction

To extract text from comic books, the speech bubble segmentations from the Multi-Task Learning (MTL) model are utilized. Instead of directly using the text from Comics Text+, which is obtained from speech bubble detections, the decision was made to use segmentations. This choice results in improved text extraction because the image used with Optical Character Recognition (OCR) models represents the pure content of the speech bubble without any background or overlapping speech bubbles, ideally. As a result, the extracted text is of even higher quality compared to Comics Text+, which already outperforms the original COMICS dataset’s texts.

The segmentation mask is extracted for every speech bubble instance if it surpasses the 0.6 score threshold. Subsequently, morphological transformations are applied to the binary mask using the cv2 library^{Footnote 3} to smooth out the edges and fill small gaps. Some examples can be seen in Fig. 4.

The Comic Text+ OCR models use examples from speech bubble segmentations as shown above. Utilizing speech bubble segmentations instead of detections is particularly useful for irregularly shaped speech bubbles.

Full size image

B Analysis Based on Word Frequencies

For a brief textual analysis, we would like to share the word frequency distributions. The most frequent words in both speech and narrative text can be observed in Figs. 6 and 5.

Frequency distribution of the top 20 common words, apart from stopwords and punctuation, in narrative boxes.

Full size image

Frequency distribution of the top 20 common words, apart from stopwords and punctuation, in speech bubbles.

Full size image

C Details of Scoring and Option Embeddings

We explain the process of obtaining option embeddings separately for each task, following the explanation of the scoring process. Considering the method of context embedding extraction, the scoring process for an option is as follows:

Since we perform L2 normalization on both the option and the context vectors, the option logit equals the cosine of the angle between those two vectors:

Given that the cross-entropy loss is used as the loss function and there are three options, which is consistent across all cloze-style tasks, the final scores are calculated as follows:

The process for extracting option embeddings for each task is as follows:

• Text-Cloze: The final panel is encoded as Comicsformer input and fed to the Comicsformer. The output feature at the position of the speech bubble, which is the third position (assuming the final panels should only have a single speech bubble text for the text cloze), is obtained and processed with the output projector, which has the same architecture as the context projector, i.e., a linear layer followed by L2 normalization. Depending on the model type, encoded panel features are masked; for instance, if it is the panel-only (image-only) model, then character input features are masked in the input.

• Visual-Cloze: Similar to Text-Cloze, but with textual (speech bubble, narrative) input features for the final panel masked. Since the task requires excluding text information from the final panel apart from the visual features, when character modality is involved, it is not masked in the input. However, only the output feature of the panel token is used from the outputs.

• Scene-Cloze: There are no restrictions for Scene-Cloze, thus all modalities of the model type are used. Similar to context extraction, the mean pooling operation is applied to get the final output features before the output projector.

• Character Coherence: Similar to Scene-Cloze, all available modality information is used in the input of the Comicsformer. However, only the character and speech output features are used during the mean pooling operation. The options construction differs from the other tasks since they use information from other panels. The correct options use panel information as is, whereas for the wrong option, the speech input token feature positions are swapped between the first and second.

• Contextual Character to Speech Attribution: This task differs from the others as it does not use the Comicsformer architecture for options. Instead, context features are fused with the character projections coming from pre-trained models. Their similarity with speech projections is measured after both projections are processed through a linear layer and L2 normalization.

Proper masking is applied for each model type. Based on the modality available during the context and option feature extraction, masking was applied accordingly. The model types are as follows:

• Text-only: Uses only text modality, speech, and narrative.

• Char-only: Uses only character modality.

• Image-only (Panel-only): In the original task definition, it was called Image-only when the model uses only panel visual features. However, we used character information based on the characters’ images. Hence, the modal is changed to Panel-only.

• Image-Text (Panel-Text): Uses panel visual information and text modality.

• Panel-Char: Uses panel visual information and character visual features.

• Char-Text: Uses character visual features with text features.

• Panel-Char-Text: Uses all available information from the three modalities.

D ComicBERT Pretraining Results

Table 5 present the pretraining results.

E No-Context (NC) Models

The No-Context (NC) model is a model type proposed by Iyyer et al. [10] to measure the impact of context. The NC model has only been employed in the Panel-Text (Image-Text) model. It can be regarded as an ablation study specific to context. Instead of context, the visual features of the final panel \(p_i\) are used in place of context for this model.

F Masked Comic Modeling Algorithm

The main learning algorithm for Masked Comic Modeling(MCM) is detailed in Algorithm 1.

. Main learning algorithm of Masked Comic Modeling (MCM)

Full size image

G Obtaining Sequential Context Embedding from Comicsformer

Figure 7 illustrates the process of context extraction from Comicsformer.

Obtaining the context embedding with Comicsformer from comic sequences. The transformer encoder outputs at the core of Comicsformer are fused using mean pooling. During mean pooling, the outputs corresponding to padding tokens are not considered. The resulting mean-pooled output is passed through a linear layer (in the experiments, the input-output dimensions are 384\(\,\times \,\)256). Finally, L2 normalization is applied. L2 normalization lets us directly obtain the cosine similarity since downstream tasks use dot product.

Full size image

H ComicBERT Pre-Training Details

During the ComicBERT pre-training phase, we employed two distinct settings, namely slow and fast, based on the duration of training. The fast setting was trained for 50 epochs and utilized a smaller dataset. This training was completed within approximately 24 h on an NVIDIA V100 GPU. In contrast, the slow setting took nearly a week on the same GPU for training. A substantial difference becomes evident when observing the pre-training outcomes for both settings (see Appendix D for pretraining results). However, this contrast is not equally reflected in the results of the cloze-style tasks. The results are presented in Table 4, which were achieved using ComicBERT trained with the slow setting. Nevertheless, its contribution over the fast setting remains marginal. Sharing the results of these two experimental settings could guide future research involving similar experiments, saving time and serving as a reference.

Reprints and permissions

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

Policies and ethics