Learning typographic style: from discrimination to synthesis

Typography is a ubiquitous art form that affects our understanding, perception and trust in what we read. Thousands of different font-faces have been created with enormous variations in the characters. In this paper, we learn the style of a font by analyzing a small subset of only four letters. From these four letters, we learn two tasks. The first is a discrimination task: given the four letters and a new candidate letter, does the new letter belong to the same font? Second, given the four basis letters, can we generate all of the other letters with the same characteristics as those in the basis set? We use deep neural networks to address both tasks, quantitatively and qualitatively measure the results in a variety of novel manners, and present a thorough investigation of the weaknesses and strengths of the approach. All of the experiments are conducted with publicly available font sets.

1. Wang et al. have studied retrieving fonts found in photographs [55]. Extraction from photographs is not addressed in this paper. However, the underlying task of font retrieval will be presented in the experimental section.

2. The subjective evaluation was conducted by an independent user experience researcher (UER) volunteer not affiliated with this project. The UER was given a paper copy of the input letters, the generated letters and the actual letter. The UER was asked to evaluate the ‘R’ along the 3 dimensions listed above. Additionally, for control, the UER was also given examples (not shown here) which included real ‘R’s in order to minimize bias. The UER was not paid for this experiment.

3. Other version of multi-task learning can incorporate different error metrics or a larger diversity of tasks. In this case, the multiple tasks are closely related, though still provide the benefit of task transfer.

4. For completeness, we also analyzed the ‘R’s generated by the one-letter-at-a-time networks. They had similar performance (when measured with D) to the ‘R’ row shown in Table 3, with (6%) higher SSE.

5. This line of inquiry was sparked by discussions with Zhangyang Wang.

6. The substantial processes of segmenting, cleaning, centering and pre-processing the fonts from photographs are beyond the scope of this paper. We solely address the retrieval portion of this task. We do this by assuming the target font can be cleaned and segmented to yield input grayscale images such as used in this study. For a review of character segmentation, please see [10].

Please see Figure 11.

Authors and Affiliations

1. Google, Inc., Mountain View, CA, USA

   Shumeet Baluja

Corresponding author

Correspondence to Shumeet Baluja.

Conflict of interest

The author declares that he has no conflict of interest.

Appendix 1: Font retrieval

In this study, we have concentrated on the task of synthesizing fonts and the assessment of the quality through the use of a discrimination ensemble. An alternate use of the discrimination ensemble is to retrieve fonts that are visually similar to a target font.

The font retrieval task arises when an unlabeled (target) font is encountered and a user/designer wants to identify the font. Manually searching through ten thousand or more candidate font samples is a daunting task. The problem is exacerbated by the fact that an exact match to the font seen may not be publicly available, in which case similar fonts should be returned.

To address this task, recall that the font discrimination ensemble is trained to determine whether a candidate letter is the same font as four input letters, BASQ. Although, in this study, we used the ensemble as a binary classifier, the font discrimination networks and ensemble have real-valued outputs (e.g., they yield a score); this information can be used for font retrieval. Recently, [55] used a neural network approach to find fonts similar to those that may appear in photographs. Here, we perform the analogous task—given a sample of a font (for the networks trained here, the sample includes the letters BASQ), is it possible to search through the entire database of fonts and find the exact, or similar, fonts?^{Footnote 6}

Retrieving similar fonts. In each of the four sets of results, the top row represents the target font. Each of the 10 lines below represents the closest matches from the total set of approximately 12,000 fonts. First (topmost): members of the same font family are retrieved as the top matches. Second: the slant and weight of strokes are matched. Third: heavily stylized fonts—but similar fonts are recovered. Fourth: the thin strokes are present in the retrieved fonts

Full size image

Retrieving similar fonts based solely on \(L_2\) pixel-wise differences of the target letters. In each of the three sets of results, the top row represents the target font. Each of the 10 lines below represents the closest matches (other than the same font) from the total set of approximately 12,000 fonts. First (topmost): successful retrieval. Middle and bottom: similar width strokes are found, but style (e.g., slant, informal nature) is largely different in retrieved fonts

Full size image

Scoring for retrieval works as follows. Like before, the discriminator ensemble takes 5 letters as input. The first four are BASQ from the known font. The fifth letter is the candidate letter to determine whether it is the same font. For this retrieval task, the first four letters are from the target font. For each font in the larger font pool, each font’s letter is, in turn, used as the fifth letter to input into the discriminator ensemble. The similarity of the candidate letter is then recorded. The candidate font’s total similarity to the target font is simply the summed (real-value) outputs of the discriminator ensemble across all the letters in the candidate font. This simple interpretation of the discrimination scores as similarity measurements yields positive results for retrieval; see Figure 12.

Unfortunately, due to the large differences in size and type of the dataset used by [55], the results shown here should not be directly compared. Our candidate retrieval pool is more than \(4\times \) as large and contains a much larger degree of diversity in font styles (many are ‘non-professional’ and are developed for specific artistic purposes, including the aforementioned ransom and picture fonts).

It is interesting to note that simpler methods, such as comparing pixel difference (for example, \(L_2\) pixel-wise difference on each character’s image), also yield reasonable answers on some of the fonts because it captures the overall thickness of the strokes and the size of the characters. However, general stylistic similarities may not be captured; for examples, see Figure 13.

In summary, this appendix is included to show the potential for using the trained networks in a novel way—for font retrieval. Recall that despite the fact that these networks were not trained to measure similarity, preliminary results show promise with no modification to the networks. In the future, training networks to explicitly measure similarity or to create task-specific embeddings to measure distances should be explored [18, 52].

Results from the sharpening network on 10 fonts—positive results and limitations shown. Top of each pair are the letters as output by the synthesis network (and as input to the repair network). Bottom of each pair are the results after the repair network. Row 1: minor cleaned edges (see ‘G,’ ‘M’). Row 2: letters have noise removed in particular, ‘C,’ ‘M,’ ‘F.’ Row 3: the blur has been reduced in almost all of the letters. Row 4: missing stroke connections (see ‘C,’ ‘O,’ ‘S’) are filled in. Row 5: pixels have been pushed away from blurry regions. Row 6: largely unchanged—also included to show one of the rare cases in which a slight degradation is witnessed (‘K’). Row 7: several letters are clearer (‘A,’ ‘W,’ ‘E’); artifacts remain (‘J,’ ‘Y’). Rows 8 and 9: both rows show improvement across many letters; in Row 9, another pass will further clean the font. Row 10: limitations of this approach—output has not significantly changed

Full size image

Appendix 2: Neural repair/sharpening

The most common unwanted artifacts in the synthesized characters are blur and missing connections between portions of the characters when there is an exceptionally thin stroke width. The first problem can be partially alleviated through the use of ‘off-the-shelf’ image sharpening and thresholding tools such as those found in many consumer-level photograph-editing packages. The second problem requires more domain-specific knowledge.

One of the primary aims of the paper was to elucidate the strengths and limitations of a straight-forward deep neural network approach to font synthesis. This should provide a strong baseline to which more sophisticated algorithms can be measured. In this ‘Appendix,’ we briefly introduce one extension to the procedures presented in the main body of the paper that help alleviate the problems noted above.

To address both problems of blur and missing connections, we created a secondary network, termed the repair network. This network takes as input: the inputs to the generation networks and the synthesized outputs from the generation networks. The output of repair network is 26 letters that have, hopefully, removed some of the unwanted artifacts.

The network architecture is exactly the same as the synthesis network, augmented with the additional inputs. As before, the hidden layer is shared between the output letters. The training procedure again employs the same fonts that were used in the main study to ensure that no other external, extra, information is introduced into procedure. To summarize, the training pairs are:

input (30 letters): original (BASQ) + synthesized (A-Z)

target outputs (26 letters): original (A-Z)

Other than the additional inputs, the most salient difference in the training algorithm is that in addition to the glyph-reconstruction \(L_2\) error used in the training of the original synthesis networks, a secondary loss function is added. The outputs (pixels) are penalized in proportion to their distance from either (0.0 or 1.0). The penalty, for output x, where \(0\le x \le 1\), is: \(penalty = 0.5 - |0.5-x|\). This function has the property that the maximum penalty is when the output, x, has a value of 0.5 and no penalty when x is either 0.0 or 1.0. This encourages the outputs to move away from the pixel activations that appear as blur. Note that because the reconstruction error is also present, adding this extra penalty does not necessarily drive the outputs to a purely binary values.

Ten sets of outputs from this repair network are shown in Fig. 14. Figure 14 includes results that demonstrate both successful repairs and repairs that had no effect. Several of the fonts have also been shown earlier in the main body of this paper to highlight differences that the repair network can make. Additionally, we found a rare case in which there was a degradation in a glyph.

The results are promising; many unwanted artifacts have been successfully removed. However, as shown in row 10, this particular sharpener is not yet a panacea for all the repairs needed for the output of the letter generation networks. In many glyphs, there was little effect on the outputs. Importantly, however, in the vast majority of trials, the results were not visually degraded, only improved or unchanged. In the future, exploration of this additional mechanism is open for further research along at least two avenues. The first includes integrating this error metric directly into the synthesis networks so that a separate network is not required. The second is developing ‘repair’ networks that are task/domain specific and that can be applied to other image generation tasks—perhaps even when the image generation itself is not done with a neural network.

Reprints and permissions