KR102649948B1 - Text augmentation apparatus and method using hierarchy-based word replacement - Google Patents

Abstract

The present invention relates to an apparatus and method for augmenting text data through layer-based word replacement, the apparatus comprising: receiving input text and separating it into a plurality of tokens; selecting at least one candidate token from among the plurality of tokens and classifying them by defining a part of speech for each candidate token; and generating augmented text related to the input text by changing the at least one candidate token into a replacement token that forms a hierarchical relationship according to the corresponding part of speech.

Description

Text data augmentation device and method through hierarchy-based word replacement {TEXT AUGMENTATION APPARATUS AND METHOD USING HIERARCHY-BASED WORD REPLACEMENT}

The present invention relates to data augmentation technology, and more specifically, to an apparatus and method for augmenting text data through layer-based word replacement that provides augmentation of text data by replacing words using layer information among the features of each part of speech of words. It's about.

As we enter the era of artificial intelligence, the demand and supply of unstructured data has increased exponentially, and accordingly, research using deep learning algorithms to analyze various types of unstructured data has been actively conducted. It is becoming. Deep learning algorithms use a neural network model created by stacking multiple layers, and can be accomplished through the process of discovering data characteristics and learning meaningful expressions each time it passes through the layers. These deep learning technologies are widely used in natural language processing, speech recognition, image classification, and object detection.

Recent deep learning technologies tend to expand into research on multimodal deep learning, which simultaneously learns data with different feature dimensions. Unlike single modal learning, which learns data with one feature dimension, multimodal deep learning can improve learning performance by complementary use of various data such as text, images, audio, and video. . In particular, research on multimodal deep learning that handles text and image data together is being actively conducted, and a representative application is 'Text to Image' synthesis. Text to Image synthesis is a technology that outputs appropriate images corresponding to input text, and various research is being conducted based on Generative Adversarial Network (GAN).

ReStGAN is a well-known application example of this Text to Image synthesis technology. ReStGAN is an algorithm developed by Amazon and applied to the clothing search system, and can generate clothing that matches the product description entered by the customer. Text to Image synthesis is receiving a lot of attention as a technology with high potential for use in various fields, but it is quite difficult to create an image that properly reflects the meaning of the text. This is because various words can be used when describing the same image in text, and even the same word can be interpreted with different meanings depending on the context. In other words, the most important key is to well map the characteristics of text data and the characteristics of image data.

In order to map the features of heterogeneous data, Text to Image synthesis basically requires a vast amount of image and text data for learning, and at this time, each image and multiple texts describing the image must be composed of a pair. do. However, since the data consisting of image and text pairs is limited, it is very difficult to secure a sufficient amount of learning data for Text to Image synthesis.

Korean Patent No. 10-1973642 (2019.04.23)

An embodiment of the present invention seeks to provide an apparatus and method for augmenting text data through layer-based word replacement that provides augmentation of text data by replacing words using layer information among the features of each part of speech of a word.

Among embodiments, an apparatus for text data augmentation through layer-based word replacement includes receiving input text and separating it into a plurality of tokens; selecting at least one candidate token from among the plurality of tokens and classifying them by defining a part of speech for each candidate token; and generating augmented text related to the input text by changing the at least one candidate token into a replacement token that forms a hierarchical relationship according to the corresponding part of speech.

The token separator may generate the plurality of tokens by applying a tokenizer to the input text.

The part-of-speech classification unit removes stopwords from a predefined stopword dictionary from among the plurality of tokens and selects n tokens (where n is a natural number) randomly selected from the tokens from which the stopwords have been removed as the at least one candidate token. can be decided.

The part-of-speech classification unit may classify the at least one candidate token into a specific part-of-speech or other parts-of-speech other than the specific part-of-speech based on the corresponding part-of-speech.

The text augmentation unit may define an independent replacement rule for each part of speech and change the at least one candidate token into a replacement token according to the replacement rule according to the corresponding part of speech.

The text enhancer may replace the candidate token with one randomly selected among a plurality of synonyms according to the corresponding part of speech.

The text augmentation unit may selectively perform an operation of replacing candidate tokens classified as the specific part of speech with a hypernym according to the hierarchical relationship and replacing candidate tokens classified as the other parts of speech with a synonym.

If a first hypernym forming a hierarchical relationship with the candidate token classified into the specific part of speech exists and a second hypernym forming a hierarchical relationship with the first hypernym exists, the text augmentation unit binds the candidate token to the first hypernym. and second superordinate words, respectively.

Among embodiments, a method for augmenting text data through layer-based word replacement includes receiving input text and separating it into a plurality of tokens; selecting at least one candidate token from among the plurality of tokens and classifying them by defining a part of speech for each candidate token; and generating augmented text related to the input text by changing the at least one candidate token into a replacement token that forms a hierarchical relationship according to the corresponding part of speech.

The step of defining and classifying the part of speech may include classifying each of the at least one candidate token into a specific part of speech or other parts of speech other than the specific part of speech based on the corresponding part of speech.

The step of generating the augmented text may include defining an independent replacement rule for each part of speech and changing the at least one candidate token into a replacement token according to the replacement rule according to the corresponding part of speech.

The step of generating the augmented text may include selectively performing an operation of replacing candidate tokens classified as the specific part of speech with a hypernym according to the hierarchical relationship and replacing candidate tokens classified as other parts of speech with a synonym. You can.

The disclosed technology can have the following effects. However, since it does not mean that a specific embodiment must include all of the following effects or only the following effects, the scope of rights of the disclosed technology should not be understood as being limited thereby.

The apparatus and method for augmenting text data through layer-based word replacement according to an embodiment of the present invention can provide augmentation of text data by replacing words using layer information among the characteristics of each part of speech possessed by the word.

The text data augmentation device and method through layer-based word replacement according to an embodiment of the present invention contributes to improving the accuracy of multimodal deep learning analysis that handles images and text together by generating more text information for a limited number of images. You can.

1 is a diagram explaining a data enhancement system according to the present invention.
FIG. 2 is a diagram explaining the functional configuration of the data enhancement device of FIG. 1.
Figure 3 is a flowchart explaining a method of text data augmentation through layer-based word replacement according to the present invention.
Figure 4 is a diagram illustrating a text data augmentation process through layer-based word replacement according to the present invention.
Figure 5 is a diagram explaining the process of constructing a final token set from input text.
Figure 6 is a diagram illustrating a specific process of the text data augmentation method through layer-based word replacement according to the present invention.
Figures 7 to 10 are diagrams illustrating experimental results regarding the text data augmentation method through layer-based word replacement according to the present invention.
Figure 11 is a diagram explaining the entire experimental process for performance analysis.
12 and 13 are diagrams explaining performance comparison analysis results.

Since the description of the present invention is only an example for structural or functional explanation, the scope of the present invention should not be construed as limited by the examples described in the text. In other words, since the embodiments can be modified in various ways and can have various forms, the scope of rights of the present invention should be understood to include equivalents that can realize the technical idea. In addition, the purpose or effect presented in the present invention does not mean that a specific embodiment must include all or only such effects, so the scope of the present invention should not be understood as limited thereby.

Meanwhile, the meaning of the terms described in this application should be understood as follows.

Terms such as “first” and “second” are used to distinguish one component from another component, and the scope of rights should not be limited by these terms. For example, a first component may be named a second component, and similarly, the second component may also be named a first component.

When a component is referred to as being “connected” to another component, it should be understood that it may be directly connected to the other component, but that other components may exist in between. On the other hand, when a component is referred to as being “directly connected” to another component, it should be understood that there are no other components in between. Meanwhile, other expressions that describe the relationship between components, such as "between" and "immediately between" or "neighboring" and "directly neighboring" should be interpreted similarly.

Singular expressions should be understood to include plural expressions unless the context clearly indicates otherwise, and terms such as “comprise” or “have” refer to implemented features, numbers, steps, operations, components, parts, or them. It is intended to specify the existence of a combination, and should be understood as not excluding in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

For each step, identification codes (e.g., a, b, c, etc.) are used for convenience of explanation. The identification codes do not explain the order of each step, and each step clearly follows a specific order in context. Unless specified, events may occur differently from the specified order. That is, each step may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the opposite order.

The present invention can be implemented as computer-readable code on a computer-readable recording medium, and the computer-readable recording medium includes all types of recording devices that store data that can be read by a computer system. . Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. Additionally, the computer-readable recording medium can be distributed across computer systems connected to a network, so that computer-readable code can be stored and executed in a distributed manner.

All terms used herein, unless otherwise defined, have the same meaning as commonly understood by a person of ordinary skill in the field to which the present invention pertains. Terms defined in commonly used dictionaries should be interpreted as consistent with the meaning they have in the context of the related technology, and cannot be interpreted as having an ideal or excessively formal meaning unless clearly defined in the present application.

Deep learning can correspond to an algorithm that learns by utilizing a neural network structure with deep hidden layers in an artificial neural network, and includes convolutional neural networks (CNN), recurrent neural networks (RNN), and generative adversarial neural networks. (GAN: Generative Adversarial Network) may be included. Deep learning algorithms can be used in multimodal learning to learn representations of heterogeneous data features, and for example, can be used in Text to Image synthesis, a technology that generates appropriate images corresponding to input text.

Various studies are being conducted on Text to Image synthesis based on GAN, a generation algorithm. GAN is a neural network in which a generator network and a discriminator network compete hostilely to learn. The generator generates fake data similar to the real thing to deceive the discriminator, and the discriminator performs learning to judge real data and generated data. can do.

In 2016, Scott Reed proposed a methodology for generating images from text through a simple model structure using GAN. However, the simple model structure of early GANs may have limitations in that they cannot generate high-resolution images. To solve this limitation, StackGAN can configure two stages by stacking two GANs. The first stage of StackGAN sketches an object corresponding to the input text, and the second stage can generate better, high-resolution images by correcting incorrect parts of the object created in the first stage and adding details.

Meanwhile, AttnGAN can apply an attention mechanism to a structure that stacks multiple GANs to solve the limitation of not being able to utilize detailed information at the word level when text is encoded only as a global sentence vector. . When AttnGAN generates a sub-region image from input text, it can pay more attention to words related to the image and give them a higher weight. In addition, AttnGAN can generate high-resolution images that more accurately reflect the meaning of the text by calculating the matching loss between the generated image and the input text to induce better generator learning.

Data augmentation is a technique that secures a sufficient number of data needed for learning by increasing the number of data through artificial changes. Data augmentation can be widely used, especially to increase the number of image data. Specifically, in addition to traditional methods of increasing the number of image data through simple transformations such as flipping, color space, cropping, rotation, and translation of image data, new data using transformation in the learned feature space, neural transfer, or GAN New data augmentation algorithms that apply deep learning technologies such as generation are also being proposed.

Recently, attempts to utilize data augmentation techniques are increasing not only in image data but also in the field of natural language processing, and a representative study is text data augmentation based on vocabulary replacement. Text data augmentation is a technique that replaces words in a sentence with synonyms and can use a thesaurus or embedding model. Thesaurus-based data augmentation mainly uses relationships between synonyms defined in a tree structure in WordNet, and through this, several sentences with similar content can be generated by replacing some words included in a sentence with synonyms. However, thesaurus-based data augmentation such as WordNet not only requires considerable cost and time to build thesaurus, but may also have limitations in that it cannot process vocabulary not included in the thesaurus. Meanwhile, data augmentation based on an embedding model is a method of finding words with vectors similar to the vectors of words included in sentences through learning about the corpus, and can be implemented through word embedding algorithms such as Word2Vec, Fasttext, and Glove. .

In addition, deep augmentation includes back-translation-based augmentation, which uses machine translation to preserve the meaning of the original sentence while adding differently expressed sentences through paraphrase, and language model-based augmentation, which fine-tunes language models that have undergone large-scale pre-training such as BERT and GPT2. Text data augmentation research using learning technology is being actively conducted.

However, these text data augmentation methods may have limitations in that text augmentation is performed only based on meaningful relationships between words and they do not take into account the hierarchical relationships of words.

Hereinafter, the text data augmentation device and method through layer-based word replacement according to the present invention will be described in more detail with reference to FIGS. 1 to 6.

1 is a diagram explaining a data enhancement system according to the present invention.

Referring to FIG. 1 , the data enhancement system 100 may include a user terminal 110, a data enhancement device 130, and a database 150.

The user terminal 110 may correspond to a computing device that is connected to the data augmentation device 130 to provide text data and receive augmented data according to data augmentation. The user terminal 110 may be implemented as a smartphone, a laptop, or a computer, but is not necessarily limited thereto, and may also be implemented as a variety of devices such as a tablet PC. The user terminal 110 may be connected to the data enhancement device 130 through a network, and a plurality of user terminals 110 may be connected to the data enhancement device 130 at the same time.

The data augmentation device 130 may be implemented as a server corresponding to a computer or program that performs the text data augmentation method through layer-based word replacement according to the present invention. The data enhancement device 130 may be connected to the user terminal 110 through a wired or wireless network and may exchange data with the user terminal 110.

In one embodiment, the data augmentation device 130 may operate in conjunction with various external systems (or servers) in the process of performing the text data augmentation method through layer-based word replacement according to the present invention. For example, the data augmentation device 130 can access various documents consisting of text through SNS services, portal sites, blogs, etc., and collect necessary data in the process of building a learning model necessary for data augmentation. .

The database 150 may correspond to a storage device that stores various information required during the operation of the data enhancement device 130. For example, the database 150 may store information for separating text tokens and tagging parts of speech, and may store learning algorithms and model information for building a learning model, but is not necessarily limited thereto, and the data augmentation device 130 ) can store information collected or processed in various forms in the process of performing the text data augmentation method through layer-based word replacement according to the present invention.

FIG. 2 is a diagram explaining the functional configuration of the data enhancement device of FIG. 1.

Referring to FIG. 2, the data augmentation device 130 can be implemented by executing the text data augmentation method through layer-based word replacement according to the present invention. Specifically, in Figure 4, the data augmentation device 130 tokenizes the input text data, selects n tokens, defines and classifies parts of speech, and uses WordNet to define and classify parts of speech. It can be composed of two stages, Phase 2, which replaces words hierarchically according to their parts of speech.

First, Phase 1 (1) separates the input original text data into tokens using NLTK's tokenizer, and (2) selects n tokens that do not correspond to NLTK's stopwords from the set of separated tokens. Second, (3) may correspond to the step of performing the task of distinguishing nouns from other parts of speech through part-of-speech tagging that defines the parts of speech for the selected words. Phase 2 is through the process of (4) extracting a hypernym from WordNet if the part of speech of the selected tokens is a noun, and synonyms if the part of speech of the selected tokens is other than a noun, and (5) replacing the extracted hypernyms or synonyms with the tokens selected in (2). Finally, this may correspond to the step of creating new text data.

In one embodiment, the data augmentation device 130 may include a token separation unit 210, a part-of-speech classification unit 230, a text augmentation unit 250, and a control unit 270. However, the data enhancement device 130 according to an embodiment of the present invention does not have to include all of the above components at the same time, and depending on each embodiment, some of the above components are omitted, or some of the above components are omitted. Alternatively, it may be implemented by selectively including all of them. Hereinafter, the operation of each component will be described in detail.

The token separator 210 may receive input text and separate it into a plurality of tokens. That is, the token separation unit 210 may perform an operation of tokenizing the input text data and separating it into a plurality of tokens (S410 in FIG. 4). More specifically, text augmentation based on word replacement may require separating text data in the form of sentences into words. Tokenization may correspond to the operation of dividing a corpus, a text data set, into meaningful units, and a token may correspond to an output created through tokenization. The unit of a token may be different depending on the purpose, and here it is used with the same meaning as a word.

In one embodiment, the token separator 210 may generate a plurality of tokens by applying a tokenizer to the input text. For example, the token separator 210 may separate sentences of the input text into a plurality of tokens using a tokenizer provided by NLTK. Thereafter, the tokens separated by the token separation unit 210 are transferred to the part-of-speech classification unit 230 and can be used in the next step of the operation process.

The part-of-speech classification unit 230 may select at least one candidate token from a plurality of tokens and classify them by defining a part-of-speech for each candidate token. That is, the part-of-speech classification unit 230 can perform an operation of selecting n tokens among tokens separated from the input text (S420 in FIG. 4), and a part-of-speech tagging operation that defines the parts of speech of the selected tokens. A part-of-speech classification (or classification) operation that divides nouns from other parts of speech can be performed (S430 in FIG. 4).

In one embodiment, the part-of-speech classification unit 230 removes stopwords from a predefined stopword dictionary from a plurality of tokens and randomly selects n tokens (where n is a natural number) from among the tokens from which the stopwords have been removed. A decision can be made with at least one candidate token. For example, the part-of-speech classification unit 230 may receive separated tokens from the token separation unit 210, and select candidate tokens from among the separated tokens as many as the number of words to be finally replaced. At this time, the NLTK stop word A dictionary may be used.

That is, the part-of-speech classification unit 230 may randomly select n tokens excluding stop words from a predefined stop word dictionary, as shown in FIG. 5 . In the case of Figure 5, it may correspond to an embodiment in which the final token set 530 is constructed from the input text 510 using a tokenizer and a stop word dictionary for input data consisting of an image and a text pair corresponding to the image. In Figure 5, the result of dividing the sentence 'a man standing on a white surfboard holding a long paddle' into tokens and finally selecting three tokens 'man', 'surfboard', and 'holding' is shown. there is. The token selection operation can be performed by selecting n random tokens from among the remaining tokens excluding stop words, and n, the number of tokens, is a hyperparameter and can be directly set by the data augmentation device 130. .

Meanwhile, the part-of-speech classification unit 230 may perform a part-of-speech tagging operation that defines the parts of speech of tokens. Here, this may be a very important operation because the replacement method of the token may vary depending on the part of speech. The accuracy of part-of-speech tags may vary depending on the situation, such as the size of the data or the number of words. For example, in an English corpus, the accuracy of NLTK's part-of-speech tags may be approximately 97%. The part-of-speech classification unit 230 can perform a part-of-speech identification operation using NLTK part-of-speech tags that show a high level of accuracy. In figure (a) of Figure 6, 'man' can be identified as 'NN', i.e., a noun, 'surfboard' as 'NN', i.e., a noun, and 'holding' as 'VBG', i.e., a verb. there is.

In one embodiment, the part-of-speech classification unit 230 may classify at least one candidate token into a specific part-of-speech or other parts-of-speech other than the specific part-of-speech based on the corresponding part-of-speech. For example, tokens with identified parts of speech may be classified into two categories based on the part of speech, one of which may correspond to a noun category and the other to a non-noun part of speech category. That is, here, a specific part of speech is a noun, and other parts of speech may correspond to parts of speech other than nouns, but of course, the part of speech is not necessarily limited thereto. Additionally, the scope of the noun can be set to 'NN' as a singular noun, 'NNS' as a plural noun, 'NNP' as a singular proper noun, and 'NNPS' as a plural proper noun in NLTK's part-of-speech tag list. In figure (a) of Figure 6, 'man' and 'surfboard' among the three tokens can be classified as 'Noun', which is a noun, and the other tokens can be classified as 'Others', which represents parts of speech other than nouns.

The text augmentation unit 250 may generate augmented text for the input text by changing at least one candidate token into a replacement token that forms a hierarchical relationship according to the corresponding part of speech. In one embodiment, the text enhancer 250 may replace the candidate token with one randomly selected among a plurality of synonyms according to the corresponding part of speech. In one embodiment, the text augmentation unit 250 may define independent substitution rules for each part of speech and change at least one candidate token according to the part of speech into a substitution token according to the substitution rule. That is, the text augmentation unit 250 may perform an augmentation operation on text data by replacing words in the original text in different ways depending on the part of speech (S440 and S450 in FIG. 4).

In one embodiment, the text augmentation unit 250 may selectively perform an operation of replacing candidate tokens classified as a specific part of speech with a hypernym according to a hierarchical relationship and replacing candidate tokens classified as other parts of speech with a synonym. For example, for tokens classified as nouns and parts of speech other than nouns, a word substitution operation can be performed based on word association information in WordNet, tokens classified as nouns can be used as hypernyms, and tokens classified as parts of speech other than nouns can be used as synonyms. can be replaced with At this time, there may be a plurality of synonyms, and the text enhancer 250 may select one synonym per token through random extraction. Figure (b) in Figure 6 is a hypothetical result of hierarchical word substitution according to part of speech, where the nouns 'man' and 'surfboard' are changed to the hypernyms 'person' and 'board', respectively, and the non-noun ' It can be seen that 'holding' has been replaced with the synonym 'keeping'.

In one embodiment, the text augmentation unit 250 operates when a first hypernym forming a hierarchical relationship with a candidate token classified into a specific part of speech exists and a second hypernym forming a hierarchical relationship with the first hypernym exists. The candidate token can be replaced with each of the first and second superordinate words. For example, the noun 'desk' can form a hierarchical relationship with the first hypernym 'table', and the first hypernym 'table' can form a hierarchical relationship with the second hypernym 'furniture'. there is. That is, a repetitive hierarchical relationship may be formed between tokens, and if there are a plurality of hypernyms forming a hierarchical relationship with one token, the text enhancer 250 selects the token among the plurality of hypernyms. Any one can be replaced.

In addition, when the hierarchical relationship between superordinate words is repeated, the text augmentation unit 250 can independently replace the corresponding token with a hypernym according to the repetition of the hierarchical relationship, and, if necessary, replace the hierarchical relationship. The number of repetitions can be limited. For example, when the number of repetitions of a replaceable hierarchical relationship is set to 2, if a first hypernym exists for a specific token and a second hypernym exists for the first hypernym, the text enhancer 250 provides the specific token. It can be replaced with 1 hypernym and 2nd hypernym respectively. If there is a third hypernym that forms a hierarchical relationship with the second hypernym, the text enhancer 250 will not perform a substitution operation for the third hypernym as the number of repetitions of the hierarchical relationship exceeds 2. You can.

The control unit 270 can control the overall operation of the data augmentation device 130 and manage the control flow or data flow between the token separation unit 210, the part-of-speech classification unit 230, and the text augmentation unit 250.

Figure 3 is a flowchart explaining a method of text data augmentation through layer-based word replacement according to the present invention.

Referring to FIG. 3, the data enhancement device 130 may receive input text through the token separator 210 (step S310) and separate the input text into a plurality of tokens (step S330).

The data enhancement device 130 may select at least one candidate token from among a plurality of tokens through the part-of-speech classification unit 230 and classify it by defining a part of speech for each candidate token (step S350).

The data augmentation device 130 may change at least one candidate token into a replacement token forming a hierarchical relationship according to the corresponding part of speech through the text augmentation unit 250 (step S370), and input text based on the token replacement result. Augmented text about can be generated (step S390).

Hereinafter, experimental results of applying the method according to the present invention to actual data will be described through FIGS. 7 to 10.

The MSCOCO data set can be used for this experiment. MSCOCO is mainly used in object recognition, segmentation, and image captioning research, and about 330,000 pieces of data are publicly available. Here, five texts assigned per image from the data released in 2014 can be used, and text data augmentation can only be applied to the text assigned to about 80,000 images of training image data, and validation purposes. ) It may not be applied to the text assigned to approximately 40,000 pieces of image data. In addition, the experimental environment can be built based on Python 3.6 and the NLTK package, and AttnGAN implemented based on Pytorch can be used as the Text to Image synthesis model, and the experiment will be conducted by applying various text augmentation methods to the AttnGAN model. You can.

Referring to Figures 7 to 9, the text given to about 80,000 training images in MSCOCO's 2014 data is divided into token units using an NLTK tokenizer, and the result is a set of n tokens to be replaced. You can check. Figure 7 shows five texts assigned to random images in the training data, and Figure 8 shows the results of converting the text to lowercase letters and separating them into tokens using the NLTK tokenizer. Figure 9 (a) shows the result of randomly extracting one token for each sentence that does not correspond to the stopwords provided by NLTK among the separated tokens.

Figure (b) in Figure 9 shows the result of the process corresponding to steps (3) to (4) in Figure 4, that is, distinguishing tokens from nouns and other parts of speech through part-of-speech identification and classification, and selective substitution criteria according to the part of speech. You can display the results of finding a replacement word by applying . As a result of part-of-speech tagging of the tokens to be replaced, 'slim' can be defined as 'JJ', i.e., an adjective, and 'screen', 'monitor', 'keyboard', and 'desk' can be defined as 'NN', i.e., a noun. Words identified as nouns can be replaced with synonyms, and parts of speech other than nouns can be replaced with synonyms. Replacement words are extracted by searching in WordNet, and if there are multiple synonyms or hypernyms, only one can be extracted at random. As a result, 'slim' was extracted as a synonym of 'slender', and 'screen', 'monitor', 'keyboard', and 'desk' were extracted as synonyms of 'display', 'display', 'device', and 'table', respectively. You can see that has been extracted.

Figure 10 shows the result of augmenting each text with a new text by replacing selected tokens with extracted words, where the text marked as 'Origin' is the original text and the text marked as 'Augmentation' is the method according to the present invention. It may correspond to text augmented through . Through this method, the augmented text can be used in the subsequent training process without distorting the meaning of the image.

Hereinafter, the performance analysis results measured by applying the method according to the present invention to actual data will be described through FIGS. 11 to 13.

Figure 11 may represent a diagram explaining the entire experimental process for performance analysis. (A) in Figure 11 is a process of text to image synthesis using text data augmented through the method according to the present invention, and (B) is text augmented through a vocabulary replacement-based augmentation method replacing synonyms. This may correspond to the process of performing Text to Image synthesis using data. Lastly, (C) may correspond to the process of Text to Image synthesis using original text data without augmentation. Ultimately, the number of texts per image used for Text to Image synthesis is (A) 10, (B) 10, and (C) 5, respectively, and learning can be performed using the AttnGAN model. Additionally, the Inception Score can be used to evaluate the quality of images generated from text. The Inception score is an evaluation scale mainly used to evaluate the quality of generated images and is measured based on the quality and diversity of the generated images. A higher Inception score can be interpreted as indicating better performance. The Inception scores for the resulting images generated by applying the three models are shown in Figures 12 and 13, respectively.

Figure 12 may correspond to a comparison graph showing the Inception score in units of 10 epochs for images generated through the three models compared in the experiment. Until the mid-epoch, the Inception scores of the model (B) using the existing text augmentation method and the model (C) without augmentation appear to be somewhat higher than the Inception scores of the method according to the present invention, but at 130 epochs when learning has sufficiently progressed, the It can be seen that the Inception score of the model according to is the highest. The epoch showing the highest Inception score may be different for each model, and the results of comparing the maximum value of the Inception score show that the method according to the present invention may have the highest performance, as shown in FIG.

As a result of the experiment, the performance of the model applying text augmentation may be higher than that of the model using a small amount of original data, and in the case of text augmentation, the text augmentation method based on the semantic hierarchy of words according to the present invention is better than the existing text augmentation method based on synonym substitution. It can show superior performance compared to .

Although the present invention has been described above with reference to preferred embodiments, those skilled in the art may make various modifications and changes to the present invention without departing from the spirit and scope of the present invention as set forth in the claims below. You will understand that you can do it.

100: Data augmentation system
110: User terminal 130: Data augmentation device
150: database
210: token separation unit 230: part-of-speech classification unit
250: text augmentation unit 270: control unit

Claims (12)

a token separator that receives input text and separates it into a plurality of tokens;
a part-of-speech classification unit that selects at least one candidate token from among the plurality of tokens and defines and classifies a part-of-speech for each candidate token; and
Generating augmented text related to the input text by changing the at least one candidate token into a replacement token that forms a hierarchical relationship according to the corresponding part of speech, and forming the hierarchical relationship with the candidate token classified into a specific part of speech A text augmentation unit that replaces the candidate token with one of the plurality of hypernyms when hypernyms exist,
Wherein the text augmentation unit applies a limited number of repetitions of a replaceable hierarchical relationship when the hierarchical relationship is repeated between the plurality of hypernyms. Text data augmentation device through hierarchy-based word replacement.
The method of claim 1, wherein the token separator
A text data augmentation device through layer-based word replacement, characterized in that generating the plurality of tokens by applying a tokenizer to the input text.
The method of claim 1, wherein the part of speech classification unit
Removing stopwords from a predefined stopword dictionary from among the plurality of tokens and determining n tokens (where n is a natural number) randomly selected from among the tokens from which the stopwords were removed as the at least one candidate token. Characterized by a text data augmentation device through layer-based word replacement.
The method of claim 3, wherein the part of speech classification unit
A text data augmentation device through hierarchy-based word substitution, characterized in that classifying the at least one candidate token into a specific part of speech or other parts of speech other than the specific part of speech based on the corresponding part of speech.
The method of claim 1, wherein the text enhancer
A text data augmentation device through hierarchy-based word substitution, characterized in that it defines an independent replacement rule for each part of speech and changes the at least one candidate token into a replacement token according to the replacement rule according to the corresponding part of speech.
The method of claim 1, wherein the text enhancer
A text data augmentation device through hierarchy-based word replacement, characterized in that a candidate token is replaced with one randomly selected from a plurality of synonyms according to the corresponding part of speech.
The method of claim 4, wherein the text enhancer
Text data through hierarchy-based word replacement, characterized in that selectively performing the operation of replacing candidate tokens classified into the specific part of speech with a hypernym according to the hierarchical relationship and replacing candidate tokens classified into the other parts of speech with synonyms. Augmentation device.
delete In a data augmentation method performed in a data augmentation device,
Receiving input text and separating it into a plurality of tokens, through a token separator;
Selecting at least one candidate token from among the plurality of tokens and classifying them by defining a part of speech for each candidate token through a part-of-speech classification unit; and
Through a text augmentation unit, the at least one candidate token is changed into a replacement token that forms a hierarchical relationship according to the corresponding part of speech to generate augmented text related to the input text, and the hierarchical relationship is established with the candidate token classified into a specific part of speech. If there are a plurality of hypernyms forming , replacing the candidate token with one of the plurality of hypernyms,
The step of generating the augmented text includes applying a limited number of repetitions of replaceable hierarchical relationships when the hierarchical relationship is repeated between the plurality of hypernyms. Text data augmentation method.
The method of claim 9, wherein the step of defining and classifying the parts of speech is
A method of augmenting text data through hierarchy-based word replacement, comprising the step of classifying the at least one candidate token into a specific part of speech or other parts of speech other than the specific part of speech based on the corresponding part of speech.
The method of claim 9, wherein generating the augmented text
A text data augmentation method through hierarchy-based word substitution, comprising the step of defining an independent replacement rule for each part of speech and changing the at least one candidate token into a replacement token according to the replacement rule according to the corresponding part of speech.
The method of claim 10, wherein generating the augmented text
Hierarchy-based word replacement, comprising the step of selectively replacing candidate tokens classified into the specific part of speech with a hypernym according to the hierarchical relationship and replacing candidate tokens classified into the other parts of speech with synonyms. Text data augmentation method through.