KR102279772B1 - Method and Apparatus for Generating Videos with The Arrow of Time - Google Patents

Abstract

Disclosed are an image generation method considering time characteristics and an apparatus therefor. An image generation learning method considering time characteristics in accordance with an embodiment of the present invention comprises: a vector acquisition step of acquiring a latent vector based on a predetermined input value; a generation processing step of generating virtual image data for a virtual image by inputting the latent vector; an image acquisition step of acquiring real image data for a real image; a conversion step of inverting temporal characteristics of each of the virtual image data and the real image data to generate reverse virtual image data and reverse real image data; and a discrimination processing step of comparing at least two data among the virtual image data, the real image data, the reverse virtual image data, and the reverse real image data, and the processing classification for each of image authenticity and moving direction of the image so that image generation is performed.

Description

A method for generating an image considering the characteristics of time and an apparatus therefor {Method and Apparatus for Generating Videos with The Arrow of Time}

The present invention relates to a method for generating an image in consideration of the characteristic of time and an apparatus therefor.

The content described in this section merely provides background information on the embodiments of the present invention and does not constitute the prior art.

Machine learning is divided into supervised learning, unsupervised learning, and reinforcement learning. Unlike supervised learning, unsupervised learning learns data that is not given a target value and mainly performs tasks such as dimensionality reduction, clustering, and feature extraction. Autoencoder, GAN (generative adversarial networks), RBM (restricted Boltzmann machine) ), etc.

Supervised learning took up a large part in the deep learning field, but there was a limit to its activities due to the need to obtain a target. On the other hand, unsupervised learning has been widely used centered on GAN of Goodfellow (2014), and has been applied in various fields including video (video), image and audio fields until now.

GAN is a neural network that learns by opposing generator and discriminator. The goal of GAN is that the generator makes an effort to create data like the real data, and the discriminator tries to distinguish the real data from the generated data, and finally, the generator makes the data like the real data. Early GANs were accompanied by problems such as instability of learning and mode-collapse. Many derived GAN models to improve this have been developed, and the Inception score (IS) and Frechet Inception distance (FID) using the Inception model have been developed to improve the performance of the GAN. could be evaluated.

Existing studies on image (video) generation have been conducted based on only the GAN learning method without considering the temporal characteristics of the image. Since an image (video) has a dimension that distinguishes time, GAN learning for images is more sophisticated than GAN learning for images. Although a method of designing a dedicated architecture considering time is being studied, the generated image is still indistinguishable from the actual image.

From vanilla GAN to StyleGAN, learning for image generation has been rigorously studied, but there are difficulties in video production due to additional characteristics (temporal characteristics).

Developed for image generation, VGAN consists of a generator and a discriminator with time-considered 3D convolution like other dimensions. After that, TGAN and MoCoGAN introduced temporal latent variables as main components for modeling temporal coherence during generation. However, the image generated by such a GAN method is less sophisticated than the generated image.

Accordingly, there is a need for a method for precisely generating an image through a model for generating an image in consideration of the characteristic of time (AoT: Arrow of Time).

The present invention provides an image generation method and an apparatus therefor in consideration of temporal characteristics through learning to process classification of reversely converted images by inverting temporal characteristics of real images and virtual images, as well as real images and generated virtual images. Its main purpose is to provide

According to an aspect of the present invention, an image generation and learning method in consideration of time characteristics performed by a computing device including one or more processors and a memory for storing one or more programs executed by the processor for achieving the above object, a vector obtaining step of obtaining a latent vector based on a predetermined input value; a generation processing step of generating virtual image data for a virtual image by inputting the latent vector; an image acquisition step of acquiring real image data for the real image; a conversion step of inverting temporal characteristics of the virtual image data and the real image data to generate reverse virtual image data and reverse real image data; and comparing at least two data among the virtual image data, the real image data, the reverse virtual image data, and the reverse real image data to process classification for each of the image authenticity and the moving direction of the image to generate an image. Differential processing steps can be performed to ensure that

Further, according to another aspect of the present invention, an image generating apparatus for achieving the above object is an apparatus for generating an image in consideration of temporal characteristics, comprising: one or more processors; and a memory storing one or more programs executed by the processor, wherein when the programs are executed by the one or more processors, the one or more processors acquires a latent vector based on a predetermined input value. ; a generation processing step of generating virtual image data for a virtual image by inputting the latent vector; an image acquisition step of acquiring real image data for the real image; a conversion step of inverting temporal characteristics of the virtual image data and the real image data to generate reverse virtual image data and reverse real image data; and comparing at least two data among the virtual image data, the real image data, the reverse virtual image data, and the reverse real image data to process classification for each of the image authenticity and the moving direction of the image to generate an image. It is possible to perform operations including a differential processing step to make it possible.

In addition, according to another aspect of the present invention, an image generating method performed by a computing device including one or more processors for achieving the above object and a memory for storing one or more programs executed by the processor, an input vector A first learning result obtained by receiving an input, extracting a feature value of the input vector, and learning whether a virtual image is authentic or not by comparing feature values of the virtual image data and the real image data, and the features of the virtual image data and the reverse virtual image data A new image may be generated by applying the second learning result obtained by learning the progress direction of the virtual image by comparing the values, and the generated new image may be output.

As described above, the present invention has the effect of enabling realistic image generation in consideration of temporal characteristics.

In addition, the present invention has the effect of automatically constructing various data sets by utilizing self-supervision.

In addition, the present invention has an effect that can lead to more efficient performance in learning while reducing human labor as the correct answer sheet is automatically generated without the need to manually modify the dataset.

1 is a block diagram schematically showing an image generating apparatus according to an embodiment of the present invention.
2 is a block diagram schematically illustrating an operation configuration for learning of a processor according to an embodiment of the present invention.
3 is a flowchart illustrating a learning method for image generation according to an embodiment of the present invention.
4 is a block diagram schematically illustrating an operation configuration for image generation of a processor according to an embodiment of the present invention.
5 is a flowchart illustrating an image generating method according to an embodiment of the present invention.
6 is an exemplary view for explaining a learning operation of the image generating apparatus according to the first embodiment of the present invention.
7 is an exemplary diagram for explaining a learning operation of the image generating apparatus according to the second embodiment of the present invention.
8A to 8C are diagrams illustrating a learning result and an application result of the image generating apparatus according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, preferred embodiments of the present invention will be described below, but the technical spirit of the present invention is not limited thereto and may be variously implemented by those skilled in the art without being limited thereto. Hereinafter, an image generating method and apparatus therefor in consideration of the characteristic of time proposed by the present invention will be described in detail with reference to the drawings.

1 is a block diagram schematically showing an image generating apparatus according to an embodiment of the present invention.

The image generating apparatus 100 according to the present embodiment includes an input unit 110 , an output unit 120 , a processor 130 , a memory 140 , and a database 150 . The image generating apparatus 100 of FIG. 1 is according to an embodiment, and not all blocks shown in FIG. 1 are essential components, and in another embodiment, some blocks included in the image generating apparatus 100 are added or changed. Or it can be deleted. Meanwhile, the image generating apparatus 100 may be implemented as a computing device, and each component included in the image generating apparatus 100 may be implemented as a separate software device or as a separate hardware device combined with software. can

The image generating apparatus 100 receives a latent vector as an input, generates virtual image data for a virtual image through a generator using the latent vector as an input, and acquires real image data for a real image, each of virtual image data and real image data Generates reverse virtual image data and reverse real image data by inverting the temporal characteristics of , and compares at least two data among virtual image data, real image data, reverse virtual image data, and reverse real image data through a discriminator that works with the generator Thus, an operation of generating an image is performed by processing the classification for each of the image authenticity and the moving direction of the image.

The input unit 110 means a means for inputting or obtaining a signal or data for performing an image generating operation in the image generating apparatus 100 . The input unit 110 may interwork with the processor 130 to input various types of signals or data, or may obtain signals or data through interworking with an external device and transmit the signals or data to the processor 130 . Here, the input unit 110 may be implemented as a module for inputting an actual image, a latent vector, a random variable, a predetermined input value, and the like, but is not limited thereto.

The output unit 120 may output various information, such as a learning result such as whether an image based on virtual image data is authentic or not and an image progress direction, and an image generation result in conjunction with the processor 130 . The output unit 120 may output various information through a display (not shown) provided in the image generating apparatus 100 , but is not limited thereto, and may perform output in various forms.

The processor 130 performs a function of executing at least one instruction or program included in the memory 140 .

The processor 130 according to this embodiment performs machine learning based on the latent vector or real image data obtained from the input unit 110 or the database 150, and a virtual image that is not pre-learned based on the machine learning result. operation to create

The processor 130 generates virtual image data for a virtual image by inputting the obtained latent vector, and acquires real image data for the real image.

Also, the processor 130 inverts temporal characteristics of the virtual image data and the real image data to generate reverse virtual image data and reverse real image data.

In addition, the processor 130 generates an image by comparing at least two data among virtual image data, real image data, reverse virtual image data, and reverse real image data to process classification of whether an image is authentic or not and a moving direction of the image. let this be done Detailed operations of the processor 130 according to the present embodiment will be described with reference to FIGS. 2 to 5 .

The memory 140 includes at least one instruction or program executable by the processor 130 . The memory 140 includes an operation of generating virtual image data, an operation of converting a moving direction of image data, an operation of processing classification of authenticity with respect to a virtual image, an operation of processing a classification of a moving direction of the virtual image, an image It may include an instruction or a program for an operation of generating and the like.

The database 150 refers to a general data structure implemented in the storage space (hard disk or memory) of a computer system using a database management program (DBMS), and performs data search (extraction), deletion, editing, addition, etc. Relational database management system (RDBMS) such as Oracle, Infomix, Sybase, DB2, Gemston, Orion ), an object-oriented database management system (OODBMS) such as O2, and an XML Native Database such as Excelon, Tamino, Sekaiju, etc. The purpose of an embodiment of the present invention It can be implemented according to the requirements, and has appropriate fields or elements to achieve its function.

The database 150 according to the present embodiment may store data related to image generation or learning for image generation, and may provide pre-stored data related to image generation or learning for image generation.

Data stored in the database 150 include latent vectors, image data (eg, real image data, virtual image data, reverse real image data, reverse virtual image data, etc.), learning results (eg, first learning result, second learning result). , image generation learning results, etc.), image generation results, and the like. The database 140 is described as being implemented in the image generating apparatus 100 , but is not necessarily limited thereto, and may be implemented as a separate data storage device.

2 is a block diagram schematically illustrating an operation configuration for learning of a processor according to an embodiment of the present invention.

The processor 130 included in the image generating apparatus 100 according to the present embodiment generates an image based on machine learning. Here, the machine learning is preferably learning using a generative adversarial network (GAN), but is not necessarily limited thereto.

The processor 130 included in the image generating apparatus 100 generates virtual image data for a virtual image by inputting the obtained latent vector, obtains real image data for the real image, and obtains virtual image data and real image data. Reverse virtual image data and reverse real image data are generated by inverting respective temporal characteristics, and at least two data among virtual image data, real image data, reverse virtual image data, and reverse real image data are compared to determine whether the image is authentic or not. An operation for generating an image is performed by processing the classification for each of the moving directions of an image, and may be mounted on any device that generates an image, or may be linked with software that generates an image.

The processor 130 according to the present embodiment may include a vector acquirer 210 , a generator 220 , an image feature value processor 230 , an image converter 240 , and a discriminator 250 . The processor 130 of FIG. 2 is according to an embodiment, and not all blocks shown in FIG. 2 are essential components, and in other embodiments, some blocks included in the processor 130 may be added, changed, or deleted. have. On the other hand, each component included in the processor 130 may be implemented as a separate software device, or may be implemented as a separate hardware device combined with software.

The vector acquisition unit 210 acquires a latent vector based on a predetermined input value.

The vector acquisition unit 210 acquires a latent vector composed of a preset number of normal distribution values. Here, the latent vector may be composed of a randomly selected normal distribution value or a normal distribution value calculated from each of a preset number of image frames.

The generator 220 generates virtual image data for the virtual image by inputting the latent vector.

The generator 220 generates virtual image data including at least one segment block in which the time generated based on the latent vector is continuous. Here, each of the at least one segment block included in the virtual image data may include a feature value for RGB. Here, it is preferable that the generator 220 generates virtual image data through convolutional neural network (CNN) learning, but is not limited thereto.

The image feature value processing unit 230 obtains virtual image data from the generator 220 and real image data for the real image from an external device.

The image feature value processing unit 230 may receive actual image data including at least one segment block in which time generated based on the actual image is continuous. Here, the number of segment blocks included in the real image data is preferably the same as the number of segment blocks included in the virtual image data.

The image feature value processing unit 230 may acquire real image data for a real image that is not related to the virtual image. Here, the actual image may be an image extracted from a pre-stored database or collected from all or part of an image searched on a network.

The image feature value processing unit 230 transmits the acquired virtual image data and real image data to the discriminator 250 . On the other hand, when the generator 220 directly transmits the virtual image data and the real image data to the discriminator 250 , the image feature value processing unit 230 may be omitted or implemented in a form included in the generator 220 . .

The image converter 240 receives virtual image data and real image data, and inverts temporal characteristics of each of the virtual image data and the real image data to generate reverse virtual image data and reverse real image data.

The image converter 240 according to the present embodiment includes a first image feature value converter 242 and a second image feature value converter 244 .

The first image feature value conversion unit 242 generates the reverse real image data by inverting the processing order based on the temporal characteristics of the actual image data. In detail, the first image feature value converter 242 may generate reverse real image data by converting the order of at least one segment block included in the real image data in a reverse order.

The second image feature value conversion unit 244 generates reverse virtual image data by inverting the processing order based on the temporal characteristics of the virtual image data. Specifically, the second image feature value converter 244 may generate reverse virtual image data by converting the order of at least one segment block included in the virtual image data in a reverse order.

The discriminator 250 performs discrimination processing for generating an image in conjunction with the image feature value processing unit 230 and the image converting unit 240 .

The discriminator 250 compares at least two data among virtual image data, real image data, reverse virtual image data, and reverse real image data to process classification of whether the image is authentic or not and the direction in which the image is moving to generate an image. let it be done

The discriminator 250 according to the present embodiment includes an image feature value extractor 252 , a first discriminator 254 , and a second discriminator 256 .

The image feature value extractor 252 may extract feature values for each of the virtual image data, the real image data, the reverse virtual image data, and the reverse real image data. The image feature value extractor 252 may include at least one convolutional layer shared with the first discriminator 254 and the second discriminator 256 .

The first discriminator 254 and the second discriminator 256 process the classification of each of the image's authenticity and the moving direction based on the feature value passed through at least one convolutional layer shared with each other.

The first discriminator 254 processes the classification of the authenticity of the virtual image by using the feature values of the virtual image data and the real image data. The first discriminator 254 compares the feature values of the virtual image data and the real image data to learn whether the virtual image is authentic or not, and outputs the learned first learning result.

The first discriminator 254 transmits feedback information to the generator 220 based on the first learning result, and repeats the virtual image until the generator 220 generates a virtual image in which the virtual image is classified as a true signal. learn the authenticity of Here, it is preferable that the first discriminator 254 performs learning based on a generative adversarial network (GAN) in order to classify the virtual image data to correspond to the true signal in conjunction with the generator 220 . It is not necessarily limited to this.

The second discriminator 256 processes the classification of the moving direction of the image by using the feature values of the virtual image data and the backward virtual image data. The second discriminator 256 learns the moving direction of the virtual image by comparing the feature values of the virtual image data and the reverse virtual image data, and outputs the learned second learning result.

The second discriminator 256 transmits feedback information to the generator 220 based on the second learning result, and the generator 220 repeatedly generates a virtual image in which the forward virtual image is classified as a true signal. Learn whether an image is authentic or not. Here, it is preferable that the second discriminator 256 perform learning based on a generative adversarial network (GAN) in order to classify the virtual image data to correspond to the true signal in conjunction with the generator 220 . It is not necessarily limited to this.

The discriminator 250 calculates feedback information based on the first learning result and the second learning result, and compares the virtual image data and the real image data in the discriminator 250 using the feedback information to obtain a forward virtual image. It learns whether the virtual image is authentic or not by iterating until it is classified as a true signal.

3 is a flowchart illustrating a learning method for image generation according to an embodiment of the present invention.

The image generating apparatus 100 acquires actual image data based on an actual image ( S310 ). Here, the actual image data may be acquired from an external device, and may include at least one segment block in which time generated based on the actual image is continuous.

The image generating apparatus 100 obtains a latent vector (S320), generates a virtual image based on the latent vector, and generates virtual image data for the generated virtual image (S330). The image generating apparatus 100 generates virtual image data including at least one segment block in which time generated based on the latent vector is continuous. Here, each of the at least one segment block included in the virtual image data may include a feature value for RGB.

The image generating apparatus 100 converts the respective reproduction directions of the virtual image data and the real image data ( S340 ). The image generating apparatus 100 generates reverse virtual image data and reverse real image data by inverting temporal characteristics of each of the virtual image data and the real image data.

The image generating apparatus 100 generates a first learning result through a first discrimination process (S350). The image generating apparatus 100 processes whether the virtual image is authentic or not by using feature values of the virtual image data and the real image data. The image generating apparatus 100 compares the feature values of the virtual image data and the real image data to learn whether the virtual image is authentic or not, and outputs the learned first learning result.

The image generating apparatus 100 generates a second learning result through a second discrimination process (S360). The image generating apparatus 100 processes the classification of the moving direction of the image by using the feature values of the virtual image data and the backward virtual image data. The image generating apparatus 100 compares the feature values of the virtual image data and the reverse virtual image data to learn the moving direction of the virtual image, and outputs the learned second learning result.

Although it is described that each step is sequentially executed in FIG. 3 , it is not necessarily limited thereto. In other words, since it may be applicable to changing and executing the steps described in FIG. 3 or executing one or more steps in parallel, FIG. 3 is not limited to a time-series order.

The image generation learning method according to the present embodiment described in FIG. 3 may be implemented as an application (or program) and recorded in a recording medium readable by a terminal device (or computer). The recording medium in which the application (or program) for implementing the image generation learning method according to the present embodiment is recorded and the terminal device (or computer) can read is any type of recording device in which data that can be read by the computing system is stored. or media.

4 is a block diagram schematically illustrating an operation configuration for image generation of a processor according to an embodiment of the present invention.

The processor 130 included in the image generating apparatus 100 according to the present embodiment includes an input vector obtaining unit 410 , a neural network processing unit 420 , a learning result application unit 430 , an image generating unit 440 , and a result output. part 450 . The processor 130 of FIG. 4 is according to an embodiment, and not all blocks shown in FIG. 4 are essential components, and in other embodiments, some blocks included in the processor 130 may be added, changed, or deleted. have. On the other hand, each component included in the processor 130 may be implemented as a separate software device, or may be implemented as a separate hardware device combined with software.

The input vector acquisition unit 410 acquires an input vector for image generation. Here, the input vector means an input value for generating an image that is not input during learning. Here, the input vector may include a value input at random or a value input by a user's manipulation.

The neural network processing unit 420 performs an operation of extracting a feature value for the obtained input vector. The neural network processing unit 420 may extract a feature value based on Convolutional Neural Networks (CNN) learning. Here, the feature value may include a plurality of image feature values.

The learning result application unit 430 compares the first learning result of learning whether the virtual image is authentic or not by comparing the feature values of the virtual image data and the real image data, and compares the feature values of the virtual image data and the reverse virtual image data to obtain a virtual image The second learning result of learning the progress direction is applied, and the image generator 440 generates a new image based on the applied learning result. Here, the new image means a virtual image.

The result output unit 450 outputs and provides the generated new image. Here, the new image may be provided as a data set for learning such as image recognition and motion recognition.

5 is a flowchart illustrating an image generating method according to an embodiment of the present invention.

The image generating apparatus 100 obtains an input vector for generating an image ( S510 ). Here, the input vector means an input value for generating an image that is not input during learning. Here, the input vector may include a value input at random or a value input by a user's manipulation.

The image generating apparatus 100 extracts a feature value by processing the obtained input vector with a neural network (S520). Here, the image generating apparatus 100 may extract a feature value of an input vector based on convolutional neural network (CNN) learning.

The image generating apparatus 100 applies the pre-learned learning result (S530) to generate a new image (S540).

The image generating apparatus 100 compares the first learning result of learning whether the virtual image is authentic by comparing the feature values of the virtual image data and the real image data, and compares the feature values of the virtual image data and the reverse virtual image data of the virtual image. The second learning result of learning the progress direction is applied, and a new image is generated based on the applied learning result. Here, the new image means a virtual image.

The image generating apparatus 100 outputs and provides the generated new image. Here, the new image may be provided as a data set for learning such as image recognition and motion recognition.

Although it is described that each step is sequentially executed in FIG. 5 , the present invention is not limited thereto. In other words, since it may be applicable to changing and executing the steps described in FIG. 5 or executing one or more steps in parallel, FIG. 5 is not limited to a chronological order.

The image generating method according to the present embodiment described in FIG. 5 may be implemented as an application (or program) and recorded in a recording medium readable by a terminal device (or computer). A recording medium in which an application (or program) for implementing the image generating method according to the present embodiment is recorded and a terminal device (or computer) readable recording medium is any type of recording device in which data that can be read by a computing system is stored or includes media.

6 is an exemplary view for explaining a learning operation of the image generating apparatus according to the first embodiment of the present invention.

The processor 130 of the image generating apparatus 100 according to the first embodiment of the present invention applies an ArrowGAN-based framework to the generator 220 and the discriminator 250 for the characteristics of time (AoT) for images like humans. : It aims to learn the ability to recognize arrows of time.

Hereinafter, the discriminator 250 in the ArrowGAN framework will be described.

The discriminator 250 distinguishes between a real image and a fake image (virtual image), and also distinguishes a forward image and a reverse image.

The discriminator 250 outputs a probability p (real | x) when the input image x is a real image, and a probability p (forward | x) when the input image x is forward.

The discriminator 250 is composed of a shared convolutional layer 252 and two output terminals 254 and 256 . Differential character 250 is, for a given image x, define the equation (1) in order to design a self-directed binary cross-entropy loss and generates a reverse image over the forward ^backward x x ^foward and reverse image of the pair.

Here, A means the AoT set {forward, backward}, P _data means the distribution of the actual image, and p(a | x ^a ) means the Bernoulli distribution of the predicted values for whether the image is forward or backward. do. By minimizing the above objective, the discriminator 250 learns a sense of the characteristic of time using Equation (2).

^{The discriminator 250 maintains the adversarial loss (L adv} ) for learning the probability p(real|x) when it is a real image, and tries to maximize it by distinguishing the real image from the generated virtual image. The complete objective function of the discriminator 250 may be defined as in Equation (3).

where α is a hypervariable to control the importance between the two terms.

As shown in Fig. 6, the present invention trains the discriminator 250 in two ways. The real image and the generated virtual image pass through the shared convolutional layer and the first discriminator 254 generating ^{L SS} _{D .}

In addition, the forward image and the backward image pass through the shared convolutional layer and the second discriminator 256 that generates ^{L adv} _{D .}

Hereinafter, the constructor 220 in the ArrowGAN framework will be described.

The generator 220 learns that a virtual image generated based on an inductive bias advances in a forward direction within time. Accordingly, the generator 220 inputs the generated virtual image G(z) to the discriminator 250 , thereby receiving a loss from the discriminator 250 . Here, the loss may be defined as Equation (4).

where p _z is the standard Gaussian distribution N (0, 1). By minimizing the above object, the generator 220 performs learning to generate a forward image. The generator 220 produces only a forward image. However, the discriminator 250 receives both the forward image and the reverse image obtained by inverting the forward image to make it similar to the actual image.

On the other hand, the generator 220 ^{minimizes the hostile loss (L adv} ) in order to realistically generate a virtual image.

The generator 220 performs learning to generate a realistic and forward-facing image by minimizing the overall purpose of the generator 220 . where β is a hypervariable to control the importance between the two terms.

The ArrowGAN framework according to the present invention can be applied to various commonly used GAN learning methods. For example, the ArrowGAN framework according to the present invention may be applied to video Generative Adversarial Nets (VGAN), Temporal Generative Adversarial Nets (TGAN), MoCoGAN, and the like.

7 is an exemplary diagram for explaining a learning operation of the image generating apparatus according to the second embodiment of the present invention.

The processor 130 of the image generating apparatus 100 according to the second embodiment of the present invention aims to generate an image by applying an ArrowGAN-based framework in a categorical manner. In other words, the processor 130 of the image generating apparatus 100 according to the second embodiment improves performance compared to the categorical MoCoGAN baseline by using the latest technology for image generation, and an ArrowGAN frame for generating a categorical image. use the work

Base lines are created in units of frames and there are two discriminators. One is a discriminator for an image, and the other is a discriminator for a frame.

To close the gap between video generation and image generation, recent and effective techniques, ranging from image generation to categorical MoCo-GANs, can be used.

First, a class label is transmitted to the constructor 220 by extending Conditional Batch Normalization (CBN) 730 .

The conditional batch normalization layer 730 modulates the function activation of the constructor by manipulating parameters for batch normalization to adjust the output image of the class label 720 . Here, the conditional placement normalization layer 730 is inserted into each frame generator 220 for image generation.

Also, the discriminator 250 is replaced by a projection discriminator, which computes an inner product between the second feature vector and the class embedding considered as the class score. In order to naturally generalize the image domain using the 3D convolution 740, the projection discriminator may be modified based on Equation (6).

Here, y is a one-hot vector representing a class label, V is a class embedding matrix, Φ is a feature extracted from the last layer of a discriminator, and ø is a fully connected layer that generates a scalar value.

Next, the present invention utilizes a spectral normalization layer in both the video discriminator and the frame discriminator for stable training. After that, mode search loss is added for stable training and various images in each domain. The difference between the images generated for the two latent vectors given in the settings increases the difference between the videos. Finally, in the present invention, by explicitly enhancing the distance between the generated images over the distance between the latent vectors, the diversity of images can be enhanced and the mode collapse problem can be alleviated. In particular, in the present invention, the minimization of the reciprocal can be performed as in Equation 7.

Here, d _z and d _v represent the distance between the latent vectors and the distance between the generated images, respectively.

The processor 130 of the image generating apparatus 100 according to the second embodiment of the present invention has a hinge version of the adversarial losses, self-supervised losses, and a mode search normalizer ( Train a categorical ArrowGAN with objective functions such as mode-seeking regularizer). The training of the generator and the discriminator is achieved by solving the given max-min problem as shown in Equation (8).

Finally, the overall purpose of the generator and the discriminator is shown in Equation (9).

Here, λ1, λ2, and λ3 may be assumed to be 1, 0.2, and 0.2, respectively. The structure of the categorical ArrowGAN is shown in FIG. 7 . The categorical ArrowGAN generates virtual images (first to fourth virtual images) corresponding to the label information through the input of the class label 720 , and through this, generative adversarial learning can be performed.

8A to 8C are diagrams illustrating a learning result and an application result of the image generating apparatus according to an embodiment of the present invention.

Fig. 8a shows an unselected image set generated in categorical ArrowGAN Figure 8a shows how well Weizmann, UCFsports and UCF-101 qualitatively generate valid videos by class label.

Fig. 8a (a, b) shows four consecutive frames for 8 different classes, and Fig. 8a (c) shows randomly sampled frames from many videos.

8A (a, b) are samples generated in the latent space with only class labels. In addition, the categorical ArrowGAN can generate images not only in a simple data set as shown in Fig. 8a (c) but also in a large data set UCF-101.

Figure 8b shows the effect of applying the ArrowGAN framework to the baseline of general GAN learning.

Figure 8b quantitatively shows the continuous improvement of IS for all baselines and all data sets when ArrowGAN is applied. In the present invention, an auxiliary self-supervised operation is added with minimal changes to the discriminator without modifying the constructor. This means that the ArrorGAN framework can be easily applied to other video-GANs.

8c shows the qualitative results of ArrowGAN. Fig. 8c (a) shows the conventional GAN learning result, and Fig. 8c (b) shows the GAN learning result to which the ArrorGAN framework is applied. In (b) of FIG. 8c , it can be observed that, in particular, improvements are made in details such as four limbs or objects.

The above description is merely illustrative of the technical spirit of the embodiment of the present invention, and those of ordinary skill in the art to which the embodiment of the present invention pertains may modify various modifications and transformation will be possible. Accordingly, the embodiments of the present invention are not intended to limit the technical spirit of the embodiment of the present invention, but to explain, and the scope of the technical spirit of the embodiment of the present invention is not limited by these embodiments. The protection scope of the embodiment of the present invention should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

100: image generating device
110: input unit 120: output unit
130: processor 140: memory
150: database
210: vector acquisition unit 220: constructor
230: image feature value processing unit 240: image conversion unit
250: discriminant

Claims (17)

In the image generation learning method performed by a computing device comprising one or more processors and a memory for storing one or more programs executed by the processor, the computing device comprising:
a vector obtaining step of obtaining a latent vector based on a predetermined input value;
a generation processing step of generating virtual image data for a virtual image by inputting the latent vector;
an image acquisition step of acquiring real image data for the real image;
a conversion step of inverting temporal characteristics of the virtual image data and the real image data to generate reverse virtual image data and reverse real image data; and
By comparing at least two data among the virtual image data, the real image data, the reverse virtual image data, and the reverse real image data, the image is generated by processing the classification of whether the image is authentic or not and the moving direction of the image. Perform a differential processing step to
The generating processing step generates the virtual image data including at least one segment block in which time generated based on the latent vector is continuous, and each of the at least one segment block includes a feature value for RGB. An image generation learning method in consideration of time characteristics, characterized in that. According to claim 1,
The vector acquisition step is
Obtaining the latent vector composed of a preset number of normal distribution values, wherein the latent vector is composed of the randomly selected normal distribution value or the normal distribution value calculated from each of the preset number of image frames An image generation learning method considering temporal characteristics. delete According to claim 1,
The image acquisition step is
The real image data including at least one continuous segment block generated based on the real image is received, and the number of the segment blocks included in the real image data is the number of segments included in the virtual image data. An image generation learning method in consideration of time characteristics, characterized in that the number of blocks is the same. 5. The method of claim 4,
The image acquisition step is
Acquire the real image data for the real image independent of the virtual image, wherein the real image is an image extracted from a pre-stored database or collected from all or a part of an image searched on a network. Considered image generation learning method. 5. The method of claim 4,
The conversion step is
a first conversion step of generating the reverse real image data by inverting a processing order based on a temporal characteristic of the real image data; and
A second transformation step of generating the reverse virtual image data by inverting a processing order based on the temporal characteristics of the virtual image data
An image generation learning method in consideration of the temporal characteristics, characterized in that it comprises a. 7. The method of claim 6,
The first conversion step is
An image generation and learning method in consideration of temporal characteristics, characterized in that the reverse real image data is generated by converting the order of the at least one segment block included in the real image data in a reverse order. 7. The method of claim 6,
The second conversion step is
An image generation and learning method in consideration of temporal characteristics, characterized in that the reverse virtual image data is generated by converting the order of the at least one segment block included in the virtual image data in a reverse order. 5. The method of claim 4,
The discrimination processing step is
a feature value extraction step of extracting a feature value for each of the virtual image data, the real image data, the reverse virtual image data, and the reverse real image data;
a first discrimination step of classifying whether the virtual image is authentic or not by using feature values of the virtual image data and the real image data; and
A second discrimination step of processing classification of the moving direction of an image by using the feature values of the virtual image data and the backward virtual image data
An image generation learning method in consideration of the temporal characteristics, characterized in that it comprises a. 10. The method of claim 9,
The first differentiation step and the second differentiation step include:
An image creation and learning method in consideration of temporal characteristics, characterized in that classification of each of the image's authenticity and the moving direction of the image is processed based on the feature value that has passed through at least one convolutional layer shared with each other. 10. The method of claim 9,
The first discrimination step is
Comparing the feature values of the virtual image data and the real image data, outputting a first learning result obtained by learning whether the virtual image is authentic or not,
In the first discrimination step, feedback information is transmitted to the generation processing step based on the first learning result, and in the generation processing step, the virtual image is repeated until a virtual image corresponding to a true signal is generated. An image creation and learning method in consideration of time characteristics, characterized in that learning whether a virtual image is authentic or not. 10. The method of claim 9,
The first discrimination step is
outputting a first learning result obtained by learning whether the virtual image is authentic or not by comparing the feature values of the virtual image data and the real image data;
The second discrimination step outputs a second learning result obtained by learning the moving direction of the virtual image by comparing the feature values of the virtual image data and the reverse virtual image data,
Feedback information is calculated based on the first learning result and the second learning result, and the virtual image data and the real image data are compared in the discrimination processing step using the feedback information to determine if the virtual image is a true signal. An image creation and learning method in consideration of a temporal characteristic, characterized in that it learns whether the image is authentic or not by iteratively until it corresponds. An apparatus for generating an image in consideration of time characteristics, comprising:
one or more processors; and
a memory storing one or more programs executed by the processor, wherein the programs, when executed by the one or more processors, in the one or more processors;
a vector obtaining step of obtaining a latent vector based on a predetermined input value;
a generation processing step of generating virtual image data for a virtual image by inputting the latent vector;
an image acquisition step of acquiring real image data for the real image;
a conversion step of inverting temporal characteristics of the virtual image data and the real image data to generate reverse virtual image data and reverse real image data; and
By comparing at least two data among the virtual image data, the real image data, the reverse virtual image data, and the reverse real image data, the image is generated by processing the classification of whether the image is authentic or not and the direction of the image to perform operations including a differential processing step of
The generating processing step generates the virtual image data including at least one segment block in which time generated based on the latent vector is continuous, and each of the at least one segment block includes a feature value for RGB. An image generating device, characterized in that. delete 14. The method of claim 13,
The image acquisition step is
The real image data including at least one continuous segment block generated based on the real image is received, and the number of the segment blocks included in the real image data is the number of segments included in the virtual image data. An image generating apparatus, characterized in that the same as the number of blocks. 16. The method of claim 15,
The conversion step is
a first conversion step of generating the reverse real image data by inverting a processing order based on a temporal characteristic of the real image data; and
A second transformation step of generating the reverse virtual image data by inverting a processing order based on the temporal characteristics of the virtual image data
An image generating apparatus comprising a. delete

Images