KR102083721B1 - Stereo Super-ResolutionImaging Method using Deep Convolutional Networks and Apparatus Therefor - Google Patents

Abstract

Disclosed are a binocular-based super resolution imaging method using deep learning and an apparatus thereof. Binocular-based super resolution imaging method according to an embodiment of the present invention comprises the steps of obtaining stereo images; And generating a super resolution image from the stereo images using a learned deep neural network, wherein generating the super resolution image comprises mapping a first predetermined parameter between the stereo images and the high resolution luminance image. Reconstructing a high resolution luminance image of the input stereo images using the learned first deep neural network; And generating a super-resolution image for the reconstructed high-resolution luminance image using a second deep neural network trained on mapping a second predetermined parameter between the high-resolution image and the super-resolution image.

Description

Binocular-based Super Resolution Imaging Method Using Deep Learning and Its Device {Stereo Super-ResolutionImaging Method using Deep Convolutional Networks and Apparatus Therefor}

The present invention relates to a binocular-based super-resolution imaging technique, and more particularly to a binocular-based system capable of improving the resolution of an image in a binocular-based (or stereo) system using deep learning technology. A super-resolution imaging method and apparatus therefor.

With the recent development of mobile photography technology, dual cameras are commonly used in mobile devices. Since dual cameras capture twice the binocular-based image (or stereo image), it should be possible to reconstruct high resolution images using superresolution algorithms rather than single image super-resolution methods. .

Since the existing binocular-based super resolution method cannot improve the resolution of the image more than the recent single image super resolution method, the dual camera advantage on the super resolution problem is not fully utilized.

Existing superresolution algorithms reconstruct high resolution images from low resolution source images obtained from jittered subsampling with subpixel precision. Here, multiple shots and video frames of the same resolution are used as inputs.

However, since stereo images are not the same due to parallax, it is not easy to combine stereo pairs to obtain super resolution images. In more detail, the conventional super resolution algorithm is obtained because the depth (or depth information) obtained from the depth-from-stereo (DFS) algorithm can be used to register the pixels of the left camera at the corresponding pixel positions of the right camera. Should be able to apply to dual camera settings. However, the obtained mismatch accuracy obtained from binocular based imaging has a problem that is significantly lower than the sub pixel level precision required for super resolution reconstruction.

Recent superresolution methods have attempted to implement high resolution images from self-patches in the source image using a single camera input, the image itself using joint learning, convolutional neural networks (CNN), and the like. However, these methods cannot be applied directly to stereo pairs because of parallax.

Accordingly, there is a need for a binocular based super resolution imaging method capable of improving the resolution of an image in a binocular based system.

Embodiments of the present invention provide a binocular-based super resolution imaging method and apparatus capable of improving the resolution of an image in a binocular-based (or stereo) system using deep learning technology.

Binocular-based super resolution imaging method according to an embodiment of the present invention comprises the steps of obtaining stereo images; And generating a super resolution image from the stereo images using the learned deep neural network.

The generating of the super resolution image may include reconstructing a high resolution luminance image of the input stereo images using a first deep neural network trained on a first predetermined parameter between the stereo images and the high resolution luminance image. step; And generating a super-resolution image for the reconstructed high-resolution luminance image by using a second deep neural network trained on mapping a second predetermined parameter between the high-resolution image and the super-resolution image.

Reconstructing the high resolution luminance image converts each of the stereo images into a luminance image, generates a plurality of luminance images with parallax movement by using the converted one luminance image, and converts the converted other The high resolution luminance image may be reconstructed by concatenating the luminance image and the plurality of luminance images having the generated movement, and then using the first deep neural network that receives the chained luminance image as an input.

The first deep neural network may learn luminance mapping between parallax movement of the stereo images and the high resolution luminance image.

The generating of the super resolution image may include concatenating the chrominance components of each of the reconstructed high resolution luminance image and the stereo images, and generating the super resolution image through the second deep neural network using a semi-residual learning technique. Can be.

The second deep neural network may generate the super resolution image without an activation function of the rectifying linear unit in the last convolutional layer among the convolutional layers constituting the second deep neural network.

The mapping for the first parameter may include luminance mapping, and the mapping for the second parameter may include color difference mapping.

Binocular-based super-resolution imaging device according to an embodiment of the present invention includes an acquisition unit for obtaining stereo images; And a generator configured to generate a super resolution image from the stereo images using the learned deep neural network.

The generation unit may include: a first deep neural network unit configured to learn a mapping of a preset first parameter between the stereo images and the high resolution luminance image and to reconstruct a high resolution luminance image with respect to the input stereo images; And a second deep neural network unit for learning a mapping to a second predetermined parameter between the high resolution image and the super resolution image, and generating a super resolution image for the reconstructed high resolution luminance image.

The first deep neural network unit converts each of the stereo images into a luminance image, and generates a plurality of luminance images with parallax movement by using the converted one luminance image, and converts the converted luminance image into another luminance image. The high resolution luminance image may be reconstructed by concatenating the plurality of generated luminance images and inputting the chained luminance image as an input.

The first deep neural network unit may learn luminance mapping between parallax movement of the stereo images and the high resolution luminance image.

The second deep neural network unit concatenates the reconstructed high resolution luminance image and the color difference component of each of the stereo images, and generates the super resolution image by using a semi-residual learning technique.

The second deep neural network unit may generate the super resolution image without an activation function of the rectifying linear unit in the last convolutional layer among the convolutional layers constituting the second deep neural network.

The mapping for the first parameter may include luminance mapping, and the mapping for the second parameter may include color difference mapping.

According to embodiments of the present invention, deep learning technology may be used to improve the resolution of an image in a binocular based (or stereo) system.

According to embodiments of the present invention, by introducing deep learning into a super resolution algorithm, a high resolution image can be obtained, and the resolution can be improved while the number of images used for super resolution imaging is small. It can also be used effectively.

The present invention can be used in fields where a camera is used, for example, a camera for photographing, as well as inspection equipment requiring a high resolution image, and in particular, in the field of mobile and consumer cameras in which dual cameras are gradually expanding. Can be. That is, according to the present invention, since the image having a higher resolution than the previously obtained resolution can be obtained using the dual camera in the mobile environment, it is possible to take a picture having a sharper picture quality than the existing picture.

FIG. 1 illustrates an example diagram for describing a case where two stereo images having parallax are combined.
2 illustrates a conceptual configuration of a luminance super resolution network structure.
3 illustrates a conceptual configuration of a color difference super resolution network structure.
4 shows an exemplary diagram comparing the difference in super-resolution performance between learning by stereo pairs and learning by stereo tensors.
5 shows an exemplary diagram comparing training influences by an activation function in the last layer of calculating a residual in a luminance super-resolution network.
6 shows an exemplary diagram comparing a bicubic interpolation method for a color difference channel with a color difference network of the present invention.
7 shows an exemplary view comparing the existing method for residual learning with the method according to the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited or limited by the embodiments. Also, like reference numerals in the drawings denote like elements.

Embodiments of the present invention provide a super-resolution imaging technique using deep learning technology in a binocular-based (or stereo) system, and the gist of the present invention is to acquire a high resolution image by introducing deep learning into a super resolution algorithm. .

The main intuition in the present invention is that the disparity is measured with discrete samples of continuous disparity shift (or shift), and mapping between discrete samples of consecutive disparity shift and superresolution images instead of disparity or depth estimation in stereo Implement (or construct) a deep neural network that learns.

At this time, the deep neural network of the present invention is composed of two deep CNN and can learn the termination mapping. The first deep CNN can learn the luminance mapping from a pair of low resolution stereo images to a super resolution luminance image, and the second deep CNN has a high resolution color image at low resolution chrominance with reconstructed super resolution luminance. Can learn color difference mapping to.

The present invention reconstructs a high resolution image from a stereo input rather than a single image super resolution using a stereo input that is not yet fully utilized in super resolution.

The present invention seeks to avoid calculating pixel-by-pixel discrete inconsistencies from stereo pairs. Instead, the present invention uses deep neural networks to directly learn end-to-end mapping from a given stereo pair of low resolution images to a high resolution output image.

For example, parallax occurs through perspective projection between two stereo images, as shown by combining two stereo images with parallax of FIG. 1. Due to geometric differences between parallax in perspective projection and stereo image transformation, for example, similarity in affine transformation, continuous movement due to parallax exists at sub-pixel resolution at the horizontal and vertical displacements of the stereo pair. do. The deep CNN in the present invention can improve image resolution by using subpixel shift due to parallax to reconstruct a high resolution image without estimating depth or inconsistency.

In the present invention, a deep neural network for learning a mapping between a low resolution stereo image pair and a high resolution image may be formulated, for example, a deep CNN.

의 3D 텐서 형태인 YCbCr 계수로 변환한다.First, instead of using the usual RGB color channels,

Convert to YCbCr coefficients in the form of 3D tensors.

은 N개의 LR 스테레오 이미지 쌍

와 그라운드 트루스(ground-truth) 고해상 이미지

으로 주어진다 가정한다.

은 스테레오에서 왼쪽 기준(reference) 컬러 이미지를 의미하고,

은 스테레오에서 오른쪽 이미지를 의미한다. 본 발명에서는 바이큐빅 보간법(bicubic interpolation)을 이용하여 소스 저해상 스테레오 이미지의 해상도를 업샘플링함으로써, 고해상 이미지의 해상도와 매칭시킬 수 있다.Here, H may mean height, W may mean width, and C may mean the number of channel colors of the input image. Training data set

Is N LR stereo image pairs

And ground-truth high resolution images

Assume that given by

Means the left reference color image in stereo,

Means the right image in stereo. In the present invention, by upsampling the resolution of a source low resolution stereo image using bicubic interpolation, it is possible to match the resolution of a high resolution image.

을 예측할 수 있는 모델 F를 학습하기 위한 것이다. 본 발명에서의 양안 초해상 모델은 두 개의 딥 CNN 예를 들어, 휘도 초해상 네트워크와 색차 초해상 네트워크로 구성될 수 있다.The main purpose of a binocular based superresolution network is to provide a high resolution image from a given input X.

This is to train the model F that can predict. The binocular superresolution model in the present invention may be composed of two deep CNNs, for example, a luminance superresolution network and a chrominance superresolution network.

의 잔차 이미지(residual image)를 정의함으로써, 싱글 카메라에서 고해상 이미지와 저해상 이미지 간의 잔차를 학습할 수 있다. 여기서, y_L은 고해상 휘도 이미지를 의미하고, x_1,L은 저해상 휘도 스테레오 이미지들 중 하나를 의미할 수 있다. Luminance Super Resolution Network (or Luminance Super Resolution Deep CNN) : Similar to conventional single image super resolution methods, luminance

By defining a residual image of, a residual between a high resolution image and a low resolution image can be learned in a single camera. Here, y _L may mean a high resolution luminance image, and x _{1, L} may mean one of low resolution luminance stereo images.

In the present invention, since the stereo input of the dual camera is used, in order to learn the residual on one side of the stereo pair, the parallax movement through the network is learned using two stereo images as inputs.

및 저해상 휘도 스테레오 이미지

간의 잔차

을 학습한다. 시차에 의한 서브 픽셀 이동을 고려하기 위해, 결합된 스테레오 이미지 텐서

을 생성하기 위해 왼쪽 이미지와 오른쪽 이미지를 아래 <수학식 1>과 같이 리패킹(repacking)할 수 있다.The first network, Luminance Super Resolution Network, provides high resolution luminance for training data sets.

And low-resolution luminance stereo images

Residual

To learn. Combined Stereo Image Tensors to Consider Subpixel Shift by Parallax

In order to generate, the left image and the right image can be repacked as shown in Equation 1 below.

[Equation 1]

는 j 번째 레이어의 이동된 오프셋을 의미하고, M은 이동 수를 의미할 수 있다.here,

Denotes a shifted offset of the j th layer, and M may denote a number of shifts.

의 평균 제곱 오차(MSE; mean squared error)를 최소화한다. 여기서, f는 저해상 휘도 이미지 x_1,L에 대해 고해상 휘도

를 예측한다. f는 주어진

로부터

를 추정하고, 네트워크를 통해 입력 디멘젼(dimension)을 줄일 수 있다.In the present invention

Minimize mean squared error (MSE). Where f is the high resolution luminance for the low resolution luminance image x _{1, L}

Predict. f is given

from

Can be estimated and the input dimension can be reduced through the network.

를 재구성할 수 있다.According to the present invention, when the residual prediction model f is learned, the left low resolution luminance input and the predicted residual are summed as shown in Equation 2 below, thereby obtaining a high resolution luminance image.

Can be reconstructed.

[Equation 2]

Here, the function f may comprise a plurality of subsets of the convolutional layer f _k .

The kth convolutional layer f _k (Z) may be expressed as Equation 3 below.

[Equation 3]

와

는 k번째 레이어의 컨볼루션 필터와 바이어스를 의미하고, *는 컨볼루션을 의미하며, max(0, )는 정류 선형 유닛(ReLU; rectified linear unit)을 의미할 수 있다.here,

Wow

Denotes a convolution filter and a bias of the k-th layer, * denotes a convolution, and max (0,) may denote a rectified linear unit (ReLU).

는

이고, 바이어스

의 크기는

이다. the kernel if the depth of the kth layer is n _k

Is

, Bias

The size of

to be.

과 스테레오

,

의 저해상 이미지의 Cb 및 Cr 채널로부터 업샘플된 색차이다. 트레이닝 데이터 세트

를 구성하기 위하여, 3 개의 입력 채널

를 아래 <수학식 4>와 같이 연쇄(concatenate)시킬 수 있다. Color difference Super resolution network (or color difference) Super Resolution Deep CNN) : Input of chrominance super resolution network is reconstructed high resolution luminance image

And stereo

,

Is the color difference upsampled from the Cb and Cr channels of the low resolution image. Training data set

To configure the three input channels

Can be concatenated as in Equation 4 below.

[Equation 4]

와

은

와

을 의미할 수 있다.here,

Wow

silver

Wow

It may mean.

의 잔차 이미지를 정의하고,

의 평균 제곱 오차를 최소화한다. 여기서, 함수 g는 저해상 색차와 고해상 휘도의 결합된 입력 이미지

로부터 고해상 컬러들의 잔차를 예측할 수 있다.The main purpose of color difference super resolution networks is to reconstruct high resolution color images. The chrominance superresolution network of the present invention learns the residual between high resolution color and half-way-through low resolution color images in stereo. To this end, the present invention is a color difference

Define the residual image of,

Minimize the mean square error. Here, function g is the combined input image of low resolution color difference and high resolution luminance.

The residual of the high resolution colors can be predicted from.

The present invention, unlike conventional residual learning, follows a semi-residual learning technique with additional convolutional layers. For example, as shown in FIG. 3, the input high-resolution luminance image and the resampled Cb and Cr elements in the low-resolution image and the high-resolution luminance image and the upsampled Cb and Cr element are used. Chain. Here, instead of adding the residual prediction to the low resolution image, the original image and the residual may be concatenated and then applied to the final convolutional layer. This semi-residual learning method improves accuracy.

를 결합시킨다. 이들의 컨볼루션을 계산하기 위하여, 예측 잔차와 입력 컬러들을 아래 <수학식 5>와 같이 연쇄시키다.The present invention is not a simple summation in the existing residual learning, but the predicted residual r is fast forwarded identity in the form of a convolutional layer.

Combine. In order to calculate their convolution, the prediction residual and input colors are concatenated as in Equation 5 below.

[Equation 5]

Here, c 'may be c-3.

을 입력으로 사용하여 아래 <수학식 6>과 같이 최종 컬러 이미지

를 재구성할 수 있다.The final sublayer h is without the rectification linear unit (ReLU) activation function

With the input as the final color image as shown in Equation 6 below.

Can be reconstructed.

[Equation 6]

의 크기는

이며, 바이어스

의 크기는

일 수 있다.Where the depth of the last layer is n _h and the kernel

The size of

Is the bias

The size of

Can be.

를 예측할 수 있는 모델 F를 학습할 수 있다.As described above, the present invention can improve high frequency information at the final output through the saturation upsampling network. As such, the present invention provides a high resolution image from a given input X.

We can learn the model F to predict.

Hereinafter, the structure (or architecture) of the above-described luminance superresolution network and chroma superresolution network will be described in detail.

1.luminance Super resolution  Network structure

Since the human resolution for contrast is primarily dependent on luminance, the present invention uses the luminance channel of the input images.

FIG. 2 illustrates a conceptual configuration of a luminance super-resolution network structure, in which the first layer includes a total of 33 images concatenating the left image used as input and the 32 right images with movement. A single image and 32 images of the right view exist with parallax shift. Here, the last convolutional layer may not use a rectifying linear unit (ReLU) in calculating the residual.

The filter size of the first convolutional layer is 33x3x3x64. The present invention uses 14 inner layers with the same kernel size of 64x3x3x64. There is a ReLU after each convolution layer. The final convolutional layer has a filter size of 64 × 3 × 3 × 1 and is used to reconstruct the residual image. Finally, by adding the residual to the reference image captured by the left camera, a high resolution luminance image is generated.

Receptive field : The size of the receptive field (or receptive area) depends on the size of the filter and the number of layers. For example, a 3x3 filter with 16 layers operates a receiving field size of 33x33. In the present invention, it is assumed that the maximum discrepancy due to parallax is 64 larger than 33.

As more layers are inserted, not only the size of the acceptance field but also the computational cost increases, and the increase in the acceptance field does not necessarily guarantee a performance improvement in terms of accuracy.

Stereo tensor with shift : The network in the present invention can detect similar patches in a shifted input stereo tensor using deep CNN instead of using classic block matching to determine discrete discrepancies. . That is, the network of the present invention can detect patch similarity in a stereo channel including a continuous disparity offset. The nearest patch in the source image can be used directly for reconstruction via CNN, enabling sub-pixel precision multisampling for image enhancement. Regardless of the patch's response to inconsistencies, finding similar patches is a more effective way to improve patch resolution.

The present invention does not require a mismatch map for super resolution reconstruction. The present invention creates a stereo image tensor using 64 shifted images at one pixel interval with a maximum shift distance of 64 pixels. Considering the general camera configuration, it is assumed that the maximum mismatch of images is less than 64 pixels.

4 shows an exemplary diagram comparing the superresolution performance difference between learning from a stereo pair and learning from a stereo tensor, showing a comparison of the superresolution performance difference for a Middlebury data set.

As can be seen from FIG. 4, it can be seen that the learning using the stereo tensor increases the resolution more than the learning using the stereo pair.

Activation functions : The last convolutional layer of the network reconstructs the residuals for high resolution luminance images. The activation function of the last layer can handle the change in contrast of the residual image.

FIG. 5 illustrates an example of comparing training influences by an activation function in the last layer of calculating a residual in a luminance super-resolution network, obtained using a case without a ReLU, Sigmoid, or activation function for a middlebury data set. Exemplary diagrams comparing performances are shown.

As can be seen from FIG. 5, the absence of an activation function shows the highest signal-to-noise ratio (PSNR), and the ReLU and sigmoid functions clamp residual values between [0, ∞] and [-1, +1], respectively. I can see that. This shows that they cause contrast compression, so the present invention does not use an activation function in the last layer to reproduce the high frequency characteristics of the target image.

2. Color difference Super resolution  Network structure

Conventional superresolution algorithms reconstruct high resolution colors by applying the superresolution algorithm independently to three color channels or by converting RGB inputs to the YCbCr color space and applying the superresolution method to Y luminance only.

Instead of using bicubic interpolation, the present invention uses an additional network that enhances the chrominance upsampling stage. Although the luminance component dominates the color image resolution, upsampling of low-resolution chrominance using bicubic interpolation can cause color bleeding problems at the edges. The present invention can overcome this problem by using a color difference super resolution network.

For example, as shown in FIG. 6, the bicubic interpolation problem causes color bleeding at the edge portion, but the method according to the present invention shows that the color bleeding problem at the edge is removed through chrominance upsampling, thereby showing a more clear color. have.

The chrominance superresolution network of the present invention follows a general residual learning structure, uses 15 convolutional layers for residual image learning, and an additional layer for generating a final image from a low resolution chromaticity channel and a reconstructed high resolution luminance image. Use The kernel size of the first convolutional layer is 3 × 3 × 3 × 64, and the internal convolutional layer can have a kernel size of 64 × 3 × 3 × 64 for each convolution operation.

Then, the residuals of the residual learning are concatenated to the reference image and then applied to a kernel size final convolutional layer of 6 × 3 × 3 × 4. At this point, the last convolutional layer does not use the activation function to calculate the final result.

Color difference Of the super-resolution residual (residuals of chrominance SR): in order to preserve the high frequency detail from the reconstructed high-resolution brightness, the present invention is semi-containing convolution layer to the terminal of network, instead of the sum of fast-forward identity and uses the residual learning . The details of the Cb and Cr channels can be lost through the convolution step, but the semi-residual approach can increase the high frequency details.

For example, as can be seen from the comparison result of the residual learning method for the Middlebury dataset shown in FIG. 7, the semi-residual learning method including an additional convolutional layer at the end of the chrominance network is bicubic up. It can be seen that the PSNR of the reconstructed image increases more than the sampling method and the existing residual learning method.

As described above, the present invention can generate super resolution color images from stereo images using two deep learning networks.

Here, the first network uses two stereo images to improve the resolution of the luminance image. Instead of using stereo pairs, the stereo network learns stereo similarity by using a stereo tensor with moving candidates.

The second network upsamples the saturation component of the image based on the high-resolution luminance image and improves the resolution of the predicted saturation high-resolution image by using an additional convolutional layer that enables semi-residual learning. Create

In addition, although the present invention has been described as luminance mapping and chrominance mapping, it is apparent to those skilled in the art that different parameter mapping related to an image may be implemented using learned deep neural networks. In addition, the present invention can be implemented using deep neural networks that have been learned based on luminance, which can cover luminance mapping and chrominance mapping.

The present invention may be implemented by a device and a method, and when the device is configured, two networks, that is, a first deep neural network including a first deep neural network and a second deep neural network including a second deep neural network, may be used. A super resolution image may be generated for input stereo images.

Here, the first deep neural network unit may include the above-described luminance network structure, that is, the network structure of FIG. 2, and the second deep neural network unit may include the above-described chroma network structure, that is, the network structure of FIG. 3.

In addition, the super resolution imaging method using deep learning reconstructs a high resolution luminance image by using a first deep neural network that inputs stereo images and inputs the upsampled saturation into the Cb and Cr channels of the high resolution luminance image and the stereo image as an input. A process of generating a super resolution image may be performed by using a second deep neural network.

In addition, it is obvious that the apparatus and method may include both the super resolution imaging process and the content using deep learning.

The system or apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the systems, devices, and components described in the embodiments may include, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors, microcomputers, field programmable arrays (FPAs). ), A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For the convenience of understanding, the processing apparatus may be described as one used, but those skilled in the art will appreciate that the processing apparatus includes a plurality of processing elements and / or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as parallel processors.

The software may include a computer program, code, instructions, or a combination of one or more of the above, and may configure the processing device to operate as desired, or process independently or collectively. You can command the device. Software and / or data may be any type of machine, component, physical device, virtual equipment, computer storage medium or device in order to be interpreted by or to provide instructions or data to the processing device. Or may be permanently or temporarily embodied in a signal wave to be transmitted. The software may be distributed over networked computer systems so that they may be stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to the embodiments may be embodied in the form of program instructions that may be executed by various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the media may be those specially designed and constructed for the purposes of the embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of footstep commands include high-level language code that can be executed by a computer using an interpreter, as well as machine code such as produced by a compiler. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the embodiments have been described by the limited embodiments and the drawings as described above, various modifications and variations are possible to those skilled in the art from the above description. For example, the described techniques may be performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different form than the described method, or other components. Or, even if replaced or substituted by equivalents, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are within the scope of the claims that follow.

Claims (14)

Obtaining stereo images; And
Generating a super resolution image from the stereo images using a learned deep neural network
Including,
Generating the super resolution image
A binocular based super resolution imaging method for generating a super resolution image from the stereo images using a deep neural network trained by parallax.
The method of claim 1,
Generating the super resolution image
Reconstructing a high resolution luminance image for the stereo images input using a first deep neural network trained on a mapping of a preset first parameter between the stereo images and the high resolution luminance image; And
Generating a super-resolution image for the reconstructed high-resolution luminance image using a second deep neural network trained on mapping a second predetermined parameter between the high-resolution luminance image and the super-resolution image
Binocular-based super-resolution imaging method comprising a.
The method of claim 2,
Reconstructing the high resolution luminance image
Converting each of the stereo images into a luminance image, and generating a plurality of luminance images with parallax movement by using the converted one luminance image, wherein the converted other luminance image and the generated parallax movement are And reconstructing the high resolution luminance image by concatenating a plurality of luminance images and using the first deep neural network as the input.
The method of claim 2,
The first deep neural network
Binocular-based super resolution imaging method, characterized in that it learns a luminance mapping between the parallax shift of the stereo images and the high resolution luminance image.
The method of claim 2,
Generating the super resolution image
Binocular-based super resolution imaging, wherein the reconstructed high resolution luminance image and the chrominance component of each of the stereo images are concatenated and the super resolution image is generated through the second deep neural network using a semi-residual learning technique. Way.
The method of claim 2,
The second deep neural network
And generating the super-resolution image in the last convolutional layer of the second deep neural network without the activation function of the rectifying linear unit.
The method of claim 2,
The mapping for the first parameter is
Including luminance mapping,
The mapping for the second parameter is
A binocular-based super resolution imaging method comprising color difference mapping.
An acquisition unit for acquiring stereo images; And
A generator for generating a super resolution image from the stereo images using the learned deep neural network
Including,
The generation unit
A binocular-based super resolution imaging apparatus for generating a super resolution image from the stereo images using a deep neural network trained by parallax.
The method of claim 8,
The generation unit
A first deep neural network unit configured to learn a mapping of a first predetermined parameter between the stereo images and the high resolution luminance image and to reconstruct a high resolution luminance image with respect to the input stereo images; And
A second deep neural network portion that learns a mapping to a second predetermined parameter between the high resolution luminance image and the super resolution image and generates a super resolution image for the reconstructed high resolution luminance image
Binocular-based super-resolution imaging device comprising a.
The method of claim 9,
The first deep neural network part
Converting each of the stereo images into a luminance image, and generating a plurality of luminance images with parallax movement by using the converted one luminance image, wherein the converted other luminance image and the generated parallax movement are And concatenating the plurality of luminance images and reconstructing the high resolution luminance image by inputting the chained luminance image.
The method of claim 9,
The first deep neural network part
Binocular-based superresolution imaging apparatus, characterized in that it learns a luminance mapping between the parallax shift of the stereo images and the high resolution luminance image.
The method of claim 9,
The second deep neural network part
And concatenating the chrominance components of the reconstructed high resolution luminance image and the stereo images and generating the super resolution image using a semi-residual learning technique.
The method of claim 9,
The second deep neural network part
And generating the super-resolution image without an activation function of the rectifying linear unit in the last convolutional layer among the convolutional layers constituting the second deep neural network.
The method of claim 9,
The mapping for the first parameter is
Including luminance mapping,
The mapping for the second parameter is
A binocular-based super resolution imaging device comprising color difference mapping.

Images