KR101513931B1 - Auto-correction method of composition and image apparatus with the same technique - Google Patents

Abstract

The invention provides an auto correction method for composition and an image apparatus with the same technique. An auto correction method for composition is composed of a phase to receive image inputs, a phase to extract the ROI(region of interest) from inputs, a phase to compose three way-segmented lattices vertically and horizontally, select a main object within the image and place the same at the intersection within the lattice, a phase to set the coordinates for the selected object, and a phase to crop the extracted ROI by using the coordinates setting. The phase to extract the ROI is composed of a phase to create one or more low level feature maps by extracting characteristics from the input image, a phase to create saliency maps from one or more low level feature maps and a phase to extract a first ROI from the saliency map.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to an image correction method,

One embodiment of the present invention relates to a method of automatically correcting a composition and a video apparatus equipped with such a function.

Recently, as digital cameras become popular, cameras with various functions are mounted on general mobile devices. However, users often do not know how to use the guideline that guides the composition correction among the functions built in the camera or the function that automatically performs face recognition.

It was a wonderful scenery when I looked at ordinary scenery or things, but when I put it in the camera, it feels like something different. This is because you can see almost everything that has been spread out in front of you from a wide angle with the human eye, but you have to put the subject in a rectangular frame of the form of a photograph in the camera. Depending on how stable the subject is in this rectangular area, the person who views the picture may change.

For example, the intention of the photographer is conveyed through the composition, such as line or shape, contrast and texture of the color, the weight of the main subject and surrounding material, perspective, direction, and state of motion. For example, when a carp in a pond is photographed, there is a problem of arranging or arranging a certain size on a certain portion of the screen. When the number of carp is increased, There is a problem of. And, when the arrangement of the space such as the fish of the different form and size and the person of the surroundings, and the perspective of the viewer are added, the composition is intentionally made.

Actually, when composing a picture, simply divide the screen into sets and use a composition method that easily uses points and lines similar to the Golden Section. In order to apply this comforting golden ratio to photography, three halves of the picture horizontally and vertically produce two vertical lines and a horizontal line, and four intersections are created. When one of these intersections is placed, The screen of the equal division is constituted. The photographs of this composition can enhance the concentration of the viewer, and can provide comfort in the overall picture composition.

However, if the composition in the picture is not correct, it will give an unstable feeling. Therefore, efforts are being made to reconstruct images and / or images in accordance with the three-segment technique.

A problem to be solved by the present invention is to provide a method of automatically correcting an appropriate composition in an image by reconstructing the image in accordance with a three-division technique, and a video device equipped with such a function.

In addition, one embodiment of the present invention is to provide an automatic correction method of a composition that can enhance aesthetics and correct a composition stably and sophistically, and a video apparatus equipped with such a function.

An embodiment of the present invention includes an input step of inputting an image, a step of extracting a region of interest (ROI) from the input image, a step of dividing the input image into three parts, Selecting an important object in the image and positioning it at an intersection of the grid, setting coordinates of the selected object, and cropping the extracted region of interest using the set coordinates Wherein extracting the region of interest (ROI) from the input image comprises: extracting features from the input image to generate at least one low-level Feature Map step; Generating a Saliency Map from the one or more feature maps; And extracting a first region of interest from the Saliency Map.

delete

In addition, the feature element of the feature map may be at least one of color, brightness and direction.

The step of extracting a region of interest (ROI) from the input image may include: detecting the first ROI from the input image; generating a result image using a color segmentation technique; Extracting a second region of interest by merging the binary image with the input image using an edge detection technique and eliminating regions of low similarity; .

In addition, the color segmentation technique may divide a region of an image by collecting similar colors into a single group using density and degree of aggregation between colors.

In addition, the cropping may be performed by applying a scenario corresponding to the type of the input image.

In addition, if the type of the input image is a person, the scenario may be applied to include a body region.

According to another embodiment of the present invention, there is provided a video apparatus equipped with the above-described automatic correction function of the composition.

According to an embodiment of the present invention, there is provided an automatic correction method of a suitable composition within an image by reconstructing the image in accordance with a three-division technique in the image, and an image device equipped with such a function.

According to an embodiment of the present invention, there is also provided an automatic correction method of a composition that can enhance aesthetics and correct a composition stably and sophistically, and a video apparatus equipped with such a function.

1A is a diagram showing an example of a Rule of Thirds.
1B is a diagram showing another example of a Rule of Thirds.
2A is a photograph showing the subject to be positioned in the center of the photograph.
FIG. 2B is a photograph in which a subject is repositioned through composition correction using a three-division method.
3A is a diagram showing an image before reconstruction by cropping.
FIG. 3B is a view showing reconstruction of FIG. 3A by cropping.
FIG. 4 is a diagram illustrating a process of finding a closest intersection point by finding a distance between an intersection of an important object and a grid in an embodiment of the present invention.
5 is a flowchart illustrating a method of controlling a camera capable of automatically correcting a composition according to an embodiment of the present invention.
FIG. 6 is a diagram illustrating an example of extracting a region map of interest (ROI) in the automatic correction method of a composition according to an embodiment of the present invention.
FIG. 7 and FIG. 8 are diagrams showing results of deriving a region of interest map (Saliency Map) through an LUV color space and a mean difference.
FIG. 9 shows the result of an image according to an embodiment of the present invention when the constants k are 2, respectively.
FIG. 10 shows the result of an image according to an embodiment of the present invention when the constants k are 5, respectively.
11 is a diagram illustrating a result of finding an optimal ROI by combining a result of performing color segmentation with a saliency map in an embodiment of the present invention.
12 is a diagram illustrating an example of applying a scenario according to an embodiment of the present invention.
FIGS. 13 and 14 illustrate an image of which the composition is automatically corrected through the above experimental example.

Hereinafter, embodiments of the present invention will be described in detail. However, the present invention is not limited thereto, and the present invention is only defined by the scope of the following claims.

Most of the photos taken using a digital camera or a camera mounted on a mobile device are not stable. This is because it is not easy for a non-specialist to take a picture of a subject by taking a picture of the subject in a rectangular area of a photograph, so that the position of the subject can be stabilized.

It is a technique for taking photographs more similar to professionals, and it can be used to create photographic compositions such as Rule of Thirds, Cropping, Viewpoint, Experimentation, Balancing Elements, Leading Lines, Symmetry and Patterns, Background, Depth, There is law. These methods are very important when composing a picture, and they are components that have great influence in the process of viewing pictures through human eyes.

However, there is a limitation in the ability to correct images by adding such factors in a digital camera or a mobile device at the moment of photographing due to limitations of computation ability and memory.

Accordingly, in the present invention, an image is acquired through photographing, and the image is processed by post-processing using Rule of Thirds and Cropping in the above-described photographic composition method, thereby enhancing esthetics .

The Rule of Thirds and Cropping are briefly described below.

1A is a diagram showing an example of a Rule of Thirds. 1B is a diagram showing another example of a Rule of Thirds.

Referring to FIGS. 1A and 1B, a photograph is divided into two vertical lines and horizontal lines dividing horizontal and vertical portions into three divisions, respectively, so that it is easy to see where a subject is located.

Particularly, in FIG. 1B, the intersection point where the subject should be located is displayed among the four intersection points resulting from intersection of two vertical lines and a horizontal line. Photographs of this composition can increase the concentration of viewers and provide comfort in the overall composition.

2A is a photograph showing the subject to be positioned in the center of the photograph. FIG. 2B is a photograph in which a subject is repositioned through composition correction using a three-division method.

Referring to FIGS. 2A and 2B, when the subject to be emphasized by the photographer is a subject, the gaze is further observed in the photograph of FIG. 2B in which the subject is repositioned through composition correction.

Cropping to be applied together with the above-described Rule of Third is a typical method of cutting out a region of interest in an image and composing the image into the region. And has been widely used as a processing technique for a region of interest (ROI) in general.

3A and 3B show before and after reconstruction of an image by cropping, respectively.

As described above, if the composition is not correct, the image has an overall unstable feel.

Accordingly, in the present invention, the image or / and the image are reconstructed using Rule of Thirds and Cropping, thereby enhancing the esthetics and automatically correcting the composition more stably and sophisticatedly.

Hereinafter, preferred embodiments of the present invention will be described in more detail.

According to an embodiment of the present invention, there is provided an image processing method comprising the steps of: selecting a subject; setting coordinates of a predetermined subject; and cropping a region of interest (ROI) The method comprising the steps of: (a) controlling the camera to automatically correct the composition;

More specifically, first, a grid is formed by dividing the width and height of the input image by 1/3, respectively, and an important subject is selected, and the subject is moved and corrected by the four intersections formed by the imaginary line.

At this time, it is necessary to move the subject to the intersection from the three-segment grid constructed. In the case of a portrait photograph, the center of the subject is the face of a person. To do this, face detection is performed on the input image, and the coordinates (x, y) of the face are extracted.

When the face coordinates of the subject are detected, it is necessary to calculate which coordinates are closest among the four intersections obtained previously. This is for cropping only the minimum area when cropping the image, which will be described later.

FIG. 4 is a diagram illustrating a process of finding a closest intersection point by finding a distance between an intersection of an important object and a grid in an embodiment of the present invention.

Referring to FIG. 4, it can be seen that the face coordinate, which is an important subject, is closest to the coordinate located at the right upper end among the four grid intersections. The method of finding the face coordinates and the distance of the intersection can be found by calculating the Euclidean distance of each coordinate.

When the face coordinates and the intersection to be moved are determined, the variables for each axis are specified for the condition check. In Fig. 4, when the width of a video image is height and the height is height, a long side is set as x1 and a long side is set as x1 and a short side is set as x2 and y2 from the coordinates of the subject.

In order to obtain the x-axis and y-axis to be cropped, x1, x2, y1, and y2 satisfying the following condition are obtained.

x1 + x2? width ----- (1)

y1 + y2? height ----- (2)

x1 = 2 * x2 && y1 = 2 * y2 ----- (3)

X1, x2, y1, and y2 satisfying the conditions (1), (2), and (3) described above can be obtained to know how much image area is used centering on the subject.

The result of face recognition on the input image is used as a weight to construct a saliency map, and a primary ROI is searched in the image through color segmentation.

The region of interest (ROI) found by the above method is cropped by applying the scenario of the present invention and applying the three-division method to each scenario, It is possible to obtain the image with maximized stability.

5 is a flowchart illustrating a method of controlling a camera capable of automatically correcting a composition according to an embodiment of the present invention.

Referring to FIG. 5, first, a method of constructing a saliency map is based on a top-down approach for finding an area of interest by an observer observing an image, and a visual stimulus given without prior knowledge of the image There is a bottom-up approach to extract regions of interest.

In one embodiment of the present invention, a saliency map is created using various factors that affect the image such as color, luminance, motion, texture, and orientation. .

Here, in one embodiment of the present invention, a method of obtaining a region map of interest (Saliency Map) is divided into three steps as follows.

First, one or more low-level feature maps are extracted by extracting features. The next step is to construct a Saliency Map from the feature maps. Finally, a core region of the Saliency Map is set as a region of interest.

More specifically, for example, a method based on fuzzy growing is generalized to calculate a Saliency Map. First, color quantization is performed on an input image, and a feature map is generated to apply fuzzy growing.

As another example, features, categories, and prior knowledge of objects are performed independently. There is a method of extracting the spectral residual of the image in the spectral region by log-spectrum analysis of the input image, and comparing the spectral residual with the saliency map of the spatial region.

Most of the methods for obtaining the Saliency Map have a resolution of the original input image of a small size due to the size change of the filter. Some interest area maps are not well extracted, and other interest area maps have good boundaries but can not get a certain result.

Accordingly, in order to solve such a problem, it is possible to construct a multi-pyramid image for various feature elements and obtain a center sur- round difference between the images. Since these values are represented by various values according to the feature elements existing in the image, an optimal ROI map should be derived by combining multiple values rather than using only one value.

FIG. 6 is a diagram illustrating an example of extracting a region map of interest (ROI) in the automatic correction method of a composition according to an embodiment of the present invention.

In the present invention, a map reflecting each characteristic is generated by using a complex feature element, and a saliency map which is most suitable for the automatic correction algorithm of the composition is derived.

For example, the brightness value, which is the most influential factor in human vision, is the main criterion for constructing the saliency map. Therefore, the intensity map is created according to the following equation (4) .

I = (r + g + b) / 3 ----- (4)

The human neural circuit that transmits color information to the brain converts the information by the three cones into the color information of the opposite pair of 'red / green' and 'yellow / blue'. In order to obtain a feature map from the input image, let r, g, and b be the red, green, and blue colors of the RGB color model.

R = r- (g + b) / 2, G = g- (r + b) / 2,

(5): B = b- (r + g) / 2, Y = r + g-

R denotes a red channel, G denotes a green channel, B denotes a blue channel, and Y denotes an yellow channel. In the present invention, the two color feature maps are created using the four channels by the following equation (6).

RG (x, y) = R (x, y) -G (x, y)

BY (x, y) = B (x, y) -Y (x, y)

Here, RG and BY are color information of the opposite pair of 'red / green' and 'yellow / blue'. Each Feature Map is transformed into a Conspicuity Map, which has a feature that emphasizes the most prominent part from the periphery. The following equation (7) shows the process of performing the Gaussian pyramid to obtain the conspicuity map.

P _j (0) = F _j ,

P _j (i) = F _j (i-1) x G ---- - (7)

In the above description, if each feature is represented by F, the nine-dimensional Gaussian pyramid P is obtained by gradually subdividing the feature map F by a Gaussian filter G and performing low-pass filtering .

For each feature P, six intermediate multi-size magnitude map (Conspicuity Maps) are obtained by the following equation (8).

_{M, 1 = | P (2} ) -P (5) |, M, 2 = | P (2) -P (5) |,

M _{, 3} = | P (3) -P (6) |, M _{, 4} =

M _{, 5} = | P (4) -P (7) |, M _{, 6} = | P (4) ----- (8)

Six Convergence Mismatch Conspicuity Maps are obtained by normalizing the sum of one Conspicuity Map using the following equation (9).

----- (9)

----- (9)

The visual attention model constructs a saliency map by determining and combining weights for each Conspicuity Map.

-----(10)

----- (10)

Where n is the number of conspicuity maps.

In the present invention, the dynamic weighting value W can be determined dynamically according to the importance of the Conspicuity Map used to construct the saliency map.

FIG. 7 and FIG. 8 are diagrams showing results of deriving a region of interest map (Saliency Map) through an LUV color space and a mean difference.

Referring to FIGS. 7 and 8, when the background and the object show distinct colors, the performance is excellent.

As described above, the most important region of interest in the photographic image is the human face. Therefore, the human face must have a high value among the areas of interest above all.

Accordingly, face detection is separately performed to detect a face region of a person. The technique of detecting a person's skin color in an image is a very important and useful method for detecting a face region.

In order to extract such skin color, when the input image is an RGB method, it is converted into a YCbCr color model and used. In the case of the RGB color model, there is a high probability that an error occurs in the process of detecting the skin color as the light state changes. The RGB color model can be converted to the YCbCr method through the following equation (11).

Y = 0.299R + 0.587G + 0.114B,

Cb = -0.169R-0.332G + 0.5B,

Cr = 0.5R-0.419G-0.081B (11)

FIG. 8 shows the scattering form of the detected skin color pixels and the histogram of Cb and Cr.

After the facial region is detected and the resultant image is obtained through color segmentation, and the resultant binary image is removed by removing the small regions of the resulting image, the process of integrating the generated binary image with the previously obtained image using the edge detection technology, A small area not belonging to the area can be removed.

After removing the small area, the coordinates of the location determined by skin color are selected using the image, and the area of the rectangle shape is designated using a color-based segmentation technique. At this time, each selection region has a unique window size.

In the case of areas such as the neck, arms, and hands located below the face detected by the skin color, the merge algorithm is used to merge the adjacent box areas into a large box area. The image matching algorithm (Image Matching Algorithm) is applied to the detected face regions, and the regions are ranked in order, and the region having the similarity is compared with the ranking of the regions having different ranking values in consideration of the distance and the window size, The face area can be detected.

However, we found a local maximum in a block of a certain size area, but there is no criterion to judge the extent of the ROI actually.

Therefore, in order to set the correct ROI, it is necessary to derive the ROI from the Saliency Map. In the present invention, the ROI is detected by fusing two results using a color segmentation technique.

Color segmentation is a method of dividing a region of an image by grouping similar colors into a group using density and degree of aggregation between colors. In one embodiment of the present invention, the K-Means Clustering Algorithm is applied.

Here, the K-Means Clustering Algorithm is a clustering method that divides a set of n objects into k clusters based on distance, and the algorithm is executed as follows.

First, the number of arbitrary k clusters is determined, and an initial value or a cluster center is assigned to each cluster to set a position. Then, the distance to k positions is obtained for each data, and belongs to the closest cluster. Here, the distance can be obtained by using the Euclidean distance described above.

Reset the location of the center of the new cluster to the minimum based on the data divided into clusters. If the position of the center of the newly obtained cluster is the same as that of the existing cluster, the algorithm is terminated. Otherwise, the distance to k positions is obtained for each of the data, and the cluster is reattached to the closest cluster.

The image by this algorithm can show different results depending on the cluster number value k. That is, a suitable constant value should be selected for use in the automatic composition correction, and it is necessary to combine with a saliency map to derive an optimal ROI value.

FIGS. 9 and 10 show the result of an image according to an embodiment of the present invention when the constants k are 2 and 5, respectively.

9 and 10, since the edge of the image can be clearly distinguished when the number of clusters is two, in the present invention, k = 2 is set to an appropriate value for use in automatic composition correction .

FIG. 11 shows a result of finding an optimal ROI by combining the result of performing color segmentation with a saliency map.

The result of the ROI obtained from the image is subjected to a cropping process so as to conform to the rule of the Thirds standard.

In this case, if only the intersection closest to the position of the ROI is searched and cropped, the image of the unintended composition may be obtained. Therefore, in order to preserve the elements inherent in the image, it is necessary to perform cropping by applying a scenario suitable for each image characteristic.

In addition, we measure the ratio of the horizontal and vertical of the image to the ROI, and apply the Rule of Thirds method to maximize the stability and aesthetics of the image.

However, image depends on various factors such as object type, ROI size, and human face recognition area. Therefore, various scenarios are created to bring out the resultant image with proper configuration of various images.

For example, in the case of face recognition, cropping is performed according to the three-division method, but the method of enhancing the stability of the image by protecting the body region is used.

If a line segment such as a horizon is detected in the image, it is maximized by placing the segment on the segment line segmented by the triple method. In this method, it is necessary to search for a line segment that is mainly represented in the image and apply a correction process while preserving the image as much as possible.

12 is a diagram illustrating an example of applying a scenario according to an embodiment of the present invention.

12 shows a resultant image in which a line segment of a horizontal line is detected and an input image and a scenario are applied.

As described above, when the input image is a person, if the face of the person is the most important ROI but the face of the person occupies the entire area of the image, an unnatural image result can be obtained if the image is cropped only in accordance with the ROI .

Therefore, in such a case, the method of maintaining the image as it is may be used to improve the stability of the image.

Experimental Example

Experimental results are derived by applying the proposed method to multiple images for the evaluation of the automatic composition correction proposed in the present invention.

As an experimental method, a color space segmentation and a saliency map were used to derive a region of interest having a maximum value in the input image, and then the composition correction was performed.

In the image, the region of interest can be located at four intersections constituting the golden partition, and the image is corrected by arranging the region of interest in the nearest golden partitioning intersection through the distance measuring operation.

FIGS. 13 and 14 illustrate an image of which the composition is automatically corrected through the above experimental example.

13 and 14, it can be seen that the subject composition in the image in which the composition is automatically corrected by the experimental result is visually more stable than the composition of the subject in the input image.

It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention as defined by the following claims. As will be understood by those skilled in the art. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive.

Claims (8)

An input step of inputting an image;
Extracting a region of interest (ROI) from the input image;
Forming a grid in which the width and height of the input image are divided into three, respectively, selecting an important subject in the image and placing the selected object at an intersection of the grid;
Setting coordinates of the selected subject; And
And cropping the extracted region of interest using the set coordinates,
The step of extracting a region of interest (ROI) from the input image may include:
Extracting a feature from the input image to generate at least one low-level Feature Map;
Generating a Saliency Map from the one or more feature maps; And
And extracting a first region of interest from the Saliency Map. ≪ Desc / Clms Page number 19 > delete The method according to claim 1,
Wherein the characteristic elements of the feature map are at least one of color, brightness, and direction. The method of claim 3,
The step of extracting a region of interest (ROI) from the input image may include:
Detecting the first region of interest from the input image, and generating a resultant image using a color segmentation technique;
Converting the resultant image into a binary image; And
Integrating the binary image with the input image using an edge detection technique, and extracting a second region of interest by removing a region having a low degree of similarity. The method of claim 4,
Wherein the color segmentation technique divides a region of an image by grouping similar colors into a single group using density and degree of aggregation between colors. The method according to claim 1,
Wherein the cropping is performed by applying a scenario corresponding to the type of the input image. The method of claim 6,
And if the type of the input image is a person, a scenario is applied so as to include a body region. A video apparatus equipped with an automatic correction function of a composition according to any one of claims 1 to 7.

Images