KR102253918B1 - Emotion analysis system and operation method of the same - Google Patents

Abstract

An emotion analysis system according to an embodiment of the present invention includes: an image photographing unit that photographs audiences during a photographing period and generates a first image; An image processing device that divides the first image into a plurality of second images and transmits the plurality of second images according to a storage unit time; And an image analysis device that analyzes the plurality of second images during an analysis period to generate emotion information for each of the plurality of second images, wherein the image processing device divides the first image A dividing unit for storing a plurality of second images; And a transmission unit configured to transmit the stored plurality of second images to the image analyzing apparatus when the plurality of second images are stored for the storage unit time.

Description

Emotion analysis system and its operation method {EMOTION ANALYSIS SYSTEM AND OPERATION METHOD OF THE SAME}

An embodiment of the present invention relates to an emotion analysis system and an operation method thereof, in particular, an emotion analysis system capable of analyzing emotions based on facial expressions and behavior patterns of the viewers by photographing audiences while video content is being screened, and an operation method thereof. will be.

There have been attempts to measure audience satisfaction with conventional video content such as movies, dramas, and games, and to predict the possibility of box office success. In this case, in order to determine the qualitative value of the video content targeting the audience, it was necessary to investigate the degree of concentration on the video content and the response to the video content.

However, since the conventional audience analysis for video content takes the form of a questionnaire survey, only the evaluation of the overall concentration of the video content is made, and there is a problem in that the subjective intention is reflected and an objectified result cannot be obtained.

Moreover, for such a survey, the audience has to take a separate time to respond to the survey, and it is limited to obtain the assessment results for the entire audience, and the reliability of the assessment data is reflected in the assessment results because the survey data on some of the unaffected respondents is reflected in the assessment results. There was a problem of deterioration. In addition, even in the case of analyzing the facial expressions of the audience, there is a problem that the user of the analysis system cannot receive the analysis result immediately after the screening of the video content is finished.

Korean Patent Laid-Open Patent No. 2018-0077680'Face expression recognition-based service providing apparatus and its method' (published on July 9, 2018) Japanese Patent Publication No. 2006-012171'Review management system and management method using biometric recognition' (published on January 12, 2006)

An object to be solved of the present invention is to provide an emotion analysis system and an operation method thereof capable of analyzing emotions based on facial expressions and behavior patterns of the audience by photographing audiences while video content is being screened.

In addition, another problem to be solved by the present invention is an emotion analysis system capable of improving the analysis speed and an operation method thereof by dividing the face image of the audience into a plurality of divided images according to the face recognition result and transmitting it to the image analysis device. Is to provide.

In addition, another problem to be solved by the present invention is to provide an emotion analysis system and an operation method thereof capable of improving the analysis speed by performing an analysis on a plurality of divided images in parallel by an image analysis apparatus.

In addition, another problem to be solved by the present invention is to provide an emotion analysis system and an operation method thereof capable of completing analysis while video content is being screened.

In order to achieve the above object, an emotion analysis system according to an embodiment of the present invention includes: an image photographing unit that photographs audiences during a photographing period and generates a first image; An image processing device that divides the first image into a plurality of second images and transmits the plurality of second images according to a storage unit time; And an image analysis device that analyzes the plurality of second images during an analysis period to generate emotion information for each of the plurality of second images, wherein the image processing device divides the first image A dividing unit for storing a plurality of second images; And a transmission unit configured to transmit the stored plurality of second images to the image analyzing apparatus when the plurality of second images are stored for the storage unit time.

In the present invention, the division unit may include: a face recognition unit configured to recognize faces of the audience included in the first image and extract the number of the audience; A divided area setting unit that compares the number of audiences and a reference value, and divides the photographing area of the first image into a plurality of divided areas in different ways according to the comparison result; An analysis target selection unit selecting at least one of the plurality of divided regions as an analysis target; And an image storage unit that stores the plurality of second images according to the analysis target.

In the present invention, when the number of audiences is less than or equal to the reference value, the divided area setting unit divides the photographing area into the plurality of divided areas according to the number of audiences, and each of the plurality of divided areas , It represents the face of one spectator.

In the present invention, when the number of audiences is greater than the reference value, the divided area setting unit receives the number of divided areas from a user, and divides the photographing area into the plurality of divided areas according to the number of the received divided areas. Divide.

In the present invention, when the number of audiences is greater than the reference value, the divided area setting unit groups adjacent audiences with each other, and divides the photographing area into the plurality of divided areas according to the number of generated groups.

In the present invention, the divided area setting unit generates a plurality of groups by grouping adjacent audiences using a K-average algorithm for the audiences.

In the present invention, the transmission unit includes: a determination unit that determines the storage unit time; A counter for counting the storage unit time while storing the plurality of second images; And an image transmitting unit to transmit the stored plurality of second images to an image analysis apparatus when the storage unit time has passed, and the counter counts the storage unit time again when the capturing period has not passed.

In the present invention, the determination unit receives the storage unit time input from the user.

In the present invention, the determination unit determines the storage unit time so that the capacity of the plurality of second images is set to be less than or equal to the maximum processing capacity of the image analysis apparatus.

In the present invention, the counter stops counting when the photographing period has elapsed.

In order to achieve the above object, a method of operating an emotion analysis system according to an embodiment of the present invention includes: photographing audiences during a photographing period and generating a first image; Dividing the first image and storing a plurality of second images; If the plurality of second images are stored for a storage unit time, transmitting the stored plurality of second images to an image analysis device; Generating emotion information for each of the plurality of second images by analyzing the plurality of second images during an analysis period; And generating integrated emotion information by integrating the emotion information.

In the present invention, the storing of the plurality of second images may include recognizing faces of the audience included in the first image and extracting the number of the audience; Comparing the number of audiences and a reference value, and dividing the photographing area of the first image into a plurality of divided areas in different ways according to the comparison result; Selecting at least one of the plurality of divided regions as an analysis target; And storing the plurality of second images according to the analysis target.

In the present invention, the step of dividing the photographing area into the plurality of divided areas may include comparing the number of audiences and a reference value; And when the number of audiences is less than or equal to the reference value, dividing the photographing area into the plurality of divided areas according to the number of audiences, wherein each of the plurality of divided areas is a face of one audience. This indicates.

In the present invention, the step of dividing the photographing area into the plurality of divided areas comprises, when the number of audiences is greater than the reference value, the number of divided areas is input from a user, and the number of divided areas is input according to the number of divided areas. And dividing the photographing area into the plurality of divided areas.

In the present invention, the step of dividing the photographing region into the plurality of divided regions comprises, when the number of viewers is greater than the reference value, grouping adjacent viewers with each other, and separating the photographing region according to the number of created groups. It further includes dividing into a plurality of divided regions.

In the present invention, the transmitting of the stored plurality of second images may include determining the storage unit time; Counting the storage unit time while storing the plurality of second images; And transmitting the plurality of stored second images when the storage unit time has elapsed, and when the photographing period has not elapsed, proceeds to counting the storage unit time again.

In the present invention, determining the storage unit time includes receiving the storage unit time from a user.

In the present invention, in the determining of the storage unit time, the storage unit time is determined so that the capacity of the plurality of second images is set to be less than or equal to the maximum processing capacity of the image analyzing apparatus.

In the present invention, the photographing period partially overlaps the analysis period.

In the present invention, the photographing period and the analysis period are the same as each other.

The emotion analysis system and its operation method according to an embodiment of the present invention may capture audiences while video content is being screened, and analyze emotions based on their facial expressions and behavior patterns.

In addition, the emotion analysis system and the operation method thereof according to an exemplary embodiment of the present invention may improve analysis speed by dividing a face image into a plurality of divided images according to a face recognition result and transmitting the divided images to the image analysis apparatus.

In addition, the emotion analysis system and the operation method thereof according to an embodiment of the present invention can improve the analysis speed by the image analysis apparatus performing analysis on a plurality of divided images in parallel.

In addition, the emotion analysis system and the operation method thereof according to an embodiment of the present invention may complete the analysis while the video content is being screened.

The effects obtainable in the present invention are not limited to the above effects, and other effects that are not mentioned will be clearly understood by those of ordinary skill in the art from the following description.

1 and 2 are diagrams illustrating an emotion analysis system according to an embodiment of the present invention.
3 is a diagram illustrating an image processing apparatus according to an embodiment of the present invention.
4 is a view showing a division unit according to an embodiment of the present invention.
5 to 7 are diagrams showing the operation of the dividing unit according to an embodiment of the present invention.
8 is a diagram illustrating a transmission unit according to an embodiment of the present invention.
9 to 11 are timing diagrams showing the operation of the emotion analysis system according to an embodiment of the present invention.
12 is a diagram showing emotion information according to an embodiment of the present invention.
13 is a flowchart illustrating a method of operating an emotion analysis system according to an embodiment of the present invention.
14 is a flowchart illustrating in detail a method of operating an emotion analysis system according to an embodiment of the present invention.
15 is a flowchart illustrating in detail a method of operating an emotion analysis system according to an embodiment of the present invention.
16 is a flowchart illustrating in detail a method of operating the emotion analysis system according to an embodiment of the present invention.
17 is a flowchart illustrating in detail an operating method of the emotion analysis system according to an embodiment of the present invention.

The present invention will be described in more detail.

Hereinafter, embodiments of the present invention and other matters necessary for a person skilled in the art to easily understand the contents of the present invention will be described in detail with reference to the accompanying drawings. However, since the present invention may be implemented in various different forms within the scope of the claims, the embodiments described below are merely exemplary regardless of whether or not they are expressed.

The same reference numerals refer to the same elements. In addition, in the drawings, thicknesses, ratios, and dimensions of components are exaggerated for effective description of technical content. "And/or" may include all combinations of one or more that the associated configurations may define.

Terms such as first and second may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another component. For example, without departing from the scope of the present invention, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. Singular expressions may include plural expressions unless the context clearly indicates otherwise.

In addition, terms such as “below”, “lower”, “above”, and “upper” are used to describe the relationship between the components shown in the drawings. The terms are relative concepts and are described based on the directions indicated in the drawings.

Terms such as "comprise" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, but one or more other features, numbers, or steps. It is to be understood that it does not preclude the possibility of addition or presence of, operations, components, parts, or combinations thereof.

That is, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms. In the following description, when a certain part is connected to another part, it is directly connected. In addition, it may include a case in which the device is electrically connected with another element interposed therebetween. In addition, it should be noted that the same components in the drawings are indicated by the same reference numerals and reference numerals as much as possible, even if they are indicated on different drawings.

1 and 2 are diagrams illustrating an emotion analysis system 10 according to an exemplary embodiment of the present invention.

1 and 2, the emotion analysis system 10 may include an image capturing apparatus 100, an image processing apparatus 200, and an image analysis apparatus 300.

The image photographing apparatus 100 may photograph the audience during the photographing period. For example, audiences may view all types of images such as TV, movies, plays, musicals, and performances, and the present invention is not limited thereto. When audiences watch a movie in a movie theater, the image capturing apparatus 100 may photograph the faces AF of the audience seated in the audience seat AS. In general, 200 or more audience seats may be arranged in a movie theater. For example, the image capturing device 100 may take a picture based on the capturing control information received from the image processing device 200 and generate a first image (ie, a basic image). That is, the first image may be an image photographing audiences watching a movie. For example, the capturing control information may include at least one of a capturing period, a resolution, a frame rate, and a codec of the image capturing apparatus 100. Depending on the embodiment, the image capturing apparatus 100 may include at least one camera.

The image processing apparatus 200 may divide the first image into a plurality of second images (ie, divided images) and store the plurality of second images. In addition, the image processing apparatus 200 may transmit a plurality of second images to the image analysis apparatus 300. In this case, the image processing apparatus 200 may transmit to the image analysis apparatus 300 according to a storage unit time. Through this, the image processing apparatus 200 according to an embodiment of the present invention divides the first image into a plurality of second images in advance before the viewer's viewing ends, and stores the first image in the image analysis device 300 in a unit time. By sending along, the analysis speed can be dramatically improved. Details of the image processing apparatus 200 will be described with reference to FIG. 3.

The image analysis apparatus 300 may perform emotion analysis on an image received from the image processing apparatus 200. In particular, the image analysis apparatus 300 may perform emotion analysis on a plurality of images received during the analysis period in parallel. In addition, the image analysis apparatus 300 may generate emotion information through emotion analysis. For example, the emotional information may include any one of general emotions such as indifference, boredom, joy, anger, sadness, joy, fear, love, disgust, hate, desire, etc. of a person, and the degree of each corresponding emotion It may further include a number expressed as a percentage.

That is, the image analysis device 300 analyzes the facial expressions of the faces (AF) of the audience included in each of the plurality of second images or the behavioral patterns of the audience (eg, yawning, conversation, away from the room, use of a mobile phone, etc.), It is possible to generate emotional information representing the emotions of the audience.

For example, the image analysis device 300 uses the types of facial expressions displayed on the faces of the viewers AF and the frequency of occurrence of the facial expressions, and the types of behaviors and the frequency of occurrences of the viewers' faces. You can analyze the facial expressions that appear and the behavior patterns of the audience.

In this case, emotion information may be generated for each of the plurality of second images. The image analysis device 300 may transmit the generated emotion information to the image processing device 200. Depending on the embodiment, the image analysis device 300 may be implemented as a separate server device. However, the present invention is not limited thereto, and according to embodiments, the image analysis apparatus 300 may be implemented as an apparatus of the emotion analysis system 10.

At least two or more of the image capturing apparatus 100, the image processing apparatus 200, and the image analysis apparatus 300 may transmit and receive data/signals through a wireless network or a wired network. In this case, the wireless network or the wired network is not particularly limited, and all types of network systems available within the range for achieving the object of the present invention may be applied.

Depending on the embodiment, at least two or more of the image capturing apparatus 100, the image processing apparatus 200, and the image analysis apparatus 300 may be integrated and implemented integrally.

3 is a diagram illustrating an image processing apparatus 200 according to an embodiment of the present invention. For convenience of explanation, the image capturing apparatus 100 and the image analysis apparatus 300 are shown together in FIG. 3.

Referring to FIG. 3, the image processing apparatus 200 may include a division unit 210, a transmission unit 220, a control unit 230, and an integration unit 240.

The dividing unit 210 may divide the first image IM1 into a plurality of second images IM2 and store the plurality of second images IM2. That is, the division unit 210 may divide the photographing area of the first image IM1 into a plurality of divided areas, and select at least one of the plurality of divided areas as an analysis target. The dividing unit 210 may generate and store a plurality of second images IM2 according to an analysis target. Details related to the division unit 210 will be described in FIG. 4.

When the plurality of second images IM2 are stored for a storage unit time, the transmitter 220 may transmit the stored second images IM2 to the image analysis apparatus 300. Details related to the transmission unit 220 are described in FIG. 8.

The controller 230 may control the operation of the image capturing apparatus 100. That is, the controller 230 may generate photographing control information (PCI) and transmit it to the image photographing apparatus 100. For example, the capturing control information may include at least one of a capturing period, a resolution, a frame rate, and a codec of the image capturing apparatus 100.

The integration unit 240 may receive the emotion information EI from the image analysis device 300. The emotion information EI corresponds to each of the plurality of second images IM2. Accordingly, the integration unit 240 may generate integrated emotion information corresponding to the first image IM1 by integrating the emotion information EI.

4 is a diagram illustrating a division part 210 according to an embodiment of the present invention. 5 to 7 are diagrams showing the operation of a division unit according to an embodiment of the present invention.

Referring to FIG. 4, the segmentation unit 210 may include a face recognition unit 211, a segmentation region setting unit 212, an analysis target selection unit 213, and an image storage unit 214.

The face recognition unit 211 may recognize faces of viewers included in the first image. In addition, the face recognition unit 211 may extract the number of audiences through face recognition.

The divided area setting unit 212 may compare the number of audiences and a reference value. In addition, the divided area setting unit 212 may divide the photographing area of the first image into a plurality of divided areas in different ways according to the comparison result. In this case, the reference value is a preset value and may have a value of 10 or less depending on the embodiment. Through this, the operation of the dividing unit 210 is different for the case where the number of spectators is less than or exceeding the reference value, so that limited performance resources can be used more efficiently.

The analysis target selector 213 may select at least one of a plurality of divided regions as an analysis target. That is, the analysis target selector 213 may select, from among a plurality of divided areas, areas to be analyzed as targets for analysis. In this case, the analysis target selector 213 may select at least one divided region as an analysis target from among a plurality of divided regions, using a separate user selection or information on a preset analysis target.

The image storage unit 214 may store a plurality of second images IM2 according to an analysis target. That is, the plurality of second images IM2 may correspond to at least one divided region selected as an analysis target, and the image storage unit 214 may store the plurality of second images IM2. Depending on the embodiment, the image storage unit 214 may be implemented as a separate storage device or a cloud server.

5 shows the operation of the division unit 210 when the number of spectators is less than or equal to the reference value. In this case, it is assumed that the number of audiences is 8 and the reference value is 10.

3 and 5, the face recognition unit 211 may recognize the faces of audiences and extract the number of audiences. The divided area setting unit 212 may compare the number of spectators extracted by the face recognition unit 211 with a reference value.

Since the number of audiences is smaller than the reference value, the divided area setting unit 212 may divide the photographing area PA into a plurality of divided areas DA1 to DA8 according to the number of audiences. In this case, the plurality of divided areas DA1 to DA8 correspond to each of the audience, and each of the plurality of divided areas DA1 to DA8 may represent the face of one audience.

Since the image analysis device analyzes the second images, each of which is generated according to the divided regions representing one viewer's face, accuracy and reliability will be improved compared to the case where multiple viewers' faces are included in one image. I can. Accordingly, through the above-described method, the emotion analysis system and its operation method according to an embodiment of the present invention, when the number of audiences is less than a certain number, generate an image for each of the audiences and perform analysis, thereby analyzing accuracy and reliability. Can improve.

6 shows the operation of the division unit 210 when the number of spectators is greater than the reference value. In this case, it is assumed that the number of audiences is at least 30 and the reference value is 10. And, it is assumed that the audience is evenly distributed in the audience seats.

3 and 6, the face recognition unit 211 may recognize the faces of audiences and extract the number of audiences. The divided area setting unit 212 may compare the number of spectators extracted by the face recognition unit 211 with a reference value.

Since the number of audiences is greater than the reference value, the divided area setting unit 212 may receive the number of divided areas from the user. That is, the user can arbitrarily set and input the number of divided areas. However, the present invention is not limited thereto, and according to embodiments, the number of divided regions may be set to a preset number.

The divided area setting unit 212 may divide the photographing area PA into a plurality of divided areas DA1 to DA4 according to the number of divided areas input from the user. In this case, the plurality of divided areas DA1 to DA4 correspond to a plurality of audiences based on the seating arrangement of the audience, and each of the plurality of divided areas DA1 to DA4 may represent faces of a plurality of audiences.

For example, the divided area setting unit 212 may set four divided areas by receiving the number of divided areas of 4 from the user and setting one horizontal line or vertical line for the photographing area PA. Accordingly, the photographing area PA may be divided into four divided areas DA1 to DA4 representing a plurality of audiences.

Since the association analysis device analyzes the second images generated according to the divided regions representing the faces of a plurality of viewers, the analysis speed may be improved compared to the case where one viewer's face is included in one image. . Therefore, through the above-described method, the emotion analysis system and its operation method according to an embodiment of the present invention, when the number of spectators is more than a certain number, generate an image for a plurality of spectators and perform analysis, thereby increasing the analysis speed. Can be improved.

7 shows the operation of the division unit 210 when the number of spectators is greater than the reference value. 7 shows an embodiment different from the embodiment shown in FIG. 6. The differences between the embodiments shown in FIGS. 6 and 7 are described below. In this case, it is assumed that the number of spectators is 20 and the reference value is 10. And, it is assumed that the audience is unevenly distributed in the audience seats.

3 and 7, the face recognition unit 211 may recognize the faces of audiences and extract the number of audiences. The divided area setting unit 212 may compare the number of spectators extracted by the face recognition unit 211 with a reference value.

Since the number of audiences is greater than the reference value, the divided area setting unit 212 may group adjacent audiences with each other. That is, the divided area setting unit 212 may set a group for audiences adjacent to each other.

According to an embodiment, the segmentation region setting unit 212 may group audiences adjacent to each other using a K-nearest neighbors algorithm or a K-means algorithm. . In this case, the K-nearest neighbor algorithm is an algorithm that classifies locally adjacent data according to a characteristic value of the data (eg, seat position). The K-means algorithm is an algorithm that combines given data into K clusters, and can operate in a manner that minimizes the variance of the difference between each cluster and the distance.

The divided area setting unit 212 may divide the photographing area PA into a plurality of divided areas DA1 to DA5 according to the set number of groups. In this case, the plurality of divided areas DA1 to DA5 correspond to adjacent audiences, and each of the plurality of divided areas DA1 to DA5 may represent faces of adjacent audiences.

For example, the divided area setting unit 212 may set five groups including adjacent audiences, and may set a plurality of divided areas corresponding to each group according to the set number of groups. Accordingly, the photographing area PA may be divided into five divided areas DA1 to DA5 representing groups of adjacent audiences.

Since the association analysis device analyzes the second images generated according to the divided regions representing the faces of a plurality of viewers, the analysis speed may be improved compared to the case where one viewer's face is included in one image. . Therefore, through the above-described method, the emotion analysis system and its operation method according to an embodiment of the present invention, when the number of spectators is more than a certain number, generate an image for a plurality of spectators and perform analysis, thereby increasing the analysis speed. Can be improved.

In addition, when the audience is unevenly distributed, if the divided regions are set according to the exemplary embodiment illustrated in FIG. 6, a plurality of empty seats are included in one second image, so that accuracy and reliability may be reduced. Accordingly, in the case of setting the divided regions according to the exemplary embodiment illustrated in FIG. 7, by minimizing empty seats in one second image, it is possible to prevent excessive and decrease in accuracy and reliability.

Depending on the embodiment, the embodiments shown in FIGS. 6 and 7 may be integrated and implemented as one. That is, when the number of spectators is greater than or equal to a certain number, the division unit 210 may analyze the distribution of spectators and adopt the embodiment of FIG. 6 differently from the embodiment of FIG.

8 is a diagram illustrating a transmission unit 220 according to an embodiment of the present invention.

Referring to FIG. 8, the transmission unit 220 may include a determination unit 221, a counter 222, and an image transmission unit 223.

The determiner 221 may determine the storage unit time. According to an embodiment, the determiner 221 may receive a storage unit time input from a user. For example, the determiner 221 may determine the storage unit time based on the user's input indicating the unit time. Through this, the emotion analysis system and the operation method thereof according to an embodiment of the present invention can improve user convenience.

According to another embodiment, the determiner 221 may determine the storage unit time to be set to a capacity of a plurality of second images equal to or less than the maximum processing capacity of the image analyzing apparatus 300 (see FIG. 1 ). That is, the image analysis apparatus 300 may have a limit on the capacity of an image that can be processed, and the capacity of the image may be determined by a shooting period, a resolution, a frame rate, a codec, and the like. Accordingly, the determiner 221 may determine a storage unit time so as not to exceed the maximum processing capacity of the image analysis apparatus 300, and the image analysis apparatus 300 may receive images in a storage unit time unit.

The counter 222 may count the storage unit time while storing the plurality of second images. That is, the counter 222 may count the storage unit time while the image storage unit 214 of the dividing unit 210 (refer to FIG. 4) stores a plurality of second images. The counter 222 may transmit a signal to the image transmission unit 223 when the storage unit time elapses according to the count result. In addition, the counter 222 may count the storage unit time again when the shooting period has not elapsed, and may stop counting when the shooting period has elapsed. Accordingly, the counter 222 may repeatedly counter the storage unit time during the photographing period.

When the storage unit time has elapsed, the image transmission unit 223 may transmit a plurality of stored second images to the image analysis apparatus 300. That is, the image transmission unit 223 determines whether the storage unit time has passed according to the signal of the counter 222, and when the storage unit time has passed, the plurality of images stored in the image storage unit 214 (refer to FIG. 4). The second images IM2 may be transmitted to the image analysis apparatus 300.

9 is a timing diagram illustrating the operation of the emotion analysis system according to an embodiment of the present invention.

Referring to FIGS. 1 to 9, audiences may view video content screened during the running time (RT).

In addition, the image capturing apparatus 100 may generate a first image IM1 by photographing audiences viewing the image content during the capturing period PT. In FIGS. 9 to 11, the running time (RT) and the photographing period (PT) are illustrated as being the same, but the present invention is not limited thereto. Depending on the embodiment, the running time (RT) and the photographing period (PT) may be different.

The image processing apparatus 200 may divide the first image IM1 into a plurality of second images IM2 and transmit the plurality of second images IM2 according to a storage unit time ST. That is, the image processing apparatus 200 may transmit the plurality of second images IM2 to the image analysis apparatus 300 every storage unit time ST.

The image analysis apparatus 300 receives a plurality of second images IM2 every storage unit time ST, and analyzes the facial expressions or behavior patterns of the viewers displayed in the plurality of second images IM2 during the analysis period AD. While you can analyze. 9, the image analysis apparatus 300 may generate emotion information EI by analyzing a plurality of second images IM2 transmitted every storage unit time ST. According to the embodiment illustrated in FIG. 9, the photographing period PT may partially overlap the analysis period AD. That is, the analysis period AD may be a period delayed by the storage unit time ST with respect to the photographing period PT. Through the above-described method, the emotion analysis system and the operation method thereof according to an embodiment of the present invention can improve the analysis speed by performing image analysis while viewers are viewing the image content.

10 is a timing diagram illustrating an operation of an emotion analysis system according to another embodiment of the present invention. In order to prevent duplication of description, a description will be made focusing on differences from the embodiment shown in FIG. 9.

1 to 10, the image analysis apparatus 300 of the emotion analysis system 10 according to the embodiment illustrated in FIG. 10 includes a plurality of second images IM2 received every storage unit time ST. The analysis may not be performed in the portion corresponding to the last storage unit time ST. The analysis period AD of the image analysis apparatus 300 may be terminated at a point in time when the photographing period PT of the image capture apparatus 100 ends. That is, the image analysis apparatus 300 may optionally terminate the analysis at the point in time when the photographing period PT of the image photographing apparatus 100 ends.

Accordingly, the emotion analysis system and its operation method according to the embodiment shown in FIG. 10 quickly derive the video analysis result, and furthermore, the running time (RT) of the video content ends, compared to the embodiment shown in FIG. 9. At the same time, by terminating the emotion analysis, the analysis result can be derived more quickly. In particular, the present embodiment may be used for a movie evaluation preview or performance rehearsal, and may have an advantage in that an analysis result can be immediately determined when viewing is terminated.

11 is a timing diagram illustrating an operation of an emotion analysis system according to another embodiment of the present invention. In order to prevent duplication of description, a description will be made focusing on differences from the embodiment shown in FIG. 9.

1 to 11, the image processing apparatus 200 of the emotion analysis system 10 according to the embodiment illustrated in FIG. 11 converts a plurality of second images IM2 to the image analysis apparatus 300 in real time. Can be transmitted. That is, in this case, the storage unit time ST may be set to a number substantially equal to zero. In this case, since the analysis is performed in real time, the analysis period AD of the image analysis apparatus 300 may be the same as the photographing period PT of the image capturing apparatus 100.

Accordingly, the emotion analysis system and its operation method according to the embodiment shown in FIG. 11 can quickly derive an image analysis result compared to the embodiment shown in FIG. 9, and compared to the embodiment shown in FIG. Reliability can be improved by performing an analysis without omission of expressions. In particular, this embodiment can be used for an evaluation preview of a genre movie, and has a strength in that it is possible to immediately find out a highly reliable analysis result when viewing is terminated.

12 is a diagram illustrating emotion information EI according to an embodiment of the present invention.

Referring to FIG. 12, the emotion information (EI) may include any one of general emotions such as indifference, boredom, joy, anger, sadness, joy, fear, love, disgust, hate, desire, indifference, boredom, etc. It may be, and may further include a numerical value representing the degree of each corresponding emotion in percent.

According to an embodiment, the emotion information EI may represent a person's momentary emotion, and according to another embodiment, the emotion information EI may represent an average emotion for a preset period (eg, storage unit time). have.

13 is a flowchart illustrating a method of operating the emotion analysis system 10 according to an embodiment of the present invention.

1 to 13, the image capturing apparatus 100 may generate a first image IM1 by photographing audiences during the capturing period PT (S10). For example, when the image capturing device 100 is installed in a movie theater, the audience can sit in the audience seat AS to watch a movie, and the image capturing device 100 shoots the audience to generate the first image IM1. have.

The image processing apparatus 200 may divide the first image IM1 and store a plurality of second images IM2 (S20). For example, the dividing unit 210 of the image processing apparatus 200 may divide and store the first image IM1 into a plurality of second images IM2.

When the plurality of second images IM2 are stored during the storage unit time ST, the image processing apparatus 200 may transmit the stored plurality of second images IM2 to the image analysis apparatus 300 (S30). That is, the transmission unit 220 of the image processing apparatus 200 may transmit the plurality of second images IM2 stored in the dividing unit 210 to the image analysis apparatus 300 every storage unit time ST.

The image analysis apparatus 300 may generate emotion information EI for each of the plurality of second images IM2 by analyzing the plurality of second images IM2 during the analysis period AD (S40 ). That is, the image analysis device 300 analyzes the facial expressions or behavior patterns of the faces AF of the viewers included in each of the plurality of second images IM2 during the analysis period, and the emotion information EI representing the emotions of the audience Can be created.

The image processing apparatus 200 may generate integrated emotion information corresponding to the first image IM1 by integrating the emotion information EI (S50). That is, the integration unit 240 of the image processing apparatus 200 may receive and integrate the emotion information EI. As a result, the integration unit 240 may generate integrated emotion information corresponding to the first image IM1.

14 is a flowchart illustrating in detail an operation method of the emotion analysis system 10 according to an embodiment of the present invention. In FIG. 14, the step S20 of storing the plurality of second images IM2 shown in FIG. 13 will be described in detail.

1 to 14, the division unit 210 may recognize the faces AF of the audience included in the first image IM1 and extract the number of audiences (S21). That is, the face recognition unit 211 of the segmentation unit 210 may extract the number of viewers by detecting faces included in the image using a general face recognition technology.

The dividing unit 210 may compare the number of audiences and a reference value, and divide the photographing area PA of the first image IM1 into a plurality of divided areas in a different manner according to the comparison result (S22). That is, the divided area setting unit 212 of the dividing unit 210 may divide the photographing area PA of the first image IM1 into a plurality of divided areas in different ways according to the comparison result.

The division unit 210 may select at least one of the plurality of divided areas as an analysis target (S23). That is, the analysis target selection unit 213 of the dividing unit 210 may select regions to be analyzed from among a plurality of divided regions as analysis targets. In this case, the analysis target selector 213 may select at least one divided region as an analysis target from among a plurality of divided regions, using a separate user selection or information on a preset analysis target.

The dividing unit 210 may store a plurality of second images IM2 according to an analysis target (S24). That is, the image storage unit 214 of the dividing unit 210 may store a plurality of second images IM2 according to an analysis target.

15 is a flowchart illustrating in detail a method of operating an emotion analysis system according to an embodiment of the present invention. In FIG. 15, an exemplary embodiment of dividing the photographing area PA shown in FIG. 14 into a plurality of divided areas (S22) will be described in detail.

1 to 15, the division unit 210 may compare the number of audiences and a reference value (S221). That is, the divided area setting unit 212 of the divided unit 210 may compare the number of audiences and a reference value.

When the number of audiences is less than or equal to the reference value (NO in S222), the division unit 210 may divide the photographing area PA into a plurality of divided areas according to the number of audiences (S223).

When the number of spectators is greater than the reference value (YES in S222), the segmentation unit 210 may receive the number of segmented regions from the user (S224).

In addition, the dividing unit 210 may divide the photographing area PA into a plurality of divided areas according to the number of divided areas (S225).

The operation method illustrated in FIG. 15 may be an operation implemented by a combination of the embodiments illustrated in FIGS. 5 and 6.

16 is a flowchart illustrating in detail a method of operating an emotion analysis system according to another embodiment of the present invention. In FIG. 16, another embodiment of the step S22 of dividing the photographing area PA shown in FIG. 13 into a plurality of divided areas (S22) will be described in detail.

16 shows an embodiment different from the embodiment shown in FIG. 15. In order to prevent duplication of description, the description will focus on differences between the embodiments shown in FIGS. 15 and 16.

Referring to FIGS. 1 to 16, when the number of audiences is greater than the reference value (YES in S222), the divider 210 may group adjacent audiences with each other (S226). According to an embodiment, the segmentation region setting unit 212 may group audiences adjacent to each other using a K-nearest neighbors algorithm or a K-means algorithm. .

In addition, the dividing unit 210 may divide the photographing area PA into a plurality of divided areas according to the number of generated groups (S227).

The operation method illustrated in FIG. 16 may be an operation implemented by a combination of the embodiments illustrated in FIGS. 5 and 7.

17 is a flowchart illustrating in detail a method of operating an emotion analysis system according to an embodiment of the present invention. In FIG. 17, an operation S30 of transmitting a plurality of stored second images IM2 shown in FIG. 14 to the image analysis apparatus 300 will be described in detail.

1 to 17, the transmission unit 220 may determine a storage unit time ST (S31). According to an embodiment, the determination unit 221 of the transmission unit 220 may receive a storage unit time ST from a user, and determine the storage unit time ST based on this. According to another exemplary embodiment, the determiner 221 may determine the storage unit time ST to be set so that the capacity of the plurality of second images IM2 is less than or equal to the maximum processing capacity of the image analysis apparatus 300.

The transmission unit 220 may count the storage unit time ST while the dividing unit 210 stores a plurality of second images (S32). That is, the counter 222 of the transmission unit 220 may count the storage unit time ST while the image storage unit 214 of the dividing unit 210 stores a plurality of second images.

When the storage unit time ST does not exceed (NO in S33), the transmission unit 220 may count the storage unit time ST again (S32).

When the storage unit time ST exceeds (YES in S33), the transmission unit 220 may transmit a plurality of stored second images IM2 (S34). That is, the image transmission unit 223 of the transmission unit 220 may transmit a plurality of stored second images to the image analysis apparatus 300 when the storage unit time ST has passed.

When the photographing period PT has not elapsed (NO in S36), the transmission unit 220 may count the storage unit time ST again (S32).

When the shooting period PT has elapsed (YES in S36), the transmission unit 220 may end the count.

That is, the counter 222 of the transmission unit 220 can count the storage unit time (ST) again when the recording period (PT) has not elapsed, and can stop counting when the recording period (PT) has elapsed. have. Accordingly, the counter 222 may repeatedly counter the storage unit time ST during the photographing period PT.

The emotion analysis system and its operation method according to an embodiment of the present invention may capture audiences while video content is being screened, and analyze emotions based on their facial expressions and behavior patterns.

In addition, the emotion analysis system and the operation method thereof according to an exemplary embodiment of the present invention may improve analysis speed by dividing a face image into a plurality of divided images according to a face recognition result and transmitting the divided images to the image analysis apparatus.

In addition, the emotion analysis system and the operation method thereof according to an embodiment of the present invention can improve the analysis speed by the image analysis apparatus performing analysis on a plurality of divided images in parallel.

In addition, the emotion analysis system and the operation method thereof according to an embodiment of the present invention may complete the analysis while the video content is being screened.

The functional operations described in this specification and embodiments related to this subject are implemented in digital electronic circuits, computer software, firmware, or hardware, including structures disclosed in this specification and structural equivalents thereof, or in a combination of one or more of them. It is possible.

Embodiments of the subject matter described herein include one or more of a computer program product, i.e., one or more relating to computer program instructions encoded on a tangible program medium for execution by or to control its operation It can be implemented as a module. The tangible program medium may be a radio wave signal or a computer-readable medium. A radio wave signal is an artificially generated signal, such as a machine-generated electrical, optical or electromagnetic signal, generated to encode information for transmission to an appropriate receiver device for execution by a computer. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a combination of materials that affect a machine-readable radio wave signal, or a combination of one or more of them.

A computer program (also known as a program, software, software application, script or code) can be written in any form of a compiled or interpreted language or a programming language, including a priori or procedural language, and can be written as a standalone program or module, It can be deployed in any form, including components, subroutines, or other units suitable for use in a computer environment.

Computer programs do not necessarily correspond to files on the file device. A program may be in a single file provided to the requested program, or in multiple interactive files (e.g., files in which one or more stores modules, subprograms, or portions of code), or in files that hold other programs or data. Some (eg, one or more stored within a markup language document may be stored within a script).

The computer program may be deployed to run on one computer or multiple computers located at one site or distributed across a plurality of sites and interconnected by a communication network.

Additionally, the logical flows and structural block diagrams described in this patent document describe the corresponding actions and/or specific methods supported by the corresponding functions and steps supported by the disclosed structural means. It can also be used to build software structures and algorithms and their equivalents.

One or more of the processes and logic flows described herein may be executed by a programmable processor, one or more executing a computer program in order to perform a function by operating on input data and generating an output.

Processors suitable for execution of computer programs include, for example, both general purpose and special purpose microprocessors and any one or more of any type of digital computer being a processor. In general, the processor will receive instructions and data from read-only memory, random access memory, or both.

The key elements of a computer are one or more memory devices for storing instructions and data, and a processor for carrying out the instructions. In addition, computers are generally operable to receive data from, transfer data to, or perform both of the mass storage devices, such as one or more for storing data, such as magnetic, magneto-optical disks or optical disks. Combined or will include. However, computers do not need to have such devices.

The present description presents the best mode of the invention, and provides examples to illustrate the invention and to enable those skilled in the art to make and use the invention. The written specification is not intended to limit the present invention to the specific terms presented.

Although the above has been described with reference to the preferred embodiments of the present invention, those skilled in the art or those of ordinary skill in the art will not depart from the spirit and scope of the present invention described in the claims to be described later. It will be appreciated that various modifications and changes can be made to the present invention within the scope of the invention.

Accordingly, the technical scope of the present invention should not be limited to the contents described in the detailed description of the specification, but should be determined by the claims.

10: emotion analysis system 100: video capture device
200: image processing device 210: division unit
220: transmission unit 230: control unit
240: integration unit 300: image analysis device

Claims (20)

An image photographing unit that photographs the audience during the photographing period and generates a first image;
An image processing device that divides the first image into a plurality of second images and transmits the plurality of second images according to a storage unit time; And
Including an image analysis device for analyzing the plurality of second images during the analysis period to generate emotion information for each of the plurality of second images,
The image processing device,
A dividing unit for dividing the first image and storing the plurality of second images; And
When the plurality of second images are stored for the storage unit time, a transmission unit for transmitting the stored plurality of second images to the image analysis device,
The division part,
A face recognition unit for recognizing the faces of the audience included in the first image and extracting the number of the audience;
A divided area setting unit that compares the number of audiences and a reference value, and divides the photographing area of the first image into a plurality of divided areas in different ways according to the comparison result;
An analysis target selection unit selecting at least one of the plurality of divided regions as an analysis target; And
And an image storage unit configured to store the plurality of second images according to the analysis target.
delete The method of claim 1,
If the number of audiences is less than or equal to the reference value,
The divided area setting unit divides the photographing area into the plurality of divided areas according to the number of the audience,
Each of the plurality of divided areas represents a face of one audience. The method of claim 3,
When the number of audiences is greater than the reference value,
The divided area setting unit receives the number of divided areas from a user, and divides the photographing area into the plurality of divided areas according to the received number of divided areas. The method of claim 3,
When the number of audiences is greater than the reference value,
The divided area setting unit groups adjacent audiences with each other, and divides the photographing area into the plurality of divided areas according to the number of generated groups. The method of claim 5,
The divided region setting unit generates a plurality of groups by grouping adjacent audiences using a K-average algorithm for the audiences. An image photographing unit that photographs the audience during the photographing period and generates a first image;
An image processing device that divides the first image into a plurality of second images and transmits the plurality of second images according to a storage unit time; And
An image analysis device that analyzes the plurality of second images during an analysis period and generates emotion information for each of the plurality of second images,
The image processing device,
A dividing unit for dividing the first image and storing the plurality of second images; And
When the plurality of second images are stored for the storage unit time, a transmission unit for transmitting the stored plurality of second images to the image analysis device,
The transmission unit,
A determination unit determining the storage unit time;
A counter for counting the storage unit time while storing the plurality of second images; And
When the storage unit time has elapsed, the stored plurality of second images include an image transmission unit to an image analysis device,
And the counter counts the storage unit time again when the photographing period has not elapsed. The method of claim 7,
And the determination unit receives the storage unit time from a user. The method of claim 7,
And the determination unit determines the storage unit time so that the capacity of the plurality of second images is set to be less than or equal to the maximum processing capacity of the image analyzing apparatus. The method of claim 7,
And the counter stops counting when the photographing period has passed. Photographing the audience during the shooting period and generating a first image;
Dividing the first image and storing a plurality of second images;
If the plurality of second images are stored for a storage unit time, transmitting the stored plurality of second images to an image analysis device;
Generating emotion information for each of the plurality of second images by analyzing the plurality of second images during an analysis period; And
Including the step of integrating the emotion information to generate integrated emotion information,
Storing the plurality of second images,
Recognizing the faces of the audience included in the first image and extracting the number of the audience;
Comparing the number of audiences and a reference value, and dividing the photographing area of the first image into a plurality of divided areas in different ways according to the comparison result;
Selecting at least one of the plurality of divided regions as an analysis target; And
And storing the plurality of second images according to the analysis target.
delete The method of claim 11,
Dividing the photographing area into the plurality of divided areas,
Comparing the number of audiences and a reference value; And
When the number of audiences is less than or equal to the reference value, dividing the photographing area into the plurality of divided areas according to the number of audiences,
Each of the plurality of divided areas is represented by a face of one audience. The method of claim 13,
Dividing the photographing area into the plurality of divided areas,
When the number of audiences is greater than the reference value, receiving the number of divided areas from a user, and dividing the photographing area into the plurality of divided areas according to the input number of the divided areas. How the emotion analysis system works. The method of claim 13,
Dividing the photographing area into the plurality of divided areas,
When the number of audiences is greater than the reference value, grouping the audiences adjacent to each other, and dividing the photographing area into the plurality of divided areas according to the number of generated groups. Method of operation. The method of claim 11,
Transmitting the stored plurality of second images,
Determining the storage unit time;
Counting the storage unit time while storing the plurality of second images; And
When the storage unit time has elapsed, transmitting the stored plurality of second images,
When the photographing period has not elapsed, the operation method of the emotion analysis system proceeds again to the step of counting the storage unit time. The method of claim 16,
The determining of the storage unit time comprises receiving the storage unit time from a user. The method of claim 16,
The determining of the storage unit time comprises determining the storage unit time so that the capacities of the plurality of second images are set to be less than or equal to the maximum processing capacity of the image analyzing apparatus. The method of claim 11,
The operation method of the emotion analysis system, characterized in that the photographing period partially overlaps the analysis period. The method of claim 19,
The operation method of the emotion analysis system, characterized in that the photographing period and the analysis period are the same.

Images