KR20090073294A - Method for social network analysis based on face recognition in an image or image sequences - Google Patents

Abstract

A human mutual relation inference method in an image through face recognition for easily searching relation between specific human and the other human is provided to recognize the relation between a person and the other of large amount of images through face recognition. One or more video information is inputted(S10). A face is detected from the inputted video information(S20). A person is recognized by labeling the detected face as identification value(S30). The frequency value of a relation frequency matrix is increased about the recognized individual(S60). Persons appearing on a screen is grouped(S70). The relation between a plurality of individuals is presumed to be through the matrices and the clustered data(S80).

Description

Method for Social Network Analysis Based On Face Recognition In An Image or Image Sequences}

The present invention relates to a human correlation inference method, and more particularly, to a method of recognizing a human face existing in image information and inferring human correlation in one or more images.

The recent development of communication technology has made much of the communication and relationship-based activities between people through the communication network. Internet blogs and e-mails are a representative example of communication activities based on communication networks. In particular, Facebook and cyworld in Korea are often video-based activities, not text-based online social activities. Among them, the face of the person is often the center, and the person needs to infer the relationship between the people through the example of several pictures without any learning process. Grouping of figures, estimation of connected persons between clusters, and estimation of persons who can integrate a large number of clusters into the least visits.

In general, the field of image retrieval has been actively studied for a set of images in which a specified input face appears for a single face. The field of image retrieval uses the same face recognition technology, of course, but is limited to 1: 1 retrieval by probabilistic calculation.

Related prior arts include a face recognition method using a combination of neural networks and hidden Markov models (Korean Patent Laid-Open Publication No. 1999-0086440), which extracts and extracts T feature vectors by acquiring a plurality of face images for different people. Each of the subjects is registered by training a neural network using half of the T feature vectors, and the trained neural network is provided with the remaining half of the feature vectors to be associated with the registered identifiers and the other half of the feature vectors. Generating a related data file and training the hidden Markov model using half the file data from the generated related data file to generate a model parameter with the smallest convergence value per subject.

In addition, the block matching-based facial feature tracking method (Korean Patent Laid-Open Publication No. 2002-0080064) detects a component of a subject face, detects position information of a component of the detected face, and normalizes the detected position information to track a feature vector. Is calculated to remove noise and the like, and to perform effective space definition and quantization of a feature vector according to facial expression recognition to recognize facial expressions based on a hidden Markov model.

Therefore, the above-described conventional technology mainly focuses on face recognition, and thus, it is difficult to grasp the relationship between a plurality of people existing in one image screen. In addition, when there are not only relationships between people in the same image but also a person who existed in one image screen in another image, it is difficult to find a connection point between such people and to identify a relationship between related people.

The present invention has been made to solve the above-described problems of the prior art, and automatically grasps the interconnection relationship between humans in the image information from one or a plurality of image information, and makes a database thereof to establish a database with a specific human and other humans. It is an object of the present invention to provide a method of inferring human interrelationship in images through face recognition which makes it easy to search for a relationship.

According to an aspect of the present invention, there is provided a method for inferring human correlation in an image through face recognition according to an embodiment of the present invention. A first step of receiving one or more image information, a second step of detecting a face from the input image information, A third step of recognizing the person by labeling the face detected in the second step with an eigen value, a fourth step of increasing the frequency of the relationship frequency matrix for the person recognized in the third step and clustering among the people appearing on the screen And a fifth step of estimating a relationship between a plurality of persons using the relationship frequency matrix and the clustered data.

Here, the method of inferring human correlation in the image through the face recognition repeats the second to third steps until the faces of all persons included in the input image information are detected.

The second step includes a first step of detecting a face candidate area from the input image and a second step of filtering an area other than the face by using a face size and a skin color among the face candidate areas.

The third step is to recognize a person using an eigenfaces recognition method, or to recognize a person using a Hidden Markov Model.

The fourth step further includes receiving an identification value of the corresponding person including a name from the user when the person recognized in the third step is not a person existing in the existing database.

The relationship frequency matrix of the fourth step is a matrix whose row is i and the column j is a _ij , where i, j is an identification number of a person, and a _ij is j when a person appears. The frequency of the appearance of a person.

The fourth step updates the relationship frequency matrix in one still image unit.

In the fourth step, when the person recognized in the third step is not a person existing in the existing database, the total size of the rows and columns of the relationship frequency matrix is increased by one and a new identification number is assigned to the recognized person. do.

The fourth step increases the corresponding frequency value of the relationship frequency matrix by 1 between persons included in one still image.

In the fourth step, people recognized in one still image are clustered into one group.

The fifth step may be a first process of setting a person appearing in common among different groups of persons as a connection point, and the relationship frequency matrix between the person set as the connection point and other people included in at least one group to which the person belongs. Estimate the relationship according to the frequency value of.

In the method of inferring human interaction in an image through face recognition according to the present invention configured as described above, a method of reaching a relationship from one person to another person in a plurality of images through face recognition may be found. This can be found through video without investigating the relations of specific members of the society, and by getting relationship information about the characters appearing from several images of their own, they have not been aware of their existing relationships. New parts can be found. In addition, it is possible to make a positive contribution to the field of forensic science and cognitive science by proposing potential relation inference method for the images which will be infinite in the future. The present invention can also contribute to academic fields such as artificial intelligence, psychology, neuroscience and the like.

Hereinafter, with reference to the accompanying drawings illustrating a specific configuration and embodiment of the present invention.

The present invention relates to a method of detecting human correlation in an image using a face recognition method. In the present invention, rather than 1: 1 search, the faces appearing in the image are modeled in a certain relationship, and the characters intersecting in two different images are the main feature points. One human network structure can be identified. This is similar to how a person uses the brain to define a relationship. In the present invention, an artificial intelligence algorithm is used to detect and recognize a main body feature of a face in a designated image.

1 is a flowchart illustrating a method of inferring human correlation in an image through face recognition according to the present invention.

Referring to FIG. 1, first, at least one image information is input (S10).

The input image information may be one still image or a moving image.

In operation S20, a face is detected from the input image information.

If the input image is a still image, the faces of people appearing in the still image are detected, and in the case of a video, the faces of people appearing in each still image for a plurality of still images sampled at each frame or at regular intervals. Is detected.

The process of detecting a face is performed as follows.

The face candidate region is first detected from the input image, and the final face region is detected by filtering to remove a region other than the face from the face candidate region.

The face candidate region is determined to be not a face through a subsequent filtering process or trimmed to a more accurate face region.

In this case, after detecting the face candidate region, it is necessary to correct the image in order to detect a more accurate face region. In this case, the face candidate region may be normalized by using variance of brightness values. In other words, by using the leveling of the brightness histogram, an excessively dark or too bright part of the face candidate area is corrected.

Next, a part other than the face is removed from the face candidate areas, and the present invention provides a color gamut (skin color) based filtering method and a size based filtering method to remove an incorrectly extracted area from among extracted candidate areas. Use

The skin color base is not used as the face candidate area when the skin color probability of the detected face candidate area is less than a predetermined value (threshold value) because the skin color is prominent in the image due to the physical feature of the face.

The size-based method uses a method that sets a size that a face can have in the image as a threshold even if an object having a face-like feature is detected by accident and does not use it as a face candidate area if it falls short of this.

That is, a predetermined reference face size is determined and a face candidate area smaller than the reference face size is excluded from consideration, or only a more accurate face area is detected by removing portions other than skin color among the face candidate areas.

After detecting the face region as described above, the detected face is labeled with a unique value to recognize a person (S30).

In this case, a variety of existing methods may be used to distinguish faces by recognizing faces. Here, face recognition has the same meaning as person recognition.

Face recognition is a method of personal recognition through face biometric information. For face recognition, first, an area corresponding to a face is detected in one image as described above, and information about the area is transmitted to the pattern recognizer.

The pattern recognizer extracts and identifies the eigenvalues from the input feature.

It can be classified as follows according to the feature (Feature) to be a pattern recognizer.

The holistic approach is to deliver the entire face image to the pattern recognizer. A representative example of this method is the eigenface recognition method. Eigenface is derived from the concept of reconstructing the remaining vectors of the same linear space using eigenvectors in linear algebra, assuming that all faces can be represented by linear combinations of eigenfaces. How to perform face recognition. Methods using this include Fisherface, Support Vector Machines, Nearest Feature Lines (NFL), and Independent Component Analysis. They are all based on Principal Component Analysis (PCA). PCA is a technique that simplifies the data set to a lower level during an iterative learning process.

Next, the feature-based approach uses the information of the face image interpreted through various image processing techniques. This interpreted information is called a feature. The information on this feature mainly represents facial structural information such as nose, eyes, distance or angle information between the eyes. The disadvantages of this approach are that they have different structures at different points in time and it is difficult to find feature points that are invariant to lighting conditions.

Next, the complex approach is the most similar to human face recognition, using a combination of the two methods mentioned above. Probably the highest confidence, but the largest amount of computing.

As an example of an artificial intelligence algorithm for face recognition, a hidden Markov model may be used.

After filtering, the final face area is normalized and passed to the Hidden Markov Model.

The hidden Markov model analyzes the face candidate region and returns a predetermined coefficient value, which can be distinguished by using the coefficient value as a unique value for identification.

In this case, the name or nickname of the person may be input from the user interface, and the person may be distinguished using the unique value.

The present invention is not limited to the above-described method, and a person may be identified by analyzing a face image using various other methods.

When the person is identified in this way, it is determined whether the identified person is a person existing in the existing database or a person newly appearing (S40).

If the newly recognized person is assigned a new identification number (S50).

Next, the frequency value of the relationship frequency matrix is increased for the recognized person and clustered among people appearing on one screen (S60 and S70).

As described above, the present invention uses a relationship frequency matrix in order to grasp the relationship between the various persons assigned the identification number.

The parameters related to the relationship frequency matrix are as follows.

N is the number of characters recognized and stored, and when i person appears, it is expressed as a _ij = p [R _j | R _i ], which is the probability or frequency of j person appearing, and R _i is unique to i person. It refers to a value, such as the coefficient value of a hidden Markov model or the name of a set person.

Further, A, which is a relation frequency matrix, is a matrix (N × N) in which a row is i, a column is j, and a matrix element is a _ij . I and j are both values 1 to N (1 ≦ <i, j ≦ <N), and are identification numbers of the respective persons.

Each time a new person is recognized, N, the maximum value of i and j, increases, and the new person has one more identification number than the last recognized person's identification number.

Relation Features having the above parameters are distributed in three-dimensional space, and each dimension element has (i, j, a _ij ). This is illustrated in FIG. 2, and FIG. 2 is a diagram showing an example of distribution of relationship features.

Matrix A Is a matrix representing the frequency of contact between one person and another and means the number of appearances in the image.

3 is a diagram illustrating an algorithm for creating a relationship frequency matrix.

Referring to FIG. 3, how the relationship frequency matrix is generated and updated will be described.

First, a face of a person appearing in one still image is recognized and an identification number is sequentially assigned to the recognized person.

FIG. 4 is a diagram illustrating an example in which an identification number is assigned to a person appearing in an arbitrary first image, and FIG. 5 is a diagram illustrating an example of a relationship frequency matrix composed of persons appearing in the first image.

Referring to FIG. 4, when three persons are recognized, the characters are checked to match those stored in the database, and in the case of a new person, the three persons are given unique identification numbers. That is, as shown in FIG. 4, 1, 2, and 3 are assigned, and the rows and columns of the relationship frequency matrix are expanded by the number of newly searched persons.

If there is no relation frequency matrix, the relation frequency matrix A consisting of three rows and three columns is newly created as shown in FIG. 5, and the matrix elements of the newly generated portion are initialized to zero.

In addition, the frequency value is increased by 1 in the matrix element where the characters appearing in the first image intersect. When this process is performed for all the figures, the completed relationship frequency matrix is created as shown in FIG. 5.

Referring to FIG. 5, the first, second and third persons have a frequency of one each. That is, it is recorded that one appearance appeared in one image.

Next, a method of adding a person recognized for the new second image to the relationship frequency matrix completed in the first image will be described.

FIG. 6 is a diagram illustrating an example in which an identification number is assigned to a person appearing in a second image, and FIG. 7 is a diagram illustrating an example of a relationship frequency matrix composed of people appearing in a second image.

Referring to FIG. 6, first, a person in a second image is recognized. The recognized person is searched with the person stored in the existing database to determine who the new person is.

In the second image, the first person is an existing person, and the other person is assumed to be a newly recognized person, and thus a new identification number is assigned to the newly recognized person. Since the number is given up to 3 times in the previous still image, the identification number is assigned as 4 and 5.

Next, the relationship frequency matrix A needs to be updated. The relationship frequency matrix was previously a 3x3 matrix, but since two new characters appeared, we increment the rows and columns by 2 each. In the case of updating the relationship frequency matrix after identifying each person, the number of rows and columns is increased once for person 4, and the rows and columns are increased once more for person 5.

FIG. 7 illustrates a relationship frequency matrix updated for two newly recognized persons as described above. Person 1 has a frequency value of 2, 3, 4, and 5 and once, and 2 and 3 have no relationship frequency with 4 and 5.

As such, the relationship frequency matrix may record frequency values among a plurality of persons appearing in a plurality of still images.

In addition, image information and person information of persons 1, 4, and 5 appearing together in the second image are grouped and stored in a database.

When a relationship frequency matrix is generated for one still image, the figures appearing in the still image are clustered together. In the first embodiment of the clustering, the identification value of the still image in which the persons appear and the information on who the person appeared in the still image are grouped and stored in the database (S70).

In a second embodiment of clustering, all the figures recognized through the image are made into a graph G = (V, E) of vertices V and edges E, and one or more groups are removed by removing predetermined edges from the graph. Can be made with The detailed description is as follows.

8 is a diagram illustrating an example of two groups for relationship estimation.

In general, experience defines a group as a group of people who have strong connections within the group and weak connections outside the group. Such social networks are easy to interpret using graph theory.

Therefore, one social relationship is defined as follows.

The graph G = (V, E) means the whole set of social relations (the whole set of figures recognized by the image), V means the vertices (individuals recognized), and E means the edges between each vertex (between the recognized figures). Relationship).

In the whole relationship set, dividing into N groups based on the group definitions defined above is essential for the social relationship structure analysis.

When going from one vertex to another, there is a characteristic that the vertices most likely to pass through have the largest betweenness value. The inter-value is defined as in Equation 1 below.

σ _st (= σ _ts ) is the number of paths from vertex s∈V to t∈V in graph G = (V, E), and σ _st (v) is the shortest path from vertex s to t number of paths through vertex v). C _B (v) is an evaluation function that is a measure of the impact of one vertex in a set of graphs.

This relationship is equally applicable to edges as well as vertices. In a given graph, σ _st (e) is the number of paths along the edge e of the path from vertex s to t, and s and t are vertices at both ends of the edge e. The shortest distance between each vertex described above is obtained using the relationship frequency matrix obtained above.

The definition of the betweenness value of the edge is shown in Equation 2 below.

Here, C _B (e) means the number of the shortest distance passing through the trunk line e. According to the definition of a group mentioned above, the method of determining a group is to remove edges having high inter-values. This is because the edge e having a high interstitial value can be regarded as a connection between groups. However, within the group, the inter-value is lower than the connection between groups. This property can be used to subdivide groups. In addition, the number of groups can be adjusted by removing edges sorted in descending order by the magnitude of the values.

Referring to FIG. 8, FIG. 8 shows two groups before splitting, where there are four vertices including a vertex on the left and seven vertices including b vertex on the right, connected by edge e. have. Edge e can be said to have a high value. That is, the edge e may be the most used in the case of going from any one vertex in the left group to any one vertex in the right group, so the value between the edges e is the largest. Thus, if the edge e is removed, it can be divided into two groups.

The process of constructing the relationship frequency matrix (S60) and the clustering process (S70) described above may be performed separately each time one person is recognized in the still image, or after all persons in the still image are recognized. It may also be done at the same time.

Next, the relationship between a plurality of persons may be estimated using the relationship frequency matrix and the clustered data.

To this end, first, people who appear in common among different groups of people are set as connection points, and according to the frequency value of the relationship frequency matrix between the person set as the connection point and other people included in at least one group to which the person belongs. Estimate the relationship.

That is, in the present invention, it may be determined that the higher the frequency value of the relationship frequency, the higher the intimacy between the two persons. If the frequency of interaction with each other is high, the probability of appearing in the same picture is high, and people appearing in one picture are regarded as a relationship known to each other.

Search the grouping information stored in the database to estimate the relationship, extract the characters that appear in several groups, and identify the identification values of the images in which the persons appear and the frequency of relations between other persons who appear like the persons Link back and store in the database.

The present invention may be configured to estimate a relationship between a specific person according to grouping information and a relationship frequency matrix stored in a database by receiving various questions or identifiers from a user through a user interface, and output the result.

Relation frequency matrix A and clustering information can be variously interpreted as follows through graph theory and clustering algorithm.

1. the number of people to pass from the set person i to person j and their estimates

2. Estimate the set of people who appear together

3. Estimating the person who is the link between the estimated set of people

4. Estimating a person that includes the largest group of people in each connection

This analysis result can be provided to the user through the user interface.

As described above, the method of inferring human correlation in an image through face recognition according to the present invention has been described with reference to the illustrated drawings, but the present invention is not limited by the embodiments and drawings disclosed herein, and the technical idea is protected. It can be applied within the range.

1 is a flowchart illustrating a method of inferring human correlation in an image through face recognition according to the present invention;

2 is a diagram showing an example of distribution of relationship features;

3 is a diagram illustrating an algorithm for creating a relationship frequency matrix;

4 is a diagram illustrating an example in which an identification number is assigned to a person appearing in an arbitrary first image;

5 is a diagram illustrating an example of a relationship frequency matrix composed of persons appearing in a first image;

6 is a diagram illustrating an example in which an identification number is assigned to a person appearing in an arbitrary second image;

7 is a diagram illustrating an example of a relationship frequency matrix composed of persons appearing in a second image;

8 is a diagram illustrating an example of two groups for relationship estimation.

Claims (18)

A first step of receiving one or more image information; A second step of detecting a face from the input image information; A third step of recognizing a person by labeling the face detected in the second step with a unique value; A fourth step of increasing the frequency value of the relationship frequency matrix with respect to the person recognized in the third step and clustering the people appearing on the screen; And And a fifth step of estimating a relationship between a plurality of persons using the relationship frequency matrix and the clustered data. The method according to claim 1, In the method of inferring human correlation in the image through the face recognition, the steps 2 through 3 are repeated until the faces of all persons included in the input image information are detected. Inference method of human correlation in image. The method according to claim 1, The method of claim 1, wherein the image information is a still image or a moving image. The method of claim 1, wherein the second step Detecting a face candidate region from the input image; And And a second process of filtering an area other than the face by using a face size and a skin color among the face candidate areas. The method according to claim 4, And in the second step, normalizing the face candidate region by using variance of brightness values after the first process. The method according to claim 1, The third step is a method of inferring human interrelationship in the image through the face recognition, characterized in that for recognizing the person using the eigenfaces recognition method. The method according to claim 1, The third step is a method of inferring human interrelationship in an image through face recognition, characterized by recognizing a person using a hidden Markov model. The method according to claim 1, The relationship frequency matrix of the fourth step is a matrix whose row is i and the column j is a _ij , where i, j is an identification number of a person, and a _ij is j when a person appears. A method of inferring human interrelationships in an image through face recognition, characterized by a frequency value at which a person appears. The method according to claim 8, And the fourth step is to update the relationship frequency matrix in units of one still image. The method according to claim 8, In the fourth step, when the person recognized in the third step is not a person existing in the existing database, the total size of the rows and columns of the relationship frequency matrix is increased by one and a new identification number is assigned to the recognized person. A method of inferring human interrelationship in an image through face recognition, characterized in that for performing. The method according to claim 10, The fourth step further includes receiving an identification value of a corresponding person including a name from the user when the person recognized in the third step is not a person existing in an existing database. A method of inferring human interaction in images through recognition. The method according to claim 8 or 10, The fourth step is a method of inferring human interrelationship in the image through the face recognition, characterized in that for increasing the corresponding frequency value of the relationship frequency matrix by one between the persons included in one still image. The method according to claim 1, The fourth step is a method of inferring human interrelationship in the image through the face recognition, characterized in that the group of people recognized in one still image grouped into one group. The method of claim 13, wherein the fifth step A first process of setting a person appearing in common among different groups of people as a connection point; And Inferring human correlation in an image through face recognition comprising a second process of estimating a relationship according to a frequency value of the relationship frequency matrix between a person set as the connection point and other people included in at least one group to which the person belongs Way. The method according to claim 14, The second process is a method of inferring human interrelationship in an image using face recognition, characterized in that to estimate the intimacy relationship between two people. The method according to claim 1, The fourth step divides the entire person recognized in the plurality of still images into one or more groups, and divides the edges between two predetermined people into one or more groups by using the shortest distance between two persons based on the relationship frequency matrix. A method of inferring human interaction in an image using face recognition, characterized in that the image. The method of claim 16, wherein the fourth step is A first process of setting all the figures recognized in the plurality of still images to a graph G = (V, E), where V is a vertex and E is an edge; Σ _st (the number of paths along any edge e of the shortest path from any vertex s to another arbitrary vertex t in the graph G = (V, E) using the relationship frequency matrix) e) a second process of finding (where σ _st (e) = σ _ts (e)); A third step of calculating and descending a value of an evaluation function as shown in the following equation with respect to the graph G = (V, E);

A fourth step of dividing the evaluation function values and the edges e sorted in descending order into one set and dividing the edges e having the largest evaluation function value in order and grouping them by a predetermined number of groups; Human interaction relation inference method in the image through the face recognition comprising a. The method according to claim 16 or 17, The fifth step is a method of inferring human interrelationship in the image through the face recognition, characterized in that for estimating the relationship between people belonging to the same group and people belonging to different groups.

Images