CN113052245B - Image clustering method and device, electronic equipment and storage medium - Google Patents
Image clustering method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN113052245B CN113052245B CN202110340584.7A CN202110340584A CN113052245B CN 113052245 B CN113052245 B CN 113052245B CN 202110340584 A CN202110340584 A CN 202110340584A CN 113052245 B CN113052245 B CN 113052245B
- Authority
- CN
- China
- Prior art keywords
- image
- clustering
- processed
- feature
- preset
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Image Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application provides an image clustering method and device, electronic equipment and a storage medium, and relates to the technical field of image clustering. The image clustering method comprises the following steps: firstly, acquiring image characteristics to be processed; secondly, inputting the image features to be processed into a preset image clustering model to perform first clustering processing to obtain a first image clustering result and a first type of center feature set; and then, inputting the first image clustering result and the first type center feature set into a preset image clustering model to perform second clustering processing, so as to obtain a second image clustering result. By the method, secondary clustering can be realized according to the same clustering model, and the problem of low image clustering efficiency caused by the fact that only pictures to be clustered are clustered in the prior art and a large number of pictures of the same individual are still distributed in a plurality of categories is solved.
Description
Technical Field
The present application relates to the field of image clustering technologies, and in particular, to an image clustering method and apparatus, an electronic device, and a storage medium.
Background
According to the research of the inventor, in an actual application scene, the prior art only clusters pictures to be clustered, so that higher file splitting and lower recall rate are easy to generate, namely, a large number of pictures of the same individual are still distributed in a plurality of categories, so that the clustering result has very limited utilization value, and the problem of low image clustering efficiency exists.
Disclosure of Invention
Accordingly, an object of the present application is to provide an image clustering method and apparatus, an electronic device and a storage medium, so as to improve the problems in the prior art.
In order to achieve the above purpose, the embodiment of the present application adopts the following technical scheme:
in a first aspect, the present application provides an image clustering method, including:
acquiring image characteristics to be processed;
inputting the image features to be processed into a preset image clustering model to perform first clustering processing to obtain a first image clustering result and a first type center feature set;
and inputting the first image clustering result and the first type center feature set into the preset image clustering model to perform second clustering processing to obtain a second image clustering result.
In an alternative embodiment, the step of acquiring the feature of the image to be processed includes:
acquiring an image feature vector set, wherein the image feature vector set comprises at least one image feature vector;
performing similarity comparison on the image feature vector set and a historical clustering result to obtain similarity of each image feature vector, wherein the historical clustering result is a clustering result obtained by performing image clustering in advance;
judging whether the similarity of each image feature vector exceeds a preset similarity threshold, and taking the image feature vector which does not exceed the preset similarity threshold as the image feature to be processed.
In an alternative embodiment, the step of obtaining the set of image feature vectors includes:
acquiring image data to be processed;
extracting the image data to be processed to obtain image characteristics and structural information;
and filtering the image features according to the structural information to obtain an image feature vector set.
In an optional embodiment, the step of extracting the image data to be processed to obtain image features and structural information includes:
performing feature extraction processing on the image data to be processed to obtain image features;
and carrying out structured information extraction processing on the image data to be processed to obtain structured information.
In an optional embodiment, the step of inputting the image feature to be processed into a preset image clustering model to perform a first clustering process to obtain a first image clustering result and a first class center feature set includes:
judging whether the number of the image features to be processed is larger than a preset number threshold;
if yes, inputting the image features to be processed which are larger than a preset quantity threshold value into a preset image clustering model for processing, and obtaining a first image clustering result and a first type center feature set.
In an optional embodiment, the step of inputting the image features to be processed greater than a preset number threshold into a preset image clustering model to process to obtain a first image clustering result and a first class center feature set includes:
processing the image characteristics to be processed based on a preset image clustering model to obtain a first image clustering result;
and carrying out weighted average calculation on the first image clustering result to obtain a first class center feature set.
In an optional embodiment, the step of inputting the first image clustering result and the first class center feature set into the preset image clustering model to perform second clustering processing to obtain a second image clustering result includes:
judging whether the clustering batch of the first image clustering result is larger than a preset batch or not;
if yes, processing the first image clustering result and the first type center feature set based on the preset image clustering model to obtain a second image clustering result.
In a second aspect, the present application provides an image clustering apparatus, comprising:
the feature acquisition module is used for acquiring the feature of the image to be processed;
the first clustering module is used for inputting the image features to be processed into a preset image clustering model to perform first clustering processing to obtain a first image clustering result and a first type of center feature set;
and the second clustering module is used for inputting the first image clustering result and the first type center feature set into the preset image clustering model to perform second clustering processing to obtain a second image clustering result.
In a third aspect, the present application provides an electronic device comprising: a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the image clustering method of any one of the preceding embodiments when the program is executed.
In a fourth aspect, the present application provides a storage medium, where the storage medium includes a computer program, where the computer program controls an electronic device in which the storage medium is located to execute the image clustering method according to any one of the foregoing embodiments.
According to the image clustering method and device, the electronic equipment and the storage medium, the image to be processed is input into the preset image clustering model to obtain the first image clustering result and the first type center feature set, and the first image clustering result and the first type center feature set are input into the preset image clustering model to obtain the second type result, so that secondary clustering according to the same clustering model is realized, the problem that in the prior art, only images to be clustered are clustered, a large number of images of the same individual are still distributed in a plurality of types is solved, the clustering value is very limited, and the image clustering efficiency is low.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments will be briefly described below, it being understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 shows a block diagram of an electronic device according to an embodiment of the present application.
Fig. 2 shows a flow chart of an image clustering method according to an embodiment of the present application.
Fig. 3 shows another flow diagram of an image clustering method according to an embodiment of the present application.
Fig. 4 shows a block diagram of an image clustering apparatus according to an embodiment of the present application.
Icon: 100-an electronic device; 110-a first memory; 120-a first processor; 130-a communication module; 400-image clustering means; 410-a feature acquisition module; 420-a first clustering module; 430-second aggregation module.
Detailed Description
In recent years, with the increasing maturation and development of artificial intelligence, face clustering is required for faces in a large number of videos or pictures in many industrial intelligent business scenes. The basic principle of face clustering is to archive faces of the same person into a file, and the file can be used as a unique mark of the person. For example, in the intelligent security industry, a large number of faces in a surveillance video stream or a snapshot photo stream need to be clustered, and the clustered unique mark can be applied to the services of querying and displaying the track of a specific person or performing track collision, fusion and the like on other tracks.
In the existing face clustering method, the face pictures to be clustered are subjected to feature extraction through a face recognition model, then are subjected to similarity comparison with a target file, and are clustered through threshold judgment, and the method is generally suitable for small-data-volume scenes, and has the problems of lower performance and lower recall rate in large-data-volume scenes; or batch clustering is carried out on the batch face pictures by using a supervised model or an unsupervised model, so that the method is suitable for batch processing of large data volume, but the real-time requirement is difficult to meet. In an actual application scene, no matter what method is adopted, if only face pictures to be clustered are clustered, higher file splitting and lower recall rate are very easy to generate, namely, a large number of faces of the same person are still distributed in a plurality of categories, so that the clustering result has very limited utilization value; in addition, for the clustering result, only one or a plurality of specific faces with highest quality scores are generally selected at present as the representative class of a certain clustering result, when the data volume is large, the problem of low recall rate or low calculation efficiency often exists, and how to define the representative characteristics of the clustering result is also a difficult problem of the current face clustering. In a word, how to simultaneously ensure or balance the instantaneity and the clustering efficiency of the face clusters, and how to improve the recall rate of the clusters and reduce the splitting situation under the condition of ensuring the clustering accuracy is a problem to be solved in the practical application process of the face clusters.
In order to improve at least one of the above technical problems, an embodiment of the present application provides an image clustering method and apparatus, an electronic device, and a storage medium, and a possible implementation manner is used to describe the technical scheme of the present application.
The present application is directed to a method for manufacturing a semiconductor device, and a semiconductor device manufactured by the method.
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments of the present application. The components of the embodiments of the present application generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the application, as presented in the figures, is not intended to limit the scope of the application, as claimed, but is merely representative of selected embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
It should be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises an element.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures.
It should be noted that the features of the embodiments of the present application may be combined with each other without conflict.
Fig. 1 is a block diagram of an electronic device 100 according to an embodiment of the present application, where the electronic device 100 in this embodiment may be a server, a processing device, a processing platform, etc. capable of performing data interaction and processing. The electronic device 100 includes a first memory 110, a first processor 120, and a communication module 130. The first memory 110, the first processor 120, and the communication module 130 are electrically connected directly or indirectly to each other to realize data transmission or interaction. For example, the components may be electrically connected to each other via one or more communication buses or signal lines.
Wherein the first memory 110 is used for storing programs or data. The first Memory 110 may be, but is not limited to, a random access Memory (Random Access Memory, RAM), a Read Only Memory (ROM), a programmable Read Only Memory (Programmable Read-Only Memory, PROM), an erasable Read Only Memory (Erasable Programmable Read-Only Memory, EPROM), an electrically erasable Read Only Memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.
The first processor 120 is used to read/write data or programs stored in the first memory 110 and perform corresponding functions. The communication module 130 is used for establishing a communication connection between the electronic device 100 and other communication terminals through a network, and for transceiving data through the network.
It should be understood that the structure shown in fig. 1 is merely a schematic diagram of the structure of the electronic device 100, and that the electronic device 100 may also include more or fewer components than shown in fig. 1, or have a different configuration than shown in fig. 1. The components shown in fig. 1 may be implemented in hardware, software, or a combination thereof.
Referring to fig. 2 in combination, a flowchart of an image clustering method according to an embodiment of the present application may be executed by the electronic device 100 of fig. 1, for example, may be executed by the first processor 120 in the electronic device 100. It should be understood that, in other embodiments, the order of some steps in the image clustering method of the present embodiment may be interchanged according to actual needs, or some steps therein may be omitted or deleted. The flow of the image clustering method shown in fig. 2 is described in detail below.
Step S210, obtaining the image characteristics to be processed.
Step S220, inputting the image features to be processed into a preset image clustering model to perform first clustering processing, and obtaining a first image clustering result and a first type center feature set.
And step S230, inputting the first image clustering result and the first type center feature set into a preset image clustering model to perform second clustering processing, and obtaining a second image clustering result.
According to the method, the image to be processed is input into the preset image clustering model to obtain the first image clustering result and the first type center feature set, and the first image clustering result and the first type center feature set are input into the preset image clustering model to obtain the second type result, so that secondary clustering according to the same clustering model is realized, the problem that in the prior art, only pictures to be clustered are clustered, a large number of pictures of the same individual are still distributed in a plurality of categories is solved, the clustering value is very limited, and the image clustering efficiency is low.
For step S210, it should be noted that the specific manner of acquiring the image feature to be processed is not limited, and may be set according to the actual application requirement. For example, in an alternative example, step S210 may include a step of taking, as the image feature to be processed, an image feature vector that does not exceed a preset similarity threshold. Therefore, on the basis of fig. 2, fig. 3 is a schematic flow chart of another image clustering method provided by an embodiment of the present application, referring to fig. 3, step S210 may include:
step S211, an image feature vector set is acquired.
Wherein the set of image feature vectors comprises at least one image feature vector.
And S212, comparing the similarity between the image feature vector set and the historical clustering result to obtain the similarity of each image feature vector.
The historical clustering result is a clustering result obtained by carrying out image clustering in advance.
Step S213, judging whether the similarity of each image feature vector exceeds a preset similarity threshold, and taking the image feature vector which does not exceed the preset similarity threshold as the image feature to be processed.
In step S211, it should be noted that a specific manner of acquiring the image feature vector set is not limited, and may be set according to actual application requirements. For example, in an alternative example, step S211 may include the sub-steps of:
acquiring image data to be processed; extracting the image data to be processed to obtain image characteristics and structural information; and filtering the image features according to the structural information to obtain an image feature vector set.
Alternatively, the specific type of the image data is not limited, and may be set according to actual application requirements. For example, in one alternative example, the image data may be a face image. That is, face images to be clustered may be acquired and noted as a first set of face images to be clustered. The face image to be clustered can be obtained from a video file or a picture file, the video file can be a real-time video or an offline video of a multi-channel front-end camera, the video file comprises a plurality of frames, and each frame can contain one or more faces; the picture file may be a picture, and a picture may contain a plurality of faces or a face. The method comprises the steps of obtaining a first face image set to be clustered by inputting a video file or a picture file into a face detection model, wherein the first face image set to be clustered at least comprises one face image. The face detection model is generally a neural network model, such as an object detection model of MTCNN, YOLO series and the like. In one example, the face image acquisition source is a real-time snapshot image of the front-end camera, and the face detection is performed through the face detection model to acquire the real-time face image.
Alternatively, the specific step of extracting the image data to be processed is not limited, and may be set according to actual application requirements. For example, in one alternative example, the following sub-steps may be included:
carrying out feature extraction processing on the image data to be processed to obtain image features; and carrying out structured information extraction processing on the image data to be processed to obtain structured information.
That is, the face feature vector and the structured information are extracted from the first face image set to be clustered, so that a second face image feature vector set to be clustered and a second face structured information set to be clustered can be obtained.
In detail, the second face image feature vector set to be clustered and the second face structural information set to be clustered are obtained by inputting the first face image set to be clustered into a face analysis model. In one example, the first set of face images to be clustered may include a plurality of sets of face images, e.g., images from different front-end cameras may be different sets of faces, which may be parsed in parallel by the same face parsing model in a plurality of parsing services. And finally obtaining a unique face feature and a piece of structured information for each face image, and finally obtaining a second face image feature vector set to be clustered and a second face structured information set to be clustered for a batch of face data, wherein the second face image feature vector set to be clustered and the second face structured information set to be clustered have a one-to-one correspondence.
Exemplary embodimentsThe ground can be if there is K way front end to take a candid photograph the camera, record respectively as: c (C) 1 ,C 2 ,…,C K In a certain unit time, for example, 10s, the number of face images captured by each path of cameras is N respectively 1 ,N 2 ,…,N K If better real-time performance is required, smaller unit time, such as every 5s, 1s, etc., can be set. For example, M face parsing services P 1 ,P 2 ,…,P M In general M<K, let us assume for C i Subscript i of (C), C i Road snapshot face image data at P i％M The analysis service analyzes the face to obtain a K face feature set and a face structural information set in the unit time, and marks a second face image feature vector set to be clustered as F 20 ＝{p i :f i The second face structural information set to be clustered is marked as F 21 ＝{p i :(t i1 ,t i2 ,…t in ) P is }, where i Representing a unique mark of a face image, and F 20 And F is equal to 21 P in (b) i One-to-one correspondence, f i Represents p i Corresponding characteristic value, (t) i1 ,t i2 ,…t in ) Represents p i Is included in the n structured information of (a). It is obvious that the number of the assumed snap faces isHere, theAnd i is an integer.
For the second face structural information set F to be clustered 21 ＝{p i :(t i1 ,t i2 ,…t in )}，t in Representing structured information may include corresponding values of face pitch angle, horizontal angle, gender, age, face mass fraction, etc., such as face pitch angle, for example: 0, horizontal angle: 10, sex: 0, age: 30, face mass fraction: 80, etc.
The face analysis model is generally one or more multi-task neural network analysis models, and the multi-task neural network analysis model represents that one model can simultaneously identify some or all structural information and face characteristic information of a face image. For example, if the face analysis model is a multitasking neural network model, the face features can be obtained through analysis by the model, and all structural information values such as face pitching angle, horizontal angle, gender, age, face quality score and the like can be predicted in a regression mode.
Further, through the one-to-one correspondence between the second face image feature vector set to be clustered and the second face structural information set to be clustered, the second face image feature vector to be clustered can be filtered according to the second face structural information set to be clustered, and a third face image feature set to be clustered can be obtained. Based on the face structured information contained in the second face structured information set to be clustered, the faces to be clustered are screened, low-quality faces are removed, and the faces are not involved in face clustering, so that the clustering efficiency and the clustering accuracy are improved.
In detail, for the second face structured information set F to be clustered 21 ＝{p i :(t i1 ,t i2 ,…t in ) Face structured information t contained in } in Condition screening is performed. Typically, the screening of face images is performed only by face quality scores, such as face image quality scores, for example<And 40, directly placing the face image into a waste film library to indicate that the face is low in identification degree and unavailable. However, in practical application, the filtering is too single or limited only by the quality score of the face image, so that a plurality of filtering conditions can be fully added, and the unavailable face can be filtered as far as possible. By way of example, faces to be clustered with overlarge pitching angles and overlarge horizontal angles can be removed, the faces are generally low in identification degree, the quality of the corresponding face features is poor, the final clustering effect is greatly and continuously influenced, dirty data can be regarded as being filtered, the faces are not involved in face clustering, and the clustering efficiency and the clustering accuracy are improved. Illustratively, the pitch angle of the face may be set>40 or horizontal angle>60, the face is processedThe image is directly put into the scrap stock. Finally obtaining a second face structured information set to be clustered after filtering, and marking as F measurement 31 ＝{p i :(t i1 ,t i2 ,…t in ) }. Through F 31 ＝{p i :(t i1 ,t i2 ,…t in ) Face image unique mark p in } i Can be related to F 20 ＝{p i :f i Obtaining a third face image feature set to be clustered, and marking the feature set as F 30 ＝{p i :f i }。
For the step S212 and the step S213, it should be noted that the similarity between the third face image feature vector set to be clustered and the class feature center of the existing clustering result may be compared, if the similarity threshold is met, the face image feature vector set to be clustered is classified directly into the corresponding class, and if the similarity threshold is not met, the face image feature vector set to be clustered is classified into the fourth face image feature vector set (i.e., the feature of the image to be processed).
The existing clustering result is an existing history clustering result, and as an example, the existing clustering result may be: f (F) h ＝{a i :[fc i ,(p i1 ,p i2 ,…p in )](wherein a) i Unique sign, fc, representing the clustering result i Representation a i Class characteristic center of (p) i1 ,p i2 ,…p in ) Representing the clustering result a i The class feature center of the face image set contained in the clustering result is a representative feature vector of each clustering result.
Feature vector set F of face image to be clustered third 30 ＝{p i :f i Face feature vector f in } i With the existing clustering result F h ＝{a i :[fc i ,(p i1 ,p i2 ,…p in )]All class feature center fc i And (5) carrying out similarity calculation, wherein the similarity is cosine similarity. Illustratively, as F 30 ＝{p i :f i Face and its feature p 1 :f 1 Calculating f 1 And all fcs i Cosine similarity to fc 1 The similarity of (2) is the largest and 95%, the similarity threshold is set to 90%,obviously p 1 If the maximum similarity 95% of the existing clustering result is greater than the set similarity threshold value by 90%, the face is classified into the category corresponding to the maximum similarity, namely p 1 Falls under the clustering result a 1 Is a face image set (p i1 ,p i2 ,…p in ) And finishing the clustering of the faces. Here, the real-time classification for each face does not make real-time changes to the class feature center, because in a large number of scenarios, the face clustering efficiency will be greatly affected. In addition, because the clustering precision is preferentially ensured when the face is classified in real time, the similarity threshold value can be set higher, the similarity between the classified face features and the class center features is obviously high, the class center features are not obviously changed before and after the change, the class center features are not changed when the face is classified in real time, and the gear gathering efficiency can be greatly improved. Setting a smaller unit time according to the example in the steps, and repeating the steps to ensure the real-time performance of clustering. If the maximum similarity between a face and the existing clustering result does not meet the set similarity threshold, the maximum similarity is classified into a fourth face image feature vector set to be clustered, and the feature vector set is not limited as F 40 ＝{p i :f i And waiting for clustering again.
For step S220, it should be noted that the specific manner of performing the first clustering process is not limited, and may be set according to the actual application requirement. For example, in an alternative example, step S220 may include the sub-steps of:
judging whether the number of the image features to be processed is larger than a preset number threshold value or not;
if yes, inputting the image features to be processed which are larger than a preset quantity threshold value into a preset image clustering model for processing, and obtaining a first image clustering result and a first type center feature set.
That is, when the fourth face image feature vector set to be clustered meets the set quantity threshold, the fourth face image feature vector set to be clustered is input into a face clustering model to perform batch face clustering, and a first face clustering result feature center set are obtained.
While a large number of faces are classified, partial unclassified faces are left, and can be classified into F 40 ＝{p i :f i In }. When F 40 ＝{p i :f i And after the number of faces in the sequence number reaches a preset number threshold, directly clustering the faces in batches. Illustratively, as when F 40 ＝{p i :f i When the number of faces reaches 10000, a batch of face clustering is started.
The batch face clustering method is a supervised neural network model or an unsupervised clustering model, wherein the common supervised neural network model for clustering is generally a neural network model based on graph convolution, such as L-GCN, GCN-D, GCN-V and other models, and DBSCAN, KMEANS is common unsupervised model for clustering. Illustratively, the neural network model GCN-V based on graph convolution can be selected for batch face clustering, and in practical application, the face clustering model based on graph convolution generally has better clustering effect than an unsupervised clustering model.
The specific mode of inputting the image features to be processed into the preset image clustering model for processing is not limited, and the image features can be set according to actual application requirements. For example, in one alternative example, the following sub-steps may be included:
processing the image characteristics to be processed based on a preset image clustering model to obtain a first image clustering result; and carrying out weighted average calculation on the first image clustering result to obtain a first class center feature set.
In detail, F 40 ＝{p i :f i Inputting the first face clustering result into the face clustering model, and exemplarily, it is assumed that the first face clustering result may be expressed as follows:
F t ＝{a i :(p i1 ,p i2 ,…p in )}；
wherein a is i Unique sign indicating clustering result, (p) i1 ,p i2 ,…p in ) Representing the clustering result a i A set of facial images contained therein.
For each clustering result, the class feature center needs to be calculated, i.e. for each a i It is necessary to calculate its class center feature fc i For each fc i All are related to a i Feature vector p of (2) in Is to select all p in And performs a weighted average calculation on some representative feature vectors. Illustratively, for a clustered result a 2 The 9 face images included are (p 21 ,p 22 ,p 23 ,p 24 …p 29 ) And (p) 21 ,p 22 ,p 23 ,p 24 …p 29 ) The corresponding face features are (f 21 ,f 22 ,f 23 ,f 24 ,…,f 29 )。
The selection average calculation method is as follows:
for a 2 In relation to p in Is not limited to the feature vector (f) 21 ,f 22 ,f 23 ,f 24 ,…,f 29 ) Generating a similarity matrix, which may be denoted as S imk Each element in the matrix may be denoted as s mk ，s mk Also denoted p im Feature vector and p of (2) ik Is cosine similarity, wherein j, k<=9. Illustratively, assume S imk The specific values of (2) are as follows:
table 1 similarity matrix
| f 21 | f 22 | f 23 | f 24 | f 25 | f 26 | f 27 | f 28 | f 29 | |
| f 21 | 1 | 0.85 | 0.78 | 0.92 | 0.74 | 0.73 | 0.73 | 0.74 | 0.72 |
| f 22 | 0.85 | 1 | 0.88 | 0.85 | 0.72 | 0.69 | 0.68 | 0.72 | 0.72 |
| f 23 | 0.78 | 0.88 | 1 | 0.74 | 0.71 | 0.74 | 0.69 | 0.73 | 0.71 |
| f 24 | 0.92 | 0.85 | 0.74 | 1 | 0.74 | 0.74 | 0.72 | 0.71 | 0.74 |
| f 25 | 0.74 | 0.72 | 0.71 | 0.74 | 1 | 0.86 | 0.92 | 0.68 | 0.74 |
| f 26 | 0.73 | 0.69 | 0.74 | 0.74 | 0.86 | 1 | 0.88 | 0.72 | 0.73 |
| f 27 | 0.73 | 0.68 | 0.69 | 0.72 | 0.92 | 0.88 | 1 | 0.73 | 0.7 |
| f 28 | 0.74 | 0.72 | 0.73 | 0.71 | 0.68 | 0.72 | 0.73 | 1 | 0.94 |
| f 29 | 0.72 | 0.72 | 0.71 | 0.74 | 0.74 | 0.73 | 0.7 | 0.94 | 1 |
For example, assuming that the similarity threshold of the subset partition is 0.75, according to the similarity matrix, as shown in the specific data in table 1, since the similarity matrix is a square of symmetry, only all the elements of the upper triangle or the lower triangle need to be observed or traversed. It can be found that f 21 、f 22 、f 23 、f 24 A subset can be formed because f 21 、f 22 、f 23 The similarity between each other is greater than 0.75, although f 23 、f 24 The similarity between them is less than 0.75, but f 23 At least with f 21 、f 22 One of the similarities is greater than 0.75, so that f is the final value 21 、f 22 、f 23 、f 24 A subset of the connections may be formed. Similarly, f 25 、f 26 、f 27 Form a subset f 28 、f 29 To form a subset, then finally a 2 The division into three subsets is possible, and may be set to subset 1, subset 2, and subset 3 in this order.
For the three subsets obtained, it is assumed that the face with the highest face quality score in each subset is p 21 、p 26 、p 29 Selecting the corresponding face feature f 21 、f 26 、f 29 As representative features of the subsets, the number of faces contained in each subset may be calculated to be 4, 3, and 2. A is calculated by weighted average 2 Class center fc of (2) 2 The specific calculation formula is as follows:
it should be noted that, in step S230, the specific manner of performing the second aggregation process is not limited, and may be set according to the actual application requirement. For example, in an alternative example, step S230 may include the sub-steps of:
judging whether the clustering batch of the first image clustering result is larger than a preset batch or not;
if yes, processing the first image clustering result and the first type center feature set based on a preset image clustering model to obtain a second image clustering result.
In detail, after clustering of the face images of multiple batches, clustering results of different batches can be clustered secondarily through the class feature center set. The batch setting or time setting may be defined for a plurality of batches of face image clusters, indicating batches or time ranges in which face image clusters are continuously performed. For example, regarding the batch setting, it may be set to 3 times, 5 times, or the like, to indicate that the face image clustering is performed 3 times or 5 times followed by the secondary clustering. Regarding the setting of the time range, for example, the time range can be set to be 6:00-23:00, which means that only face images are clustered in the time range of 6:00-23:00, and secondary clustering is performed in the time range of not 6:00-23:00.
In an alternative example, after every 5 batches of face clustering, a secondary clustering of clustered results is performed, and the existing clustering result F is set h ＝{a i :[fc i ,(p i1 ,p i2 ,…p in )]Using each class a } i Class center feature fc i And adopting the same face clustering model to perform batch secondary clustering. The secondary clustering is to cluster the existing face clustering results again, so that the final face clustering is more sufficient, the recall rate of the clustering results is improved, the splitting situation is reduced, and the utilization value of the clustering results is improved.
Illustratively, the result of the secondary clustering is denoted as F rh ＝{a ri :[fc ri ,(a ri1 ,a ri2 ,…,a rin )](wherein a) ri Unique flag fc representing secondary clustering result ri Representation class a ri Class-characteristic center of (a) ri1 ,a ri2 ,…,a rin ) Representation class a ri From the combination of existing face clustering results, and for fc ri Can be calculated by adopting a weighted average modeAnd (5) calculating.
Regarding fc ri Specific calculation mode and fc of (C) i In a similar manner, since each fc i The method is characterized by being a representative feature of each face clustering result, and the method does not need to divide and select the representative feature by sub-sets, and can directly perform weighted average. Illustratively, a certain secondary clustering result a r1 From (a) r11 ,a r12 ,…,a r1n ) Clustering, for each a r1j The center of class is characterized by fc r1j Each a r1j The number of faces is c r1j Then class center feature fc ri The specific calculation mode of (a) is as follows:
further, face image clustering and secondary clustering are alternately performed in sequence, and a final face clustering result is obtained. That is, the face images to be clustered are continuously clustered, the clustered results are secondarily clustered according to the set face clustering batch or time interval in step S230, and the two processes are alternately executed to perform full clustering, so that a final clustering result is obtained.
Through the method, the application provides a face clustering method, under a large-scale face clustering scene, firstly, threshold judgment clustering is carried out through similarity comparison to ensure the real-time performance and accuracy of clustering, and model clustering of mass faces which are not classified through similarity is carried out to ensure the efficiency and recall rate of face clustering; the clustering result is clustered more fully through the secondary clustering of the class center features, the recall rate of the clustering result is improved, the splitting condition is reduced, and the practical application value of the face clustering result is greatly improved; the method for constructing the class center features by selecting the weighted average can effectively extract all the representative features in each clustering result through subset division, and generates class center features with representative significance through weighted fusion.
In connection with fig. 4, the embodiment of the present application further provides an image clustering device 400, where the functions implemented by the image clustering device 400 correspond to the steps performed by the above-mentioned method. The image clustering apparatus 400 may be understood as a processor of the electronic device 100, or may be understood as a component that is independent of the electronic device 100 or the processor and that is controlled by the electronic device 100 to implement the functions of the present application. The image clustering apparatus 400 may include a feature acquisition module 410, a first clustering module 420, and a second clustering module 430, among others.
The feature acquisition module 410 is configured to acquire features of an image to be processed. In an embodiment of the present application, the feature acquisition module 410 may be used to perform step S210 shown in fig. 2, and the description of step S210 may be referred to above with respect to the relevant content of the feature acquisition module 410.
The first clustering module 420 is configured to input the image features to be processed into a preset image clustering model to perform a first clustering process, so as to obtain a first image clustering result and a first class center feature set. In an embodiment of the present application, the first clustering module 420 may be used to perform step S220 shown in fig. 2, and the description of step S220 may be referred to above with respect to the relevant content of the first clustering module 420.
The second clustering module 430 is configured to input the first image clustering result and the first class center feature set into a preset image clustering model for performing a second clustering process, so as to obtain a second image clustering result. In an embodiment of the present application, the second aggregation module 430 may be used to perform step S230 shown in fig. 2, and the description of step S230 may be referred to above for the relevant content of the second aggregation module 430.
In addition, the embodiment of the application also provides a computer readable storage medium, and the computer readable storage medium stores a computer program which executes the steps of the image clustering method when being executed by a processor.
The computer program product of the image clustering method provided by the embodiment of the present application includes a computer readable storage medium storing a program code, where instructions included in the program code may be used to execute steps of the image clustering method in the above method embodiment, and specifically, reference may be made to the above method embodiment, which is not described herein.
In summary, the image clustering method and device, the electronic device and the storage medium provided by the embodiment of the application have the advantages that the image to be processed is input into the preset image clustering model to obtain the first image clustering result and the first type center feature set, and the first image clustering result and the first type center feature set are input into the preset image clustering model to obtain the second type result, so that the secondary clustering according to the same clustering model is realized, the problem that in the prior art, only the images to be clustered are clustered, a large number of images of the same individual are still distributed in a plurality of categories is avoided, the clustering value is very limited, and the image clustering efficiency is low.
The above is only a preferred embodiment of the present application, and is not intended to limit the present application, but various modifications and variations can be made to the present application by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.
Claims (8)
1. An image clustering method, comprising:
acquiring image characteristics to be processed;
inputting the image features to be processed into a preset image clustering model to perform first clustering processing to obtain a first image clustering result and a first type center feature set;
judging whether the clustering batch of the first image clustering result is larger than a preset batch or not;
if yes, performing second clustering on the first image clustering result and the first type center feature set based on the preset image clustering model to obtain a second image clustering result, wherein after the first image clustering processing of each preset batch is completed, performing second clustering on all the existing first image clustering results and first type center feature sets once, and the image features to be processed, which are aimed at by the first image clustering processing of each batch, are obtained based on different images;
the step of acquiring the image characteristics to be processed comprises the following steps:
acquiring an image feature vector set, wherein the image feature vector set comprises at least one image feature vector;
comparing the similarity between the image feature vector set and a historical clustering result to obtain the similarity of each image feature vector, wherein the historical clustering result is a clustering result obtained by carrying out image clustering in advance, the historical clustering result comprises a plurality of class feature centers, and the similarity of the image feature vector is the maximum similarity between the image feature vector and all class feature centers;
judging whether the similarity of each image feature vector exceeds a preset similarity threshold, and taking the image feature vector which does not exceed the preset similarity threshold as the image feature to be processed.
2. The image clustering method as claimed in claim 1, wherein said step of acquiring a set of image feature vectors comprises:
acquiring image data to be processed;
extracting the image data to be processed to obtain image characteristics and structural information;
and filtering the image features according to the structural information to obtain an image feature vector set.
3. The image clustering method as claimed in claim 2, wherein the step of extracting the image data to be processed to obtain the image features and the structural information includes:
performing feature extraction processing on the image data to be processed to obtain image features;
and carrying out structured information extraction processing on the image data to be processed to obtain structured information.
4. The image clustering method as claimed in claim 1, wherein the step of inputting the image features to be processed into a preset image clustering model to perform a first clustering process to obtain a first image clustering result and a first class center feature set includes:
judging whether the number of the image features to be processed is larger than a preset number threshold;
if yes, inputting the image features to be processed which are larger than a preset quantity threshold value into a preset image clustering model for processing, and obtaining a first image clustering result and a first type center feature set.
5. The image clustering method as claimed in claim 4, wherein the step of inputting the image features to be processed greater than a preset number threshold into a preset image clustering model for processing to obtain a first image clustering result and a first type of center feature set includes:
processing the image characteristics to be processed based on a preset image clustering model to obtain a first image clustering result;
and carrying out weighted average calculation on the first image clustering result to obtain a first class center feature set.
6. An image clustering apparatus, comprising:
the feature acquisition module is used for acquiring the feature of the image to be processed;
the first clustering module is used for inputting the image features to be processed into a preset image clustering model to perform first clustering processing to obtain a first image clustering result and a first type of center feature set;
the second clustering module is used for judging whether the clustering batch of the first image clustering result is larger than a preset batch or not; if yes, processing the first image clustering result and the first type center feature set based on the preset image clustering model to obtain a second image clustering result, wherein after the first image clustering processing of which the number meets the preset batch is completed, performing second clustering processing on all the existing first image clustering results and first type center feature sets once, and the image features to be processed, which are aimed at by the first image clustering processing of each batch, are obtained based on different images;
the feature acquisition module is specifically configured to:
acquiring an image feature vector set, wherein the image feature vector set comprises at least one image feature vector;
comparing the similarity between the image feature vector set and a historical clustering result to obtain the similarity of each image feature vector, wherein the historical clustering result is a clustering result obtained by carrying out image clustering in advance, the historical clustering result comprises a plurality of class feature centers, and the similarity of the image feature vector is the maximum similarity between the image feature vector and all class feature centers;
judging whether the similarity of each image feature vector exceeds a preset similarity threshold, and taking the image feature vector which does not exceed the preset similarity threshold as the image feature to be processed.
7. An electronic device, comprising: memory, a processor and a computer program stored on the memory and executable on the processor, which processor implements the image clustering method according to any one of claims 1 to 5 when executing the program.
8. A storage medium comprising a computer program which, when run, controls an electronic device in which the storage medium is located to perform the image clustering method of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110340584.7A CN113052245B (en) | 2021-03-30 | 2021-03-30 | Image clustering method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110340584.7A CN113052245B (en) | 2021-03-30 | 2021-03-30 | Image clustering method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113052245A CN113052245A (en) | 2021-06-29 |
| CN113052245B true CN113052245B (en) | 2023-08-25 |
Family
ID=76516527
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110340584.7A Active CN113052245B (en) | 2021-03-30 | 2021-03-30 | Image clustering method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113052245B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113673550A (en) * | 2021-06-30 | 2021-11-19 | 浙江大华技术股份有限公司 | Clustering method, clustering device, electronic equipment and computer-readable storage medium |
| CN113743533B (en) * | 2021-09-17 | 2023-08-01 | 重庆紫光华山智安科技有限公司 | Picture clustering method and device and storage medium |
| CN113920353B (en) * | 2021-11-04 | 2022-07-29 | 厦门市美亚柏科信息股份有限公司 | Unsupervised face image secondary clustering method, unsupervised face image secondary clustering device and unsupervised face image secondary clustering medium |
| CN118135571B (en) * | 2024-05-10 | 2024-08-13 | 上海蜜度科技股份有限公司 | Image semantic segmentation method, system, storage medium and electronic equipment |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8582872B1 (en) * | 2011-06-30 | 2013-11-12 | Google Inc. | Place holder image detection via image clustering |
| CN106503656A (en) * | 2016-10-24 | 2017-03-15 | 厦门美图之家科技有限公司 | A kind of image classification method, device and computing device |
| CN108229419A (en) * | 2018-01-22 | 2018-06-29 | 百度在线网络技术（北京）有限公司 | For clustering the method and apparatus of image |
| CN109063737A (en) * | 2018-07-03 | 2018-12-21 | Oppo广东移动通信有限公司 | Image processing method, device, storage medium and mobile terminal |
| CN109213732A (en) * | 2018-06-28 | 2019-01-15 | 努比亚技术有限公司 | A kind of method, mobile terminal and computer readable storage medium improving photograph album classification |
| CN109710728A (en) * | 2018-11-26 | 2019-05-03 | 西南电子技术研究所（中国电子科技集团公司第十研究所） | News topic automatic discovering method |
| CN109800744A (en) * | 2019-03-18 | 2019-05-24 | 深圳市商汤科技有限公司 | Image clustering method and device, electronic equipment and storage medium |
| CN110245679A (en) * | 2019-05-08 | 2019-09-17 | 北京旷视科技有限公司 | Image clustering method, device, electronic equipment and computer readable storage medium |
| CN111369521A (en) * | 2020-03-02 | 2020-07-03 | 名创优品（横琴）企业管理有限公司 | Image filtering method based on image quality and related device |
| CN111738341A (en) * | 2020-06-24 | 2020-10-02 | 佳都新太科技股份有限公司 | Distributed large-scale face clustering method and device |
| US10909167B1 (en) * | 2020-09-17 | 2021-02-02 | Pure Memories Ltd | Systems and methods for organizing an image gallery |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2659400A4 (en) * | 2010-12-30 | 2017-01-11 | Nokia Technologies Oy | Method, apparatus, and computer program product for image clustering |
| US10997147B2 (en) * | 2018-07-17 | 2021-05-04 | Snowflake Inc. | Incremental clustering of database tables |
-
2021
- 2021-03-30 CN CN202110340584.7A patent/CN113052245B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8582872B1 (en) * | 2011-06-30 | 2013-11-12 | Google Inc. | Place holder image detection via image clustering |
| CN106503656A (en) * | 2016-10-24 | 2017-03-15 | 厦门美图之家科技有限公司 | A kind of image classification method, device and computing device |
| CN108229419A (en) * | 2018-01-22 | 2018-06-29 | 百度在线网络技术（北京）有限公司 | For clustering the method and apparatus of image |
| CN109213732A (en) * | 2018-06-28 | 2019-01-15 | 努比亚技术有限公司 | A kind of method, mobile terminal and computer readable storage medium improving photograph album classification |
| CN109063737A (en) * | 2018-07-03 | 2018-12-21 | Oppo广东移动通信有限公司 | Image processing method, device, storage medium and mobile terminal |
| CN109710728A (en) * | 2018-11-26 | 2019-05-03 | 西南电子技术研究所（中国电子科技集团公司第十研究所） | News topic automatic discovering method |
| CN109800744A (en) * | 2019-03-18 | 2019-05-24 | 深圳市商汤科技有限公司 | Image clustering method and device, electronic equipment and storage medium |
| CN110245679A (en) * | 2019-05-08 | 2019-09-17 | 北京旷视科技有限公司 | Image clustering method, device, electronic equipment and computer readable storage medium |
| CN111369521A (en) * | 2020-03-02 | 2020-07-03 | 名创优品（横琴）企业管理有限公司 | Image filtering method based on image quality and related device |
| CN111738341A (en) * | 2020-06-24 | 2020-10-02 | 佳都新太科技股份有限公司 | Distributed large-scale face clustering method and device |
| US10909167B1 (en) * | 2020-09-17 | 2021-02-02 | Pure Memories Ltd | Systems and methods for organizing an image gallery |
Non-Patent Citations (1)
| Title |
|---|
| 基于自适应阈值的自动提取关键帧的聚类算法;王方石,须德,吴伟鑫;计算机研究与发展(第10期);第109-114页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113052245A (en) | 2021-06-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113052245B (en) | Image clustering method and device, electronic equipment and storage medium | |
| US12061989B2 (en) | Machine learning artificial intelligence system for identifying vehicles | |
| EP3779774B1 (en) | Training method for image semantic segmentation model and server | |
| US10891465B2 (en) | Methods and apparatuses for searching for target person, devices, and media | |
| CN111967429B (en) | Pedestrian re-recognition model training method and device based on active learning | |
| US9639780B2 (en) | System and method for improved classification | |
| US12099574B1 (en) | Artificial intelligence system for inspecting image reliability | |
| CN105760488B (en) | Image expression method and device based on multi-level feature fusion | |
| JP2017033547A (en) | Information processing apparatus, control method therefor, and program | |
| CN111783743A (en) | Image clustering method and device | |
| CN113936175A (en) | Method and system for identifying events in video | |
| WO2021211226A1 (en) | Face clustering in video streams | |
| CN112101156A (en) | Target identification method and device and electronic equipment | |
| CN113920353B (en) | Unsupervised face image secondary clustering method, unsupervised face image secondary clustering device and unsupervised face image secondary clustering medium | |
| CN105678333B (en) | Method and device for determining crowded area | |
| CN110309790B (en) | Scene modeling method and device for road target detection | |
| JP7133085B2 (en) | Database update method and device, electronic device, and computer storage medium | |
| CN113743533B (en) | Picture clustering method and device and storage medium | |
| CN112214639B (en) | Video screening method, video screening device and terminal equipment | |
| CN116980472A (en) | Push data processing method, data push model training method and device | |
| CN114359605A (en) | Face clustering method, device and equipment | |
| CN115908882A (en) | Picture clustering method and device | |
| CN112100427A (en) | Video processing method and device, electronic equipment and storage medium | |
| CN110889438A (en) | Image processing method and device, electronic equipment and storage medium | |
| CN117333926B (en) | Picture aggregation method and device, electronic equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |