CN111314760B - Television and smiling face shooting method thereof - Google Patents
Television and smiling face shooting method thereof Download PDFInfo
- Publication number
- CN111314760B CN111314760B CN202010060099.XA CN202010060099A CN111314760B CN 111314760 B CN111314760 B CN 111314760B CN 202010060099 A CN202010060099 A CN 202010060099A CN 111314760 B CN111314760 B CN 111314760B
- Authority
- CN
- China
- Prior art keywords
- smiling face
- face
- smiling
- image
- images
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
- H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
- H04N21/4223—Cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/174—Facial expression recognition
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
- H04N21/44213—Monitoring of end-user related data
- H04N21/44218—Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program
Landscapes
- Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Multimedia (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Evolutionary Computation (AREA)
- Computational Linguistics (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- General Engineering & Computer Science (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- Data Mining & Analysis (AREA)
- Artificial Intelligence (AREA)
- Signal Processing (AREA)
- Social Psychology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Human Computer Interaction (AREA)
- Computer Networks & Wireless Communication (AREA)
- Databases & Information Systems (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Studio Devices (AREA)
Abstract
The invention relates to a television and a smiling face shooting method thereof.A shooting module is opened if the television body is detected to be in a working state; shooting a user using the television body by using the shooting module, and detecting a human face image in a shot image; the face image input to the predetermined smiling face recognition model in will shooing the image if utilize smiling face recognition model discerns the face image who shoots in the image and has at least one for the smiling face, then preserves shoot the image, and the TV set can shoot the user, can also further preserve the picture of shooing that contains the smiling face image, can improve the intellectuality of TV set, and convenience of customers' record is in the same time nice and beautiful when using the TV set.
Description
Technical Field
The invention relates to the technical field of images, in particular to a television and a smiling face shooting method thereof.
Background
The television is very common in homes, and the functions of the television are more and more powerful, for example, the Android system-based smart television enables users to install various entertainment applications, such as movies, fun, sports and the like, and greatly enriches the experience of the users.
Smiling face expressions of a user in a process of using a television, particularly happy and nice moments of watching the television together with family people, cannot be recorded by the current television.
Disclosure of Invention
Accordingly, there is a need for a television and a smiling face photographing method thereof.
In a first aspect, a smiling face photographing method for a television is provided, the television including:
a television body;
the shooting module is arranged on the television body;
the smiling face photographing method includes:
if the television body is detected to be in the working state, the shooting module is opened;
shooting a user using the television body by using the shooting module, and detecting a human face image in a shot image;
and inputting the face images in the shot images into a preset smiling face recognition model, and if at least one of the face images in the shot images is identified to be a smiling face by using the smiling face recognition model, storing the shot images.
In one embodiment, if at least one of the face images in the captured image is a smiling face, the step of saving the captured image includes:
if at least one of the face images in the shot images of the same type is identified to be a smiling face by using the smiling face identification model, the number of smiling face expressions in the shot images is identified, the definition of each shot image of the same type is calculated, and the shot image with the largest smiling face expression and the highest definition is stored.
In one embodiment, the number of the shooting modules is multiple, and at least one shooting module is arranged in the center of the top end of the television screen, the center of the bottom end of the television screen and the centers of two side edges of the television screen.
In one embodiment, the method further includes a step of constructing the smiling face recognition model, which includes:
establishing a standard non-smiling face library and a standard smiling face library, wherein smiling face images and non-smiling face images are processed into preset sizes, the preset labels of the standard non-smiling face library comprise non-smiling face labels, and the preset smiling face category labels of the standard smiling face library comprise smiling labels and ugly label;
establishing a light-weight convolutional neural network structure for mobilenevv 3 deep learning, wherein the number of deep convolutional neural network layers of the light-weight convolutional neural network structure is 20, the initial weight and bias parameters of each neuron of each layer are both between positive and negative 1, the convolution kernels of each layer are respectively 1, 3, 5 and 7, the convolution kernel size of an input layer is 3 x 3, the convolution kernel size of an output layer is 1 x 1, the convolution kernel size of a pooling layer is 7 x 7, convolution step sizes are 1 and 2, the input layer uses a 224 x 3 matrix, and the output layer uses a 1 x 3 matrix, so that an initial smiling face recognition model is constructed;
and training the smiling face recognition model by using a loss function until the loss function converges to a loss value which is not reduced, wherein in the training of the light-weight convolutional neural network structure, the characteristic difference of the facial expressions with smiling face labels and non-smiling face labels is enlarged, the characteristic difference between the expressions which are non-smiling face labels is reduced, the characteristic difference of the smiling face expressions with smiling and clown type labels is enlarged, and the expression characteristic difference of the smiling face expressions with smiling or clown type labels is reduced.
In one embodiment, the step of recognizing the smiling face expression of the human face image in the photographed image by using the smiling face recognition model includes:
and after the face image in the shot image is zoomed into the preset size and normalized, the face image in the shot image is input into the smiling face recognition model after training, the smiling face recognition model extracts the smiling face expression characteristic value of the face image and sends the smiling face expression characteristic value into a classifier to judge and classify the smiling face, and smiling face recognition is realized.
In one embodiment, the loss function includes a first loss function, which is used for judging the closeness degree between a real facial expression category and an actually output facial expression category, wherein the higher the closeness degree is, the smaller the loss value is; the first loss function is:
wherein, Loss (m, n, a) is the Loss value of the first Loss function in the training process, x is the expression characteristic value of the extracted face image, and aiTo extract the class label of the facial image, m (x) is the true facial expression distribution probability, n (x) is the expected output facial expression distribution probability, cos (θ)n) To predict the inner product angle of the probability, i.e.
cos(θm) The inner product included angle of the true probability,
when the expression characteristic value of the extracted face image is a non-smiling face label, f (x) is the average probability of the face distribution probability of a standard non-smiling face library, and when the expression characteristic value of the extracted face image is a smiling label, f (x) is the average probability of the face distribution probability of the smiling face in the standard smiling face library; when the expression characteristic value of the extracted face image is the ugly smile label, f (x) is the average probability of the distribution probability of the ugly smile face in the standard smile library.
In one embodiment, the loss function includes a second loss function, configured to identify whether two images belong to the same label in a training process, and calculate a loss value of the two images, where the second loss function is:
Triloss(xi,xj,cij) A loss value, x, characterizing a second loss functioni、xjAre all input standard library images, cij1 denotes an expression in which two input images are the same kind of label, cij0 denotes an expression where two input photographs are different labels, xmIs the average of the expressive features of the standard smiley face library images.
In one embodiment, if at least one of the face images in the captured image is a smiling face, the step of saving the captured image includes:
and if the face images in the shot images are identified to be smiling labels, storing the shot images.
In one embodiment, if at least one of the face images in the captured image is a smiling face, the step of saving the captured image includes:
establishing a folder, and identifying the folder according to a preset smiling face type label, wherein the number of the folders is the same as that of the preset smiling face type labels;
and storing the shot images of the same smiling face class label into a folder with a corresponding identifier.
In a second aspect, a television is provided, comprising:
a television body;
a main control module;
the shooting module is connected with the main control module and is used for shooting a user using the television body;
the file management module is respectively connected with the main control module and the shooting module;
the main control module is used for turning on the shooting module if the television body is detected to be in a working state, shooting a user using the television body by using the shooting module, detecting a human face image in a shot image, inputting the human face image in the shot image into a preset smiling face recognition model, and recognizing the expression of the human face image in the shot image by using the smiling face recognition model;
the file management module is used for saving the shot images when at least one of the face images in the shot images is a smiling face.
According to the television and the smiling face shooting method thereof, the television can shoot the user, the shot image containing the smiling face image can be further stored, the intelligence of the television can be improved, and the user can record the beautiful moment when using the television conveniently.
Drawings
FIG. 1 is a flowchart illustrating a smiling face photographing method according to an embodiment of the present invention;
fig. 2 is a schematic flow chart of constructing a smiling face recognition model according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The embodiment of the invention provides a smiling face shooting method for a television. The television in the embodiment of the invention comprises:
a television body;
and the shooting module is arranged on the television body. The shooting module can be a camera, and particularly can be a high-definition wide-angle camera.
In one embodiment of the present invention, the television includes a plurality of shooting modules, and at least one shooting module is disposed in the center of the top end of the television screen, the center of the bottom end of the television screen, and the centers of two sides of the television screen.
In other embodiments, the shooting module may be disposed at other positions of the television body.
It should be noted that, if the shooting module is disposed at another position, for example, on a wall or a desk, separately from the television main body, but is connected to the television main body in a wired or wireless manner, the shooting module may also be regarded as being disposed on the television main body.
One of the shooting modules can be started as required, for example, the user can start the top center of the television screen when the shooting effect of the shooting module at a higher position is more consistent with the expectation.
Can start the module of shooting wherein more than two as required, can enlarge the visual angle scope of shooting the module, when the module of shooting more than two starts, can shoot more information, then fuse the image of shooing, then the face image input in the shooting image after will fusing is to predetermined smiling face recognition model.
Fig. 1 is a flowchart illustrating a smiling face photographing method for a television according to an embodiment, as shown in fig. 1, the smiling face photographing method for a television includes steps 102 to 106:
the smiling face photographing method includes:
step 102: and detecting the running state of the television body, and if detecting that the television body is in the working state, opening the shooting module.
Step 104: and shooting the user using the television body by using the shooting module, and detecting the face image in the shot image.
The user who shoots the use of the television body may be a user who is watching a television program.
Step 106: and inputting the face images in the shot images into a preset smiling face recognition model, and if at least one of the face images in the shot images is identified to be a smiling face by using the smiling face recognition model, storing the shot images. The storage mode can be one frame of image, and can also be a mode of starting video recording for storage.
For example, if it is detected that 3 faces exist in a captured image and it is recognized that a smiling face exists in the captured image, the captured image is saved. The photographed image can be saved by a file management module such as a memory, and the user can view and delete the saved image.
In one embodiment of the present invention, if at least one of the face images in the captured image is identified as a smile using the smile recognition model, the step of saving the captured image includes:
if at least one of the facial images in the same type of shot images is identified as a smile by using the smiling face identification model, the number of smiling face expressions in the shot images is identified, and the shot images with the largest smiling face expressions are stored.
Further, if at least one of the face images in the shot images of the same type is identified as a smiling face by using the smiling face identification model, the number of smiling face expressions in the shot images is identified, the definition of each shot image of the same type is calculated, and the shot image with the largest smiling face expression and the highest definition is stored. The definition of the captured image can be calculated by specifically calculating the resolution.
The photographed images of the same user are the same type of photographed image. For example, three, four and five, which indicate that the three photographed images are the same type of photographed image and the image with the largest number of smiling faces among the three photographed images is stored, are all captured by the user among the three photographed images.
In one embodiment of the present invention, as shown in fig. 2, the construction of the predetermined smiling face recognition model includes the following steps:
step 202: establish standard non-smiling face storehouse and standard smiling face storehouse, smiling face image and non-smiling face image are all handled into and are preset the size, wherein the predetermined label in standard non-smiling face storehouse includes the non-smiling face label, the predetermined smiling face classification label in standard smiling face storehouse includes smiling label and ugly label, and smiling label and ugly label still possess the smiling face label jointly simultaneously.
When labeling the data of the smiling face library, a good smiling face image of a smile is marked as a smiling label, and a very ugly smiling face image of the smile is marked as a ugly smiling label, for example, a smiling face image of a deformed smile to five sense organs is marked as a ugly smiling label.
In other embodiments, the preset smiling face category label may be further subdivided, such as a smile label, a jean label, a grin-through label, a closed-eye smile label, a michelia label, a cheer label, a laugh label, a smile label, a squint label, a sell-on smile, and so forth.
Step 204: the method comprises the steps of establishing a light-weight convolutional neural network structure for mobilencev 3 deep learning, wherein the number of deep convolutional neural network layers of the light-weight convolutional neural network structure is 20, initial weight values and bias parameters of all neurons of all layers are between positive and negative 1, convolution kernels of all layers are 1, 3, 5 and 7 respectively, the convolution kernel size of an input layer is 3 x 3, the convolution kernel size of an output layer is 1 x 1, the convolution kernel size of a pooling layer is 7 x 7, convolution step sizes are 1 and 2, a 224 x 3 matrix is used for the input layer, and a 1 x 3 matrix is used for the output layer, so that an initial smiling face recognition model is constructed.
The output layer uses a 1 x 3 matrix to indicate that the classifier has three classifications. In other embodiments, the classification number is equal to the smiley face tag number.
In other implementations, other deep networks may be established to construct the initial smiling face recognition model, which is not limited herein.
The television has limited computing capacity, the mobile lightweight neural network is used, compared with other deep network models, the computational complexity is small, the television cannot be greatly influenced, and the smiling face recognition accuracy is high due to the mobile lightweight neural network smiling face recognition model.
And training the smiling face recognition model by using a loss function until the loss function converges to a loss value which is not reduced, wherein in the training of the light-weight convolutional neural network structure, the characteristic difference of the facial expressions with smiling face labels and non-smiling face labels is enlarged, the characteristic difference between the expressions which are non-smiling face labels is reduced, the characteristic difference of the smiling face expressions with smiling and clown type labels is enlarged, and the expression characteristic difference of the smiling face expressions with smiling or clown type labels is reduced.
The smiling face recognition model further refines the classification of the smiling face category, and the smiling face label of the face image in the image is shot in subsequent recognition, if the face image with the ugly smile expression exists in the shot image, the face image can not be stored, if the face image in the shot image is the smile label, the shot image is stored, the storage space is saved, and more shot images which accord with the expected effect of the user are output.
In other embodiments, a folder may be established, and the folder is identified according to the preset smiling face category tag, where the number of folders is the same as the number of the preset smiling face category tags; and storing the shot images of the same smiling face class label into a folder with a corresponding identifier. For example, a smile folder and a clown smile folder are established, if a face image with a clown smile expression exists in the shot image, the face image is stored in the clown smile folder, and if the face images with the smile expression exist in the shot image, the face image is stored in the smile folder. The user can look over the picture of shooing according to the smiling face classification, satisfies user's diversified demand, improves browsing efficiency.
Specifically, with respect to step 106, the step of recognizing a smiling face expression of a human face image in a photographed image using the smiling face recognition model includes: and after the face image in the shot image is scaled to the preset size in the step 202 and normalized, the face image in the shot image is input into the smiling face recognition model which finishes training, and the smiling face recognition model extracts the smiling face expression characteristic value of the face image and sends the smiling face expression characteristic value into a classifier to judge and classify the smiling face so as to realize smiling face recognition.
Regarding the loss function in step 206, in one embodiment of the present invention, the loss function includes a first loss function, which is used to determine a closeness degree between the real facial expression category and the actually output facial expression category, where the higher the closeness degree is, the smaller the loss value is; the first loss function is:
wherein, Loss (m, n, a) is the Loss value of the first Loss function in the training process, x is the expression characteristic value of the extracted face image, and aiTo extract the class label of the facial image, m (x) is the true facial expression distribution probability, n (x) is the expected output facial expression distribution probability, cos (θ)n) To predict the inner product angle of the probability, i.e.
cos(θm) The inner product included angle of the true probability,
(x) an average probability representing the distribution probability of the extracted face label corresponding to all library images, for example, when the expression feature value of the extracted face image is a non-smiling face label, f (x) is the average probability of the face distribution probability of a standard non-smiling face library, and when the expression feature value of the extracted face image is a smiling label, f (x) is the average probability of the smiling face distribution probability in the standard smiling face library; when the expression characteristic value of the extracted face image is the ugly smile label, f (x) is the average probability of the distribution probability of the ugly smile face in the standard smile library. Because the smiling face library comprises two class labels of smiling and ugly smile, the average probability of the face distribution probability of the standard library can better distinguish the two classes of smiling and ugly smile in the training process, and the feature difference of the smiling face image and the non-smiling face image can be enlarged.
In one embodiment of the present invention, the loss function includes a second loss function, configured to identify whether two images belong to the same label in a training process, and calculate a loss value of the two images, where the second loss function is:
Triloss(xi,xj,cij) A loss value, x, characterizing a second loss functioni、xjAll input standard library images are input, wherein cij is 1 to indicate that two input images are expressions of the same label, cij is 0 to indicate that two input images are expressions of different labels, and x ismIs the average of the expressive features of the standard smiley face library images.
For example, both images are non-smile labels, smile labels or ugly smile labels, cij1. The two images are a combination of a non-smile label and a smile label, a non-smile label and a ugly smile label, or a smile label and a smile label, cij＝0。
The second loss function reduces loss when the expression of the facial image to be recognized is a smiling face, and increases loss when the expression of the facial image to be recognized is a non-smiling face.
The loss function in step 206 may be other loss functions commonly used in the art, and is not limited to the first loss function and the second loss function.
According to the smiling face shooting method in the embodiment of the invention, the television can shoot the user, and the shot image containing the smiling face image can be further stored, so that the intelligence of the television can be improved, and the user can record the beautiful moment when using the television conveniently.
It should be understood that, although the steps in the flowchart of fig. 1 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a portion of the steps in fig. 1 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
The embodiment of the invention also provides a shooting device of the television, which comprises
The shooting module opening module is used for opening the shooting module if the television body is detected to be in the working state;
the face image detection module is used for shooting a user using the television body by using the shooting module and detecting a face image in a shot image;
and the photographed image storage module is used for inputting the human face images in the photographed images into a preset smiling face recognition model, and storing the photographed images if at least one of the human face images in the photographed images is recognized as a smiling face by using the smiling face recognition model.
The division of each module in the photographing device of the television is only for illustration, and in other embodiments, the photographing device of the television can be divided into different modules as needed to complete all or part of the functions of the photographing device of the television.
For specific limitations of the photographing apparatus of the television, reference may be made to the above limitations of the smiling face photographing method of the television, and details thereof are not repeated here. All or part of the modules in the photographing device of the television can be realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
An embodiment of the present invention further provides a television, including:
a television body;
a main control module;
the shooting module is connected with the main control module and is used for shooting a user using the television body;
the file management module is respectively connected with the main control module and the shooting module;
the main control module is used for turning on the shooting module if the television body is detected to be in a working state, shooting a user using the television body by using the shooting module, detecting a human face image in a shot image, inputting the human face image in the shot image into a preset smiling face recognition model, and recognizing the expression of the human face image in the shot image by using the smiling face recognition model;
the file management module is used for saving the shot images when at least one of the face images in the shot images is a smiling face.
For specific limitations of the television, refer to the foregoing embodiments, and are not described herein again.
The embodiment of the application also provides a computer readable storage medium. One or more non-transitory computer-readable storage media containing computer-executable instructions that, when executed by one or more processors, cause the processors to perform the steps of a smiling face photography method for a television.
A computer program product containing instructions which, when run on a computer, cause the computer to perform a smiling face photography method for a television.
Any reference to memory, storage, database, or other medium used herein may include non-volatile and/or volatile memory. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM), which acts as external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), synchronous Link (Synchlink) DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and bus dynamic RAM (RDRAM).
The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (9)
1. A smiling face photographing method for a television, the television comprising:
a television body;
the shooting module is arranged on the television body;
the smiling face photographing method includes:
if the television body is detected to be in the working state, the shooting module is opened;
shooting a user using the television body by using the shooting module, and detecting a human face image in a shot image;
inputting the face images in the shot images into a preset smiling face recognition model, and if at least one of the face images in the shot images is identified to be a smiling face by using the smiling face recognition model, storing the shot images;
the smiling face photographing method further includes the step of constructing the smiling face recognition model, which includes:
establishing a standard non-smiling face library and a standard smiling face library, wherein smiling face images and non-smiling face images are processed into preset sizes, the preset labels of the standard non-smiling face library comprise non-smiling face labels, and the preset smiling face category labels of the standard smiling face library comprise smiling labels and ugly label;
establishing a light-weight convolutional neural network structure for mobilenevv 3 deep learning, wherein the number of deep convolutional neural network layers of the light-weight convolutional neural network structure is 20, the initial weight and bias parameters of each neuron of each layer are both between positive and negative 1, the convolution kernels of each layer are respectively 1, 3, 5 and 7, the convolution kernel size of an input layer is 3 x 3, the convolution kernel size of an output layer is 1 x 1, the convolution kernel size of a pooling layer is 7 x 7, convolution step sizes are 1 and 2, the input layer uses a 224 x 3 matrix, and the output layer uses a 1 x 3 matrix, so that an initial smiling face recognition model is constructed;
and training the smiling face recognition model by using a loss function until the loss function converges to a loss value which is not reduced, wherein in the training of the light-weight convolutional neural network structure, the characteristic difference of the facial expressions with smiling face labels and non-smiling face labels is enlarged, the characteristic difference between the facial expressions with non-smiling face labels is reduced, the characteristic difference of the smiling face expressions with smiling and clown category labels is enlarged, and the expression characteristic difference of the smiling face expressions with smiling or clown category labels is reduced.
2. The method according to claim 1, wherein if at least one of the face images in the captured image is identified as a smiling face using the smiling face recognition model, the step of saving the captured image comprises:
if at least one of the face images in the shot images of the same type is identified to be a smiling face by using the smiling face identification model, the number of smiling face expressions in the shot images is identified, the definition of each shot image of the same type is calculated, and the shot image with the largest smiling face expression and the highest definition is stored.
3. The method of claim 1, wherein the plurality of camera modules are provided, and at least one camera module is provided at the top center of the TV screen, the bottom center of the TV screen and the centers of two sides of the TV screen.
4. The method according to claim 1, wherein the step of recognizing the smiling face expression of the human face image in the photographed image using the smiling face recognition model includes:
and after the face image in the shot image is zoomed into the preset size and normalized, the face image in the shot image is input into the smiling face recognition model after training, the smiling face recognition model extracts the smiling face expression characteristic value of the face image and sends the smiling face expression characteristic value into a classifier to judge and classify the smiling face, and smiling face recognition is realized.
5. The method of claim 1, wherein the loss function comprises a first loss function, and is used for judging the closeness degree of the real facial expression category and the actually output facial expression category, and the higher the closeness degree is, the smaller the loss value is; the first loss function is:
wherein, Loss (m, n, a) is the Loss value of the first Loss function in the training process, x is the expression characteristic value of the extracted face image, and aiTo extract the class label of the facial image, m (x) is the true facial expression distribution probability, n (x) is the expected output facial expression distribution probability, cos (θ)n) To predict the inner product angle of the probability, i.e.
cos(θm) The inner product included angle of the true probability,
when the expression characteristic value of the extracted face image is a non-smiling face label, f (x) is the average probability of the face distribution probability of a standard non-smiling face library, and when the expression characteristic value of the extracted face image is a smiling label, f (x) is the average probability of the face distribution probability of the smiling face in the standard smiling face library; extracting expression characteristic value of face image as ugly smile markWhen signing, f (x) is the average probability of the distribution probability of the ugly smile faces in the standard smile library.
6. The method of claim 5, wherein the loss function comprises a second loss function for identifying whether two images belong to the same label in the training process and calculating a loss value of the two images, and the second loss function is:
Triloss(xi,xj,cij) A loss value, x, characterizing a second loss functioni、xjAre all input standard library images, cij1 denotes an expression in which two input images are the same kind of label, cij0 denotes an expression where two input photographs are different labels, xmIs the average of the expressive features of the standard smiley face library images.
7. The method according to claim 1, wherein if at least one of the face images in the captured image is identified as a smiling face using the smiling face recognition model, the step of saving the captured image comprises:
and if the face images in the shot images are identified to be smiling labels, storing the shot images.
8. The method according to claim 1, wherein if at least one of the face images in the captured image is identified as a smiling face using the smiling face recognition model, the step of saving the captured image comprises:
establishing a folder, and identifying the folder according to a preset smiling face type label, wherein the number of the folders is the same as that of the preset smiling face type labels;
and storing the shot images of the same smiling face class label into a folder with a corresponding identifier.
9. A television set, comprising:
a television body;
a main control module;
the shooting module is connected with the main control module and is used for shooting a user using the television body;
the file management module is respectively connected with the main control module and the shooting module;
the main control module is used for turning on the shooting module if the television body is detected to be in a working state, shooting a user using the television body by using the shooting module, detecting a human face image in a shot image, inputting the human face image in the shot image into a preset smiling face recognition model, and recognizing the expression of the human face image in the shot image by using the smiling face recognition model; the main control module is further used for constructing the smiling face recognition model: establishing a standard non-smiling face library and a standard smiling face library, wherein smiling face images and non-smiling face images are processed into preset sizes, the preset labels of the standard non-smiling face library comprise non-smiling face labels, and the preset smiling face category labels of the standard smiling face library comprise smiling labels and ugly label; establishing a light-weight convolutional neural network structure for mobilenevv 3 deep learning, wherein the number of deep convolutional neural network layers of the light-weight convolutional neural network structure is 20, the initial weight and bias parameters of each neuron of each layer are both between positive and negative 1, the convolution kernels of each layer are respectively 1, 3, 5 and 7, the convolution kernel size of an input layer is 3 x 3, the convolution kernel size of an output layer is 1 x 1, the convolution kernel size of a pooling layer is 7 x 7, convolution step sizes are 1 and 2, the input layer uses a 224 x 3 matrix, and the output layer uses a 1 x 3 matrix, so that an initial smiling face recognition model is constructed; training the smiling face recognition model by using a loss function until the loss function converges to a loss value which is not reduced, wherein in the training of the lightweight convolutional neural network structure, the feature difference of the facial expressions with smiling face labels and non-smiling face labels is enlarged, the feature difference between the facial expressions with non-smiling face labels is reduced, the feature difference of the smiling face expressions with smiling and clown category labels is enlarged, and meanwhile, the expression feature difference of the smiling face expression with smiling or clown category labels is reduced;
the file management module is used for saving the shot images when at least one of the face images in the shot images is a smiling face.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010060099.XA CN111314760B (en) | 2020-01-19 | 2020-01-19 | Television and smiling face shooting method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010060099.XA CN111314760B (en) | 2020-01-19 | 2020-01-19 | Television and smiling face shooting method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111314760A CN111314760A (en) | 2020-06-19 |
| CN111314760B true CN111314760B (en) | 2021-07-02 |
Family
ID=71148324
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010060099.XA Active CN111314760B (en) | 2020-01-19 | 2020-01-19 | Television and smiling face shooting method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111314760B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112637487A (en) * | 2020-12-17 | 2021-04-09 | 四川长虹电器股份有限公司 | Television intelligent photographing method based on time stack expression recognition |
| CN113155292B (en) * | 2021-03-30 | 2024-06-28 | 芯算一体(深圳)科技有限公司 | Face temperature measuring method, face temperature measuring instrument and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101325658A (en) * | 2007-06-13 | 2008-12-17 | 索尼株式会社 | Imaging device, imaging method and computer program |
| CN102427553A (en) * | 2011-09-23 | 2012-04-25 | Tcl集团股份有限公司 | Television program playing method and system, television and server |
| CN103873914A (en) * | 2014-02-27 | 2014-06-18 | 深圳市九洲电器有限公司 | Audience life image shooting method, device and multimedia equipment |
| CN105512624A (en) * | 2015-12-01 | 2016-04-20 | 天津中科智能识别产业技术研究院有限公司 | Smile face recognition method and device for human face image |
| CN108174095A (en) * | 2017-12-28 | 2018-06-15 | 努比亚技术有限公司 | Photographic method, mobile terminal and computer-readable medium based on smiling face's identification |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20120064563A (en) * | 2010-12-09 | 2012-06-19 | 한국전자통신연구원 | Apparatus for controlling facial expression of virtual human using heterogeneous data |
-
2020
- 2020-01-19 CN CN202010060099.XA patent/CN111314760B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101325658A (en) * | 2007-06-13 | 2008-12-17 | 索尼株式会社 | Imaging device, imaging method and computer program |
| CN102427553A (en) * | 2011-09-23 | 2012-04-25 | Tcl集团股份有限公司 | Television program playing method and system, television and server |
| CN103873914A (en) * | 2014-02-27 | 2014-06-18 | 深圳市九洲电器有限公司 | Audience life image shooting method, device and multimedia equipment |
| CN105512624A (en) * | 2015-12-01 | 2016-04-20 | 天津中科智能识别产业技术研究院有限公司 | Smile face recognition method and device for human face image |
| CN108174095A (en) * | 2017-12-28 | 2018-06-15 | 努比亚技术有限公司 | Photographic method, mobile terminal and computer-readable medium based on smiling face's identification |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111314760A (en) | 2020-06-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110135406B (en) | Image recognition method and device, computer equipment and storage medium | |
| CN110399799B (en) | Image recognition and neural network model training method, device and system | |
| CN112597941B (en) | Face recognition method and device and electronic equipment | |
| CN111989689B (en) | Method for identifying an object in an image and mobile device for executing the method | |
| US20170083792A1 (en) | Similarity-based detection of prominent objects using deep cnn pooling layers as features | |
| Han et al. | Face recognition with contrastive convolution | |
| CN113435330B (en) | Video-based micro-expression recognition method, device, equipment and storage medium | |
| CN108388876A (en) | A kind of image-recognizing method, device and relevant device | |
| Agarwal et al. | Anubhav: recognizing emotions through facial expression | |
| Dhall et al. | Finding happiest moments in a social context | |
| US20200349414A1 (en) | Systems and methods for neuronal networks for associative gestalt learning | |
| CN108960080A (en) | Based on Initiative Defense image to the face identification method of attack resistance | |
| CN106339719A (en) | Image identification method and image identification device | |
| CN109063626B (en) | Dynamic face recognition method and device | |
| CN110287836B (en) | Image classification method and device, computer equipment and storage medium | |
| CN113076905B (en) | Emotion recognition method based on context interaction relation | |
| Xia et al. | Face occlusion detection based on multi-task convolution neural network | |
| CN106803054B (en) | Faceform's matrix training method and device | |
| CN111314760B (en) | Television and smiling face shooting method thereof | |
| CN111183455A (en) | Image data processing system and method | |
| JP2021515321A (en) | Media processing methods, related equipment and computer programs | |
| US11631277B2 (en) | Change-aware person identification | |
| WO2021217919A1 (en) | Facial action unit recognition method and apparatus, and electronic device, and storage medium | |
| Wang et al. | High-order tensor pooling with attention for action recognition | |
| CN116645700B (en) | Feature extraction model processing method and device and feature extraction method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20230712 Address after: 13C-18, Caihong Building, Caihong Xindu, No. 3002, Caitian South Road, Gangsha Community, Futian Street, Futian District, Shenzhen, Guangdong 518033 Patentee after: Core Computing Integrated (Shenzhen) Technology Co.,Ltd. Address before: 518000 1001, building G3, TCL International e city, Shuguang community, Xili street, Nanshan District, Shenzhen City, Guangdong Province Patentee before: Shenzhen Aishen Yingtong Information Technology Co.,Ltd. |
|
| TR01 | Transfer of patent right |