CN110570476A

CN110570476A - System, method and storage medium for execution on computing device

Info

Publication number: CN110570476A
Application number: CN201910104295.XA
Authority: CN
Inventors: 郭家祯; 黄鹤超
Original assignee: British Cayman Islands Business By Ltd By Share Ltd
Current assignee: British Cayman Islands Business By Ltd By Share Ltd
Priority date: 2018-06-06
Filing date: 2019-02-01
Publication date: 2019-12-13

Abstract

a system, method and storage medium for execution on a computing device. A computing device has a digital video camera. The computing device obtains a reference image depicting at least one reference color and calibrates parameters of the digital camera based on the at least one reference color. The computing device captures a digital image containing an individual with the digital camera using the calibrated parameters. The computing device defines a region of interest within a facial region of the individual in a digital image captured by the digital camera. The computing device generates skin color information in pixels in the area of interest and displays a predetermined recommended cosmetic product based on the skin color information.

Description

system, method and storage medium for execution on computing device

Technical Field

The present invention relates to a system and method for generating skin tone information for an individual represented by a digital image, and more particularly, to a system, method and storage medium for execution on a computing device.

background

With the popularity of smart phones, tablets and other displays, people can take digital images at any time, and applications that can manage and edit the digital content taken are also popular. However, there are many settings for cameras that are adjusted for color temperature, ambient light. It is difficult to accurately estimate the attributes (e.g., skin tone) of the facial region of the individual depicted therein, using digital images alone. Therefore, there is a need for an improved system for estimating skin tone information.

Disclosure of Invention

The present invention is directed to a system, method and storage medium for execution on a computing device that addresses the deficiencies of the prior art.

in one embodiment, a method performed on a computing device includes: a reference image is obtained, and at least one reference color is drawn on the reference image. The computing device has a digital camera, and the parameters of the digital camera are calibrated according to at least one reference color. Using the calibrated parameters, a digital image containing an individual is captured by the digital camera. In a digital image captured by a digital video camera, a region of interest is defined within a facial region of an individual. A skin tone information in pixels is generated in the region of interest. And displaying a predetermined recommended cosmetic product according to the skin color information.

Preferably, the reference image is depicted on an object that is: a white balance card, a color correction card, a bank note, a credit card, a copy paper, a thin paper, a mobile phone, or a matte white object.

Preferably, the object on which the reference image is depicted is at a predefined distance from the digital camera.

Preferably, the parameters of the digital video camera include at least one of: white balance level, exposure compensation, and gamma correction.

preferably, the step of defining a region of interest within the face region comprises: a color distance between pixels of the face region and one or more predetermined target skin tones is determined. One or more pixels having a color distance within a threshold of a predetermined target skin tone are designated as the region of interest.

preferably, the step of defining a region of interest within the face region comprises identifying the location of a plurality of predetermined feature points within the face region. A boundary of the region of interest is defined based on the locations of the plurality of predetermined feature points.

preferably, the step of generating a skin tone information in pixels in the region of interest comprises: a luminance histogram of the pixels in the region of interest is generated. A predetermined portion of the luminance histogram is removed to produce a target portion histogram. And determining a dominant color value according to the target partial histogram. Generating skin color information according to the determined dominant color value.

Preferably, the step of determining a dominant color value from the target portion histogram further comprises one of: the average of the target portion histogram is calculated. The peak of the target portion histogram is calculated. And calculating the weighted average value of the target part histogram. Or calculating the average value of the target part histogram according to the Mean-Shift clustering algorithm.

Preferably, the step of generating a skin tone information in pixels in the region of interest comprises: a light emitting layer and a reflective layer are obtained from the pixels in the region of interest. Skin color information is generated from the reflective layer.

Preferably, the step of generating a skin tone information in pixels in the region of interest comprises one of: converting a detected skin color from a first color space to a second color space according to a predefined conversion matrix; or mapping a detected skin tone from a first category to a second category according to a predefined look-up table.

in yet another embodiment, a system executing on a computing device comprises: the system comprises a digital video camera, a memory and a processor, wherein the memory stores a plurality of instructions. The processor is coupled to the memory and configured with a plurality of instructions, the plurality of instructions comprising: a reference image is obtained, and at least one reference color is drawn on the reference image. The parameters of the digital camera are calibrated according to at least one reference color. Using the calibrated parameters, a digital image containing an individual is captured by the digital camera. In a digital image captured by a digital video camera, a region of interest is defined within a facial region of an individual. A skin tone information in pixels is generated in the region of interest. And displaying a predetermined recommended cosmetic product according to the skin color information.

Preferably, the instructions for the processor to define a region of interest within the face region comprise: a color distance between pixels of the face region and one or more predetermined target skin tones is determined. One or more pixels having a color distance within a threshold of a predetermined target skin tone are designated as the region of interest.

preferably, the instructions for the processor to define a region of interest within the face region comprise: locations of a plurality of predetermined feature points are identified within the face region. A boundary of the region of interest is defined based on the locations of the plurality of predetermined feature points.

Preferably, the instructions for the processor to generate a skin tone information in pixels in the region of interest comprise: a luminance histogram of the pixels in the region of interest is generated. A predetermined portion of the luminance histogram is removed to produce a target portion histogram. And determining a dominant color value according to the target partial histogram. Generating skin color information according to the determined dominant color value.

preferably, the instructions for the processor to generate a skin tone information in pixels in the region of interest comprise: a light emitting layer and a reflective layer are obtained from the pixels in the region of interest. Skin color information is generated from the reflective layer.

In yet another embodiment, a non-transitory computer readable storage medium stores instructions for execution by a computing device having a processor, the instructions when executed by the processor at least perform: a reference image is obtained, and at least one reference color is drawn on the reference image. Parameters of a digital camera are calibrated according to at least one reference color. Using the calibrated parameters, a digital image containing an individual is captured by the digital camera. In a digital image captured by a digital video camera, a region of interest is defined within a facial region of an individual. A skin tone information in pixels is generated in the region of interest. And displaying a predetermined recommended cosmetic product according to the skin color information.

For a better understanding of the features and technical content of the present invention, reference should be made to the following detailed description of the invention and accompanying drawings, which are provided for purposes of illustration and description only and are not intended to limit the invention.

Drawings

FIG. 1 illustrates a block diagram of a computing device generating skin tone information in accordance with various embodiments of the invention.

FIG. 2 illustrates a schematic diagram of the computing device of FIG. 1 in various embodiments of the invention.

Fig. 3 illustrates a top level flow diagram for performing the functionality of a portion of the computing device of fig. 1 to generate skin tone information in various embodiments of the present invention.

FIG. 4 is a diagram illustrating the computing device of FIG. 1 defining a region of interest in various embodiments of the invention.

FIG. 5 illustrates a luminance histogram of pixels in a region of interest generated by the computing device of FIG. 1 in various embodiments of the invention.

Detailed Description

The following is a specific example that illustrates how to accurately generate skin tone information for an individual depicted in a digital image. Accurate determination of skin color information is important for virtual application of makeup effects or for recommending appropriate makeup product applications. The system for generating skin tone information will be described in detail below, followed by a description of the operation of the components within the system. Fig. 1 is a block diagram of when a computing device 102 generates skin tone information. The computing device 102 may be implemented as, for example, but not limited to: a smart phone, a desktop computing device, and a notebook computer.

A document generator 104 executes on a processor of the computing device 102, the document generator 104 including a reference color acquirer 106, a calibration unit 108, a camera interface 110, and a content analyzer 112. The reference color acquirer 106 is configured to acquire a reference image depicting one or more reference colors, which may be depicted on a white balance card, a banknote, or other source having a known color reference. The calibration unit 108 is configured to calibrate parameters of the digital camera based on the one or more reference colors.

the camera interface 110 is configured to generate a digital camera to capture a digital image of the individual. As known to those skilled in the art, the original image can be encoded in the form of, for example but not limited to: JPEG (Joint photographic Experts group) File, TIFF (tagged Image File Format) File, PNG (Portable Network graphics) File, GIF (graphics Interchange Format) File, BMP (bitmap) File or other types of digital File forms, but are not limited thereto. In addition, the original image can also be obtained from a still image of the video, such as but not limited to: MPEG-1(Motion Picture Experts Group-1), MPEG-2, MPEG-4, H.264, 3GPP (third Generation Partnership project), 3GPP-2, SD-Video (Standard-Definition Video), HD-Video (High-Definition Video), DVD (Digital Video disc) multimedia, VCD (Video Compact disc) multimedia, HD-DVD (High-Definition Digital Video disc) multimedia, DTV/HDTV (Digital Video/High-Definition Digital Video) multimedia, AVI Media (Audio Video inter), DV (Digital Video), Windows (Windows) QT file, WMV Video, Video Audio Video (Video) multimedia, Video System (WM) multimedia, Video (Video) Video, Video (MPEG-4, H.264, 3GPP Video, HD-Video-HD-Video, HD-DVD-HD-Video-HD-DVD (Video-HD-DVD), Windows (Video-HD-Video-HD-DVD-HD-Video-DVD (Video-HD-Video-HD-DVD (Windows-Video-HD-TV) multimedia, Windows-Video-HD-Video-HD-, 3D Scan Model (3D Scan Model) or other kind of digital form.

the content analyzer 112 is configured to define a region of interest in a facial region of the individual depicted in the digital image captured by the digital video camera. The content analyzer 112 is further configured to generate skin tone information in pixels within the region of interest. The content analyzer 112 is further configured to obtain the recommended cosmetic products 118 from a database 116 based on the generated skin tone information and to display the recommended cosmetic products 118 to the user of the computing device 102 using a user interface.

FIG. 2 illustrates a block schematic diagram of the computing device 102 in FIG. 1. Computing device 102 may be implemented as any of a variety of wired or wireless computing devices. Such as a desktop Computer, a portable Computer, a Dedicated Server Computer (dedicate Server Computer), a multitasking computing device (Multiprocessor computing device), a smart phone or tablet, and so forth. Referring to fig. 2, the computing device 102 includes a memory 214, a processing device 202, a plurality of Input/Output interfaces (I/O interfaces) 204, a network Interface 206, a display 208, a peripheral Interface 211, and a mass storage 226, each of which is connected via a Local Data Bus (Local Data Bus) 210.

the Processing device 202 may include any custom made or commercially available processor, a Central Processing Unit (CPU) or a coprocessor among several computing devices 102, a semiconductor microprocessor (in the form of a microchip), a Macroprocessor (microprocessor), one or more Application Specific Integrated Circuits (ASICs), a plurality of suitably configured digital logic gates, and other conventional electronic configurations including discrete components for coordinating the overall operation of the computing system, both individually and in various combinations.

The Memory 214 may include either Volatile Memory components (Volatile Memory Elements) or Nonvolatile Memory components (Nonvolatile Memory Elements). For example, the volatile Memory components include Random Access Memory (RAM), such as Dynamic Random Access Memory (DRAM) or Static Random Access Memory (SRAM). The nonvolatile Memory component may be a Read-Only Memory (ROM), a hard disk, a magnetic tape, a compact disc Read-Only Memory (CDROM). The memory 214 generally includes a native operating System 216, one or more native applications (native Application), Emulation System (Emulation System), or Emulated Application (Emulated Application) for any kind of operating System and/or Emulated hardware platform or Emulated operating System. For example, the aforementioned applications (i.e., native applications or simulation applications) may include specific software, i.e., include some or all of the components of computing device 102 in FIG. 1. In such embodiments, the components are stored in the memory 214 and executed by the processing device 202, and thus, the processing device 202 may perform the operations/functions of the features disclosed herein. The components in the memory 214 are well known to those skilled in the art, and therefore some of the components in the memory 214 are not described in detail for the sake of brevity. In some embodiments, the computing device 102 may execute in hardware and/or software.

Input/output interface 204 provides any number of interfaces to input or output data. For example, when the computing device 102 comprises a personal computer, the aforementioned components may be connected to one or more input/output interfaces 204, such as a keyboard and mouse, as shown in FIG. 2. The display 208 includes a computer display, a plasma screen of a personal computer, a Liquid Crystal Display (LCD) of a handheld device, a touch screen, or other display device.

In the present disclosure, a non-transitory computer readable medium stores a program for use by or in connection with an instruction execution system, apparatus, or device. More specifically, specific examples of the computer-readable medium can include, but are not limited to, a Portable computer diskette, a random access Memory, a Read-Only Memory, an Erasable Programmable Read-Only Memory (Erasable Programmable Read-Only Memory, EPROM, EEPROM, or Flash Memory), and a Portable Compact Disc Read-Only-Memory (CDROM).

Referring to fig. 3, fig. 3 is a flow diagram 300 illustrating various embodiments of generating skin tone information performed by the computing device 102 of fig. 1. The flow diagram 300 in fig. 3 is merely exemplary of different types of functional arrangements as to the operation of various components of the computing device 102 in fig. 1. In other words, the flowchart 300 of fig. 3 may be considered to describe one or more embodiments of steps in performing a method of the computing device 102.

Although a particular order of execution is disclosed in the flowchart 300 of fig. 3, the order of execution is merely to aid in understanding the present invention, and the actual order of operation may vary from that described. For example, the order of execution of two or more block diagrams may be adjusted, reversed, or the blocks may be combined. Also, multiple block diagrams in fig. 3 with a sequential order may be performed simultaneously or partially simultaneously. And such modifications and alterations are still within the scope of the present disclosure.

at block 310, the computing device 102 obtains a reference image depicting at least one reference color. In some embodiments, the reference image may be depicted on an object such as: a White Balance Card (White Balance Card), a Color Card (Color Checker), a bank note, a credit Card, a copy paper, a tissue, a mobile phone, or a White object without luster. In the above embodiment, the object with the reference image is located a predefined distance from the digital camera.

at block 320, the computing device 102 calibrates parameters of the digital camera according to at least one reference color, such as: white Balance Level (White Balance Level), Exposure Compensation (Exposure Compensation), Gamma Correction (Gamma Correction), and the like. At block 330, the computing device 102 captures a digital image including an individual with the digital camera using the calibrated parameters.

At block 340, the computing device 102 defines a region of interest within a facial region of the individual in the digital image captured by the digital camera. In some embodiments, the computing device 102 designates pixels of one or more predetermined target skin tones with a color distance within a threshold as the region of interest by determining a color distance (Acolor distance) between the pixels of the face region and the one or more predetermined target skin tones.

in some embodiments, the computing device 102 defines the region of interest by identifying a plurality of predetermined feature points within the face region and then defining a boundary of the region of interest based on the positions of the plurality of predetermined feature points. Further, FIG. 4 illustrates the definition of a region of interest 404 in various embodiments. As shown in FIG. 4, the computing device 102 analyzes the face region 402 of the individual and identifies a plurality of feature points (represented in dotted lines in FIG. 4). The computing device 102 generates a region of interest 404 based on the locations of the feature points.

In some embodiments, the computing device 102 detects the location of a plurality of feature points, including, for example, eyes, nose, mouth, eyebrows, and the like. The feature points may also include the entire outline of the user's face. The computing device 102 then defines a region of interest 404 based on the locations of the feature points. As shown in fig. 4, the computing device 102 may also define the boundary according to a series of parabolic curves that define or approximate a plurality of feature points. As shown in the exemplary example of fig. 4, the region of interest 404 includes cheek and nose regions of the user. That is, in some embodiments, the region of interest may be predefined based on a user-specific target region or feature (e.g., cheek and nose regions), and then a boundary may be defined around the target region or feature based on where the actual feature is detected on the user's face.

Referring back to FIG. 3, in block 350, the computing device 102 generates skin tone information in pixels within the area of interest. In some embodiments, the computing device 102 generates the skin tone information by: generating a luminance histogram of pixels in the attention area, removing a predetermined portion of the luminance histogram to generate a target portion histogram, determining a dominant color value according to the target portion histogram, and finally generating skin color information according to the determined dominant color value. The dominant color value may be determined in the following manner: calculating an average value of the histogram of the target portion, calculating a peak value of the histogram of the target portion, calculating a weighted average value of the histogram of the target portion, or calculating an average value of the histogram of the target portion according to a Mean-Shift Clustering Algorithm (Mean-Shift Clustering Algorithm).

The luminance histogram indicates the distribution of pixel luminance in the region of interest, which is usually calculated from the Y component of the YUV color space or the L component of the Lab (or CIELAB) color space. Further, referring to FIG. 5, FIG. 5 shows a luminance histogram 502 of pixels in a region of interest according to various embodiments of the present invention. In some embodiments, the computing device 102 removes predetermined portions 504, 506 of the luminance histogram 502 to generate a target portion histogram.

One of the predetermined portions 506 may include, for example, the pixels 30% of the luminance in the luminance histogram 502 (i.e., the pixels of the reflective region portion). In particular, the high brightness portions in the region of interest are typically caused by light reflections. The further predetermined portion 504 may comprise, for example, pixels 30% of the luminance in the luminance histogram 502 (i.e., pixels of the shaded area portion). In particular, the low brightness portions in the region of interest are typically caused by shadows.

An average dominant color value is determined based on the remaining histogram portion 508 (i.e., the target portion histogram), and skin color information is determined based on the average dominant color value. By excluding lighter and darker portions of the luminance histogram 502, the computing device 102 may reduce the impact of light reflections and shadows on the calculation of dominant color values, making compensatory corrections to correctly predict skin color.

In some embodiments, the computing device 102 generates the skin tone information by: a light emitting layer and a reflecting layer are obtained from pixels in the attention area, and skin color information is generated according to the reflecting layer. In some embodiments, the computing device 102 generates the skin tone information by converting a detected skin tone from a first color space to a second color space according to a predefined conversion matrix. Alternatively, the computing device 102 generates the skin tone information by mapping a detected skin tone from a first category to a second category according to a predefined look-up table.

in block 360, the computing device 102 displays the predetermined and recommended cosmetic product according to the skin tone information. In some embodiments, each recommended cosmetic product 118 in the database 116 (fig. 1) also includes a target RGB value or a range of target RGB values for a target skin tone. It is noted that in addition to the target RGB values, a target YUV value or a range of target YUV values may be stored in each of the recommended cosmetic products 118. Similarly, a target Lab value or a range of target Lab values may also be stored in each of the recommended cosmetic products 118. By pairing the estimated skin color information to one or more target RGB/YUV/Lab values for the corresponding product, the computing device 102 may obtain one or more recommended cosmetic products 118 from the database 116. Finally, the flow chart of fig. 3 ends.

The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the claims of the present invention, so that all the modifications of the equivalent technology using the contents of the present specification and the drawings are included in the scope of the claims of the present invention.

Claims

1. A method performed on a computing device, comprising:

Obtaining a reference image, wherein the reference image is depicted with at least one reference color;

The computing device is provided with a digital camera, and the parameters of the digital camera are calibrated according to at least one reference color;

shooting a digital image containing an individual through the digital camera by using the calibrated parameters;

Defining a region of interest in a facial region of the individual in the digital image captured by the digital camera;

Generating a skin tone information in pixels within the region of interest; and

And displaying a preset recommended cosmetic product according to the skin color information.

2. The method of claim 1, wherein the reference image is depicted on an object selected from the group consisting of: a white balance card, a color correction card, a bank note, a credit card, a copy paper, a thin paper, a mobile phone, or a matte white object.

3. the method of claim 2, wherein the object depicted with the reference image is a predefined distance from the digital camera.

4. The method of claim 1, wherein said parameters of said digital video camera include at least one of: white balance level, exposure compensation, and gamma correction.

5. The method of claim 1, wherein the step of defining the region of interest within the face region comprises:

determining a color distance between pixels of the face region and one or more predetermined target skin tones; and

Designating pixels of one or more of the predetermined target skin tones having the color distance within a threshold as the region of interest.

6. the method of claim 1, wherein the step of defining the region of interest within the face region comprises:

identifying locations of a plurality of predetermined feature points within the face region; and

And defining a boundary of the attention area according to the positions of the plurality of preset characteristic points.

7. the method of claim 1, wherein generating a skin tone information in pixels within the region of interest comprises:

Generating a luminance histogram of pixels in the region of interest;

removing a predetermined portion of the luminance histogram to generate a target portion histogram;

Determining a dominant color value according to the target partial histogram; and

Generating the skin color information according to the determined dominant color value.

8. The method of claim 7, wherein the step of determining a dominant color value from the target partial histogram further comprises one of:

Calculating an average value of the target portion histogram;

Calculating a peak value of the target portion histogram;

calculating a weighted average of the target portion histogram; or

And calculating the average value of the target part histogram according to a Mean-Shift clustering algorithm.

9. the method of claim 1, wherein generating a skin tone information in pixels within the region of interest comprises:

obtaining a light emitting layer and a reflecting layer from pixels in the attention area; and

Generating the skin color information according to the reflection layer.

10. The method of claim 1, wherein the step of generating a skin tone information in pixels within the region of interest comprises one of:

Converting a detected skin color from a first color space to a second color space according to a predefined conversion matrix; or

a detected skin tone is mapped from a first category to a second category according to a predefined look-up table.

11. A system executed on a computing device, comprising:

a digital video camera;

A memory, said memory storing a plurality of instructions; and

A processor coupled to the memory and configured with a plurality of the instructions, the plurality of instructions comprising:

calibrating parameters of the digital camera according to at least one reference color;

Generating a skin tone information in pixels within the region of interest; and

12. The system of claim 11, wherein the reference image is depicted on an object selected from the group consisting of: a white balance card, a color correction card, a bank note, a credit card, a copy paper, a thin paper, a mobile phone, or a matte white object.

13. The system of claim 11, wherein the parameters of the digital video camera include at least one of: white balance level, exposure compensation, and gamma correction.

14. the system of claim 11, wherein the instructions that the processor defines the region of interest within the face region comprise:

15. the system of claim 11, wherein the instructions that the processor defines the region of interest within the face region comprise:

16. the system of claim 11, wherein the instructions for the processor to generate a skin tone information in pixels within the region of interest comprise:

Generating a luminance histogram of pixels in the region of interest;

17. the system of claim 11, wherein the instructions for the processor to generate a skin tone information in pixels within the region of interest comprise:

Generating the skin color information according to the reflection layer.

18. A non-transitory computer readable storage medium storing instructions for execution by a computing device having a processor, wherein the instructions when executed by the processor cause the computing device to at least perform:

calibrating parameters of a digital camera according to at least one of the reference colors;

generating a skin tone information in pixels within the region of interest; and

19. the non-transitory computer readable storage medium of claim 18, wherein the reference image is depicted on an object that is: a white balance card, a color correction card, a bank note, a credit card, a copy paper, a thin paper, a mobile phone, or a matte white object.

20. The non-transitory computer readable storage medium of claim 18 wherein the parameters of the digital video camera include at least one of: white balance level, exposure compensation, and gamma correction.

21. The non-transitory computer-readable storage medium of claim 18, wherein the instructions that the processor defines the region of interest within the face region comprise:

22. the non-transitory computer-readable storage medium of claim 18, wherein the instructions that the processor defines the region of interest within the face region comprise:

23. The non-transitory computer-readable storage medium of claim 18, wherein the instructions for the processor to generate a skin tone information in pixels within the region of interest comprise:

Generating a luminance histogram of pixels in the region of interest;