KR20230118191A - digital makeup artist - Google Patents

Abstract

The digital makeup artist system includes a database system that stores a mobile device, makeup routine information, common makeup looks, cosmetics for skin type and race, and user look preferences of users. The mobile device includes a user interface for interacting with the digital makeup artist. The digital makeup artist conducts an interactive conversation with the user to capture the user's needs, including the type of makeup look, indoor or outdoor look, skin condition, facial problem areas, and favorite facial features. The computer operating circuitry analyzes the user's facial image to identify facial parts, analyzes the facial image to determine facial features, and determines the analyzed facial image, user's needs, stored makeup routine information, normal makeup look, skin type and Generate image frames to be displayed in synchronization with interaction with the digital makeup artist based on cosmetic, ethnic, and user look preferences.

Description

digital makeup artist

CROSS REFERENCES TO RELATED APPLICATIONS

This application is a regular application No. 17/138,078 filed on December 30, 2020, a regular application No. 17/138,143 filed on December 30, 2020, and a French application serial number filed on July 22, 2021 2107909, claims the benefit of priority to French Application Serial No. 2107906, filed July 22, 2021, the entire contents of which are incorporated herein by reference.

technical field

This disclosure relates to digital makeup artists and methods for interactive makeup advice and makeup tutorials.

Mobile applications or apps that assist in searching for and selecting cosmetics to purchase have recently been developed. Apps can provide tools for searching for a particular type of makeup, or searching for a user's favorites or previously used products that can be purchased. Some apps display a color palette to help you choose the color of your lipstick or eyeshadow. Some apps offer color-matching features that help you find colors that match the color of your clothing, accessories, or photos. Also, videos are available on how to apply certain types of makeup.

Some apps use the cameras of smartphones, tablets and laptops to provide try-on applications. Some of these applications are implemented as web applications or apps. Some of these applications involve using the camera to take a self-portrait picture with a smartphone camera, upload the picture to a web application, and then apply virtual makeup products to the uploaded image. These applications may offer a variety of options, such as smoothing skin, lifting cheekbones, and adjusting eye color. These applications may provide the user with the ability to change color intensity as well as add any type and color of makeup.

However, the try-on applications provided so far create the look through photo editing tools. Some of the existing try-on applications provide a one-step feature that starts with an uploaded photo, overlays the makeup type and color, and then allows you to edit the makeup photo. Additionally, photo editing tools for mobile devices with cameras have become popular for use with social media. One photo-sharing app, VSCO, allows users to add filters and edit photos before sharing them. Many VSCO filters are available to achieve specific effects. A VSCO filter can consist of values that can be applied to a picture, including exposure, temperature, contrast, fade, saturation, tint, and skintone. These photo editing tools provide the ability to edit photos, but photo editing does not provide a personal makeup experience. Also, previous try-on application tools do not provide for the creation of a custom look. For example, a user may want a date night look. Previous try-on applications may provide a date night look, but do not provide advice on which makeup products to use that might be best for the user and how different types of makeup products can be applied to create the look. . Instead, some editing may be performed by the user on the face image in an attempt to obtain a custom look. Also, the user may want a date night look based on his mood or the mood he wants to express.

These previous try-on web applications or apps lack complete personalization in part because they perform processes rather than customized makeup routines. A typical prior virtual try-on application process relies on templates and looks created for others. When a user wants a particular look he has in mind, or wants to experiment with a new look, the user may be faced with having to edit a look created for someone else. There is a need to provide a custom try-on experience for a particular user that allows interaction in a manner comparable to the experience the user might have with a personal makeup artist. There is a need for a makeup experience in which a personal makeup artist guides the user through the steps to achieve the look desired for their face.

Social media apps were developed to help users create their own cartoon avatars. These cartoon avatars can be customized with hairstyles, hair colors, facial shapes and colors, makeup, eyebrows, nose shapes, and the like. However, a user may wish to post a made-up image of his real face to social media.

The foregoing "Background" description is intended to generally present the context of the present disclosure. The work of the presently named inventors, to the extent set forth in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing are not expressly or implicitly admitted as prior art to the present invention.

One aspect is a digital makeup artist system comprising: a mobile device having a display device, computation circuitry and memory; a database system for storing user's makeup routine information, common makeup looks, cosmetics for skin type and ethnicity, and user look preferences; a machine learning system for analyzing images of faces; and the mobile device comprising a user interface for interacting with the digital makeup artist, wherein the digital makeup artist determines the type of makeup look, indoor or outdoor look, skin condition, facial problem areas, and favorite facial features. Performs an interactive dialog with the user to capture the user's request, including one or more of the following: The computer manipulation circuit unit inputs the user's face image, analyzes the user's face image through the machine learning system to identify a face part, and analyzes the face image to determine skin tone, eye color, hair color, determine a facial characteristic, including one or more of lip color, and skin texture, and generate image frames to be displayed on the display device in synchronization with interaction with the digital makeup artist. . The image frames include the analyzed face image of the user, the user's request obtained through interaction with the user, stored makeup routine information, a normal makeup look, cosmetics for skin type and race, and user look preferences. created based on one or more

One aspect is a digital makeup artist system comprising: a mobile device having a display device, computer operating circuitry and a memory; a database system that stores user's makeup routine information, common makeup looks, cosmetics for skin type and race, and user look preferences; a machine learning system for analyzing images of faces; and the mobile device includes a user interface for interacting with a digital makeup artist, wherein the digital makeup artist requests a makeup consultation, a type of makeup look, an indoor or outdoor look, a skin condition, a facial problem area, a favorite Conducting an interactive conversation with the user to provide advice including obtaining initial information including one or more of the facial features. The computer manipulation circuit unit inputs the user's face image, analyzes the user's face image through the machine learning system to identify a face part, and analyzes the face image to determine skin tone, eye color, lip color, determine facial features, including one or more of hair color, and skin texture, and generate image frames to be displayed on the display device synchronized with interaction with the digital makeup artist to provide advice. The image frames include one of the analyzed face image of the user, initial information obtained through interaction with the user, stored makeup routine information, a normal makeup look, cosmetics for skin type and race, and user look preferences. Based on the above, it is created in synchronization with the interaction.

The foregoing general description of exemplary embodiments and the detailed description below are merely exemplary aspects of the teachings of the present disclosure, and are not limiting.

A more complete understanding of the present disclosure and its many attendant advantages will be readily obtained when a better understanding is obtained by reference to the following detailed description when considered in conjunction with the accompanying drawings.
1 is a diagram of a system according to an exemplary aspect of the present disclosure.
2 is a block diagram of a computer system for a mobile device.
3 illustrates a user interface screen with an avatar in accordance with an exemplary aspect of the present disclosure.
4 illustrates a user interface screen for a mobile device according to an exemplary aspect of the present disclosure.
5 is a sequence diagram for interaction between a digital makeup artist and a user for a custom tutorial in accordance with an exemplary aspect of the present disclosure.
6 is a user interface for entering a type of look according to an exemplary aspect of the present disclosure.
7 is a flow diagram of facial analysis in accordance with an exemplary aspect of the present disclosure.
8 is a diagram of an exemplary architecture for a convolution neural network for classification of facial features.
9 is a diagram of an exemplary deep learning neural network for facial landmark detection.
10 is a diagram of a recommender system according to an exemplary aspect of the present disclosure.
11 illustrates a non-limiting look-feature matrix according to an exemplary aspect of the present disclosure.
12 is a user interface in a mobile application in accordance with an exemplary aspect of the present disclosure.
13 is a user interface in a mobile application in accordance with an exemplary aspect of the present disclosure.
14 is an exemplary mobile application according to an exemplary aspect of the present disclosure.
15 is a flow diagram for a SLOW DOWN video control command in accordance with an exemplary aspect of the present disclosure.
16 is a flow diagram for a PAUSE video control command in accordance with an exemplary aspect of the present disclosure.
17 is a flow diagram for a RE-DO LAST STEP video control command in accordance with an exemplary aspect of the present disclosure.
18 is a flow diagram for a SKIP CURRENT STEP video control command in accordance with an exemplary aspect of the present disclosure.
19 is a block diagram of a video playback component in accordance with an exemplary aspect of the present disclosure.
20 illustrates a blending process that can be used to create video frames based on desired and original features.
21A-21F are sequence diagrams of example interactions using a digital makeup artist for makeup tutorials in accordance with example aspects of the present disclosure.
22A-22E are sequence diagrams for example interactive use cases of using digital makeup artist 520 for a makeup tutorial in accordance with example aspects of the present disclosure.
23 is a sequence diagram for interaction between a digital makeup artist and a user for makeup consultation in accordance with an exemplary aspect of the present disclosure.
24A-24D are sequence diagrams of example interactions using a digital makeup artist for a makeup consultation in accordance with example aspects of the present disclosure.
In the following detailed description, reference is made to the accompanying drawings, which form a part of this application. In the drawings, like symbols typically identify like components, unless the context dictates otherwise. The illustrative embodiments described in the detailed description, drawings, and claims are not intended to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein.

Aspects of this disclosure relate to a digital makeup artist that can be consulted for makeup advice and tutorials in a manner comparable to the experience a user may have with a personal makeup artist. The disclosed digital makeup artist provides a makeup experience that guides the user through the steps to achieve a desired look for his/her face.

1 is a diagram of a system according to an exemplary aspect of the present disclosure. Embodiments include software applications, or mobile applications (apps). For purposes of this disclosure, herein the term mobile application (app) will be used interchangeably with software application, and makeup application will be used in reference to the process of virtually or physically applying makeup. The software application may run on a desktop computer or laptop computer 103 . A mobile application may run on a tablet computer or other mobile device 101 . For purposes of this disclosure, software applications and mobile applications are described in terms of mobile application 111 . In each case, the mobile application 111 may be downloaded and installed on the respective device 101 , 103 . In some embodiments, a desktop computer or laptop computer 103 may be configured with a microphone 103a as an audio input device. Microphone 103a may be a device that connects to desktop computer or laptop computer 103 via a USB port or audio input port or wirelessly via Bluetooth wireless protocol. The mobile device 101 may be equipped with a microphone. In some embodiments, a software application or mobile application may include communication functionality for operating with cloud service 105 . The cloud service 105 may include a database management service 107 and a machine learning service 109 . The database management service 107 may be any type of database management system provided in the cloud service 105, for example, the database management service 107 may include a database accessed using structured query language (SQL); or unstructured databases accessed by key, commonly referred to as No SQL. The machine learning service 109 may perform machine learning to allow for scaling up and high performance computing as may be required for machine learning. Additionally, a software application or mobile application may be downloaded from cloud service 105 . 1 shows a single cloud service, laptop computer and mobile device, it should be understood that any number of mobile devices, laptop computers, as well as desktop computers and tablet computers, may be connected to one or more cloud services.

2 is a block diagram of a mobile computing device. In one implementation, the functions and processes of mobile device 101 may be implemented by one or more respective processing/computation circuits 226 . The same or similar processing/computer operation circuitry 226 may be applied to a tablet computer or a notebook computer. The processing circuit includes a processor programmed as a processor comprising circuitry. The processing circuitry may also include devices such as application specific integrated circuits (ASICs) and conventional circuit components arranged to perform the noted functions. Note that circuitry refers to a circuit or system of circuits. Here, the circuitry may reside in one computer system or may be distributed across a network of computer systems.

Next, a hardware description of the processing/computer operation circuit 226 according to exemplary embodiments is described with reference to FIG. 2 . In FIG. 2 , processing/computer operation circuitry 226 includes a mobile processing unit (MPU) 200 that performs the processes described herein. Process data and instructions may be stored in memory 202 . These processes and instructions may be stored on a portable storage medium or stored remotely. The processing/computer operation circuitry 226 may have a replaceable Subscriber Identity Module (SIM) 201 containing information specific to the network service of the mobile device 101 .

Furthermore, the claimed advances are not limited by the form of computer readable medium on which the instructions of the process of the present invention are stored. For example, the instructions are flash memory, secure digital random access memory (SDRAM), random access memory (RAM), read only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM) , Electrically Erasable Programmable Read Only Memory (EEPROM), solid state hard disk, or any other information processing device with which processing/computer operation circuitry 226 communicates, such as a server or computer.

Further, the claimed advances may be provided as a utility application that runs in conjunction with the MPU 200 and the operating system, a background daemon, or a component of the operating system, or a combination thereof. The operating system may be for a laptop computer or desktop computer such as Mac OS, Windows 10, or a Unix operating system. For mobile devices such as smartphones or tablet computers, mobile operating systems such as Android, Microsoft® Windows® 10 Mobile, Apple iOS® and other systems known to those skilled in the art may be used.

To achieve the processing/computer manipulation circuitry 226, the hardware elements may be realized by various circuitry elements known to those skilled in the art. For example, MPU 200 may be a Qualcomm mobile processor, an Nvidia mobile processor, an Intel Corporation of America, Atom® processor, a Samsung mobile processor, or an Apple A7 mobile processor, or other processor types as will be recognized by one skilled in the art. there is. Alternatively, MPU 200 may be implemented on a Field-Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Programmable Logic Device (PLD) or using discrete logic circuits, as will be appreciated by those skilled in the art. . Further, MPU 200 may be implemented as multiple processors working cooperatively in parallel to carry out the instructions of the processes of the present invention described above.

The processing/computer operation circuitry 226 of FIG. 2 also includes a network controller 206 for interfacing with the network 224, such as an Intel Ethernet PRO network interface card from Intel Corporation of America. As can be appreciated, network 224 may be a public network such as the Internet, or a private network such as a LAN or WAN network, or any combination thereof, and may include a PSTN or ISDN subnetwork. Network 224 may also be wired, such as an Ethernet network. The processing circuitry may include various types of communication processors for wireless communications including 3G, 4G and 5G wireless modems, WiFi®, Bluetooth®, GPS, or any other known wireless form of communication.

A processing/computation circuit 226 includes a universal serial bus (USB) controller 225 that may be managed by the MPU 200 .

Processing/computer operation circuitry 226 further includes a display controller 208 such as an NVIDIA® GeForce® GTX or Quadro® graphics adapter from NVIDIA Corporation of America for interfacing with display 210 . I/O interface 212 interfaces with buttons 214, such as volume controls. In addition to I/O interface 212 and display 210 , processing/computer operation circuitry 226 may further include a microphone 241 and one or more cameras 231 . Microphone 241 may have associated circuitry 240 for processing sound into digital signals. Similarly, the camera 231 may include a camera controller 230 that controls an image capture operation of the camera 231 . The camera 231 may include a charge coupled device (CCD). Processing/computer operation circuitry 226 may include audio circuitry 242 for generating sound output signals, and may include an optional sound output port.

Power management and touch screen controller 220 manages power used by processing/computer operation circuitry 226 and touch controls. Communication bus 222, which is an Industry Standard Architecture (ISA), Extended Industry Standard Architecture (EISA), Video Electronics Standards Association (VESA), Peripheral Component Interface), etc. A description of the overall features and functionality of the display controller 208, power management controller 220, network controller 206, and I/O interface 212, as well as the display 210, buttons 214, will be discussed below. Because features are known, they are omitted here for brevity.

3 illustrates a user interface screen with an avatar in accordance with an exemplary aspect of the present disclosure. For purposes of this disclosure, the disclosed avatar is a graphical representation of a digital makeup artist, but may also be a photograph of the makeup artist 305 and represent a person who can communicate with the user using voice or text. Voice input can be activated by selecting the icon 311 representing a microphone. A subscreen 301 containing an avatar or picture 305 may provide an area for interaction. The avatar or picture 305 may output audio to the audio output 242 of the computer or external speakers connected to the computer 101 . The avatar or photo 305 may output text 309 . The subscreen may include an input box 307 into which a user may type text. In some embodiments, when a user speaks into microphone 241 , the voice is converted to text and the text is displayed in input box 307 . Subscreen 301 may be contained in user interface window 310 . Window 310 may be a graphical object controlled by an operating system of computer system 103 . Window 310 may display a browser window or a window of a software application or mobile application. The window 310 may include a menu icon 303 capable of displaying a menu of items that can be selected upon selection and that can perform a specified function.

Avatar 305 may be implemented as a software object that performs animation. In some embodiments, as described further below, avatar 305 may be synchronized with video playing on video component 320 . Control of the video component 320 may be performed by a voice command using the microphone 241 when the microphone icon 311 is activated. Avatar 305 can respond to voice input by performing natural language processing on the input and outputting a voice response via audio output 242 . To synchronize the avatar with the video, the avatar 305 can forward messages to the video component 320, and the video component 320 can share information with the avatar 305, such as timing information. . A video reproduced by the video component 320 may be divided into chapters. A chapter can be identified by a chapter name and time. The time can be from the start of the video in seconds or can be a percentage of the total length of the video. In some embodiments, the video is a sequence of makeup application steps created based on an image of the user's face and makeup to which the frames of the video are to be applied. For purposes of this disclosure, applying makeup may include applying a particular type of makeup to a particular portion of a user's facial image. A particular type of makeup may include a particular makeup product, and certain features of a makeup product.

In some embodiments, avatar 305 is implemented as a conversational agent capable of asking questions and providing solutions to questions. The interactive agent may also respond to certain types of questions queried by the user. The interactive agent may respond to statements made by the user that are responses to questions posed by the interactive agent. The interactive agent stores state information in the cloud service 105 . In some implementations, the interactive agent may have been previously created using software tools such as the Bot Framework SDK available for Microsoft Azure cloud services. The operation of the interactive agent will become more apparent from the description below.

4 illustrates a user interface screen for a mobile device according to an exemplary aspect of the present disclosure. In some embodiments, the user interface is provided for smaller screens, such as the screen in smartphone 101 . Given a smaller screen, interaction with the avatar 305 may be via audio and/or text. Voice input can be activated by selecting the icon 311 representing a microphone. A section of the display screen 210 may display controls and text associated with conducting a conversation. The avatar 305 may output audio to an audio output 242 of the computer or an external speaker connected to the computer 101 . The avatar 305 may output text 309 to the display 210 . A section of the display screen 210 may include an input box 307 where a user may enter text. In some embodiments, when a user speaks into microphone 241 , the voice is converted to text and the text is displayed in input box 307 . The display screen 210 may also display an image or video 401 of the user's face.

5 is a sequence diagram for interaction between a user and a digital makeup artist for a custom tutorial in accordance with an exemplary aspect of the present disclosure. Digital makeup artist 520 may take the form of avatar 305 or may be an interactive agent interacted with via voice. Digital makeup artist 520 may be performed by mobile application 111 and may represent a fictional person or a real makeup artist. System 510 may be a desktop computer, laptop computer 103, tablet computer, mobile device 101, or other computer system having display capabilities. System 510 may be a combination of cloud service 105 and any of a desktop computer, laptop computer, tablet computer, or mobile device. The system may have audio capabilities. User input 530 may be in the form of a text input area and/or a microphone input.

Conventional tutorial videos for applying makeup include basic video functions of Play, Pause, and Stop, and the time to move the starting point of video playback while video playback is paused. A time slider may be included. In the disclosed embodiments, digital makeup artist 520 is used to provide interaction with the custom tutorial.

The user may use the mobile application 111 with his/her own look in mind, but may use it for the purpose of showing how a digital makeup artist may create the look. Mobile application 111 may perform a custom tutorial to create the look. At 501 , digital makeup artist 520 of mobile application 111 may conduct an initial conversation with user 530 to obtain some information about the user, particularly the user's needs. Digital makeup artist 520 may start the conversation by displaying and/or saying an initial question, such as "What type of look do you want to create?"

At 503, the user can mention the desired type of look. The user's statement may be input as text in the input block 307 . Alternatively, the user may choose to interact with the digital makeup artist 520 via voice by selecting the microphone icon 311 . System 510 may extract keywords from user statements to be used to select pre-stored looks from database 107 . 6 is a user interface for inputting a user description for a type of look.

The user interface may provide a list of looks of various types. Examples of types of looks are season look (spring, summer, fall), event look (date night, dinner with girlfriend, special date, outing with mother-in-law, vacation, party, New Year's Eve, Bridal, Prom), looks based on completion time (quick makeup, average makeup, take-your-time makeup), mood look ( cheery, happy, notice-me), style (natural, evening, glam, gothic, work, beach), Aesthetic looks (VSOC, eGirl, soft girl) may be included.

The digital makeup artist 520 may further display and/or speak a question such as “What is your level of experience in performing makeup?” to obtain information about the user's level of experience. The user may be presented with a list of experience levels to choose from, or may input experience levels via voice. In one or more embodiments, information about user 530, including level of experience, may be found in a stored user profile.

The experience level may include a novice/beginner look, an experienced level, an expert level, and a professional. A novice/beginner level may be a user with little or no experience in applying makeup. An experienced level can be a user who has done makeup before, and thus has some experience. The expert level may be a user who has been wearing makeup for some time, such as for example, one year or more, and has gone through a step of learning the correct makeup method. A professional level can be a user applying to others.

The Digital Makeup Artist 520 may further display and/or ask questions such as "Is this a makeup look primarily for indoor or outdoor wear?" to refine the types of makeup that may be recommended. I can tell you.

The digital makeup artist 520 may additionally display and/or speak a question such as "Is your skin dry?" to obtain information about the user's skin condition, which may be difficult to obtain through image processing techniques. Other questions may relate to obtaining information about facial problem areas and preferred facial areas that the user may wish to consider.

The digital makeup artist 520 may perform a custom tutorial that teaches the steps to create the look desired by the user. Digital makeup artist 520 can use the user's own face to teach the steps, and the steps can apply makeup in a way that creates a look that will look when applied to the user's face. At 505, camera 231 may be used to capture an image or video of the user's face. A user can view the captured image or video on the display screen 210 .

At 507, the image or video may be analyzed by system 510 to obtain information of the user's face. For example, an image may be analyzed to identify a face contained in the image, determine a face shape, and determine parts of the face and other features that may be used to provide guidance during the tutorial. Conventional image processing algorithms are used to identify features within an image, such as image segmentation, brightness level, and average color. However, in the case of face recognition, an artificial neural network has recently been developed. Some artificial neural networks for face recognition are based on the structure of the human visual cortex. Architectures for artificial neural networks for computer vision continue to improve. For purposes of this disclosure, two non-limiting artificial neural networks are described that can be used to analyze a captured image or video.

In some embodiments, one or both of the artificial neural networks may be performed by mobile device 101 or laptop computer 103 if there are sufficient computer resources to perform the processing of the artificial neural networks. In some embodiments, one or both of the artificial neural networks may be performed by machine learning service 109 within cloud service 105 . In some embodiments, artificial neural networks may be trained in machine learning service 109 , and the trained artificial neural networks may be performed on mobile device 101 or laptop computer 103 .

To perform facial recognition, mobile application 111 may perform image processing operations to enhance image features, such as improving lighting. For example, when bright light or sunlight comes from a direction behind the user, the user may inadvertently take a self-picture. The mobile application 111 may brighten the user's face image. Other image processing operations may be performed to improve image quality.

7 is a more detailed flow diagram of the face analysis step.

At 701 , analysis may be performed on the captured image to determine a face shape. A machine learning model may be used to detect facial features of the user's captured face. A machine learning model can be trained to classify facial features using facial images with known facial features. Recently, image classification is performed using artificial neural networks inspired by how the visual cortex of the human brain behaves when recognizing objects. This image classification artificial neural network is a family of models known as convolutional neural networks (CNNs). There are other approaches proposed for image classification, which continue to improve. These different approaches for image classification that may be used for image classification include linear regression, decision trees, and support vector machines, to name a few. As mentioned above, the machine learning model can be trained using the machine learning service 109 of the cloud service 105 .

In the disclosed embodiments, the architecture of a machine learning model that can be used to classify a face shape is a CNN. 8 is a block diagram of a CNN for classifying a face shape. The CNN's dimensions and activation functions can vary depending on the available processing power and desired accuracy. Dimensions include the number of channels, the size of each layer and the number of layers. Activation functions include logistic, rectified linear unit, among others.

A convolutional neural network (CNN) can be composed of several types of layers. The convolution component 803 may consist of a convolution layer 803a, a pooling layer 803c, and a rectifying linear unit layer 803b. The convolution layer 803a is for developing a two-dimensional activation map that detects the special location of a feature at all given spatial locations. The pooling layer 803c acts as a form of downsampling. The commutation linear unit layer 803b applies an activation function to increase the decision function and non-linear properties of the entire network without affecting the receptive field of the convolutional layer itself. A fully connected layer 805 contains neurons with connections to all activations between previous layers. The loss layer specifies how the network training will penalize the deviation between the predicted layer and the actual layer. Loss layer 807 detects any class in a set of mutually exclusive classes.

In some embodiments, loss layer 807 may be a softmax function. The softmax function gives the probabilities for each class. In an exemplary embodiment, classes 809 may include square, rectangle, round, oval, oblong, diamond, triangle, and heart face shapes.

At 703, the mobile application can analyze facial features and landmarks. Similar to face shape, facial features and landmarks of a user's captured face may be detected using a machine learning model. A machine learning model can be trained to detect facial landmarks. As with face shape classification, other approaches to classification may be used. Also, a CNN architecture similar to that of FIG. 8 can be used for face landmark detection.

9 is a diagram of a deep learning neural network for facial landmark detection. Similar to the architecture of FIG. 8, the deep learning neural network is a convolutional neural network. To enhance learning of a deep learning neural network, residual connections may be included. In one or more embodiments, inverted residual structures may be included in which residual connections are made to previous layers of the network. Regarding the architecture of FIG. 9 , the network is presented as two stages 903 and 905 . The first stage 903 is a convolution stage for performing feature extraction. The second stage 905 performs prediction in regions of interest.

The architecture of the first stage 903 includes a convolution section 903a that, given an input face image 901 , performs convolution and max pooling operations. The convolutional section 903a is connected to the inverted residual structure 903b. The mask layer 903c is connected to the inverted residual structure 903b. The size of the mask layer 903c is based on the number of landmarks (eg 2 x L, number of landmarks). The mask layer 903c encodes the spatial layout of the input object.

The architecture of the second stage 905 includes an inverted residual structure 905b coupled to the inverted residual structure 903b of the first stage 903 . In addition, the mask layer 903c of the first stage 903 is applied to the results of the inverted residual structure 905b and performs region-of-interest cropping in the ROI and Concatenate Block 911. It is provided as an input for The ROI and concatenation block 911 is based on the number of landmarks and the number of channels in the inverted residual structure 905b. The prediction block 913 predicts landmarks and approximate locations in the mask layer 905c. The predictions for the regions of interest in the second stage 903 are combined with the landmarks estimated by the mask 903c for the total image to obtain the output landmarks in the output layer 907 .

In one or more embodiments, landmarks for the face include the area around the eyes, including the eyes, nose, lips, cheekbones, eyebrows, and eyelids, as well as hair. In some embodiments, landmarks may include possible facial anomalies.

The specific dimensions of each layer and the number of layers may depend on parameters including the desired accuracy, the hardware to perform the machine learning model, and the length of time to train the machine learning model. A machine learning model can be trained using the machine learning service 109 of the cloud service 105 .

Analysis of facial features 507 may further include detection of lip shape 705 , eyelid shape 707 , and hair style 709 . Detected landmarks may be used to compute contours of lips, eyes, and hair styles. In addition, other facial features such as skin color/tone, eye color, lip color, hair color 711 and skin texture 713 may also be determined from the facial image. Skin color/tone and skin texture can be determined using image processing techniques. A skin color/tone can be analyzed to determine RGB coordinates, and can be assigned a name for the skin color/tone. Analysis of skin color may reveal differences in skin tone. Skin tone types can include fair, light, medium or deep. In a similar manner, eye color, lip color and hair color can be determined using image processing techniques. Skin texture can be analyzed using analysis of brightness changes.

An additional feature of the face image may be lighting (image brightness). At 715, image illumination (brightness) may also be determined using image processing techniques. Brightness can be defined as a measure of the total amount of light perceived in an image. In some embodiments, the brightness of the image may be increased or decreased from its initial level as the captured brightness level.

At 717 , past look preferences may be retrieved from database 107 . Past look preferences may include makeup product characteristics including color coverage, shade, and finish used in past looks. Past user preferences may include makeup product characteristics for a particular part of the face, and may also include selection of a particular look, particularly makeup products applied for a particular part of the face.

At 509 , one or more makeup filters may be selected/retrieved from database 107 based on past look preferences and facial features (parts of the face) determined by facial analysis 507 and FIG. 7 . Some saved makeup face filters may be filters previously created through the creation of a custom look during a makeup consultation or custom tutorial with digital makeup artist 520 .

A makeup filter is a feature mask that can be blended into a base frame, i.e., an image with a frontal view of a user's face. The feature mask is blended with the base frame to create an image with the resulting facial image having modified features. The app can make adjustments to the feature mask to align with features of the face in the base frame using the location of the boundaries of the facial feature determined using face analysis 507 . A feature mask consists of RGB pixel values.

One or more makeup filters may be retrieved from database 107 using a recommender system. 10 is a diagram of a recommender system. The recommender system 1000 can be used to retrieve makeup filters to be used to show how to apply virtual makeup (515 in FIG. 5). The recommender system 1000 operates an indexed database 1005 of makeup filters and image data that may be stored in a database 107 .

The indexed database 1005 may also include answers to general questions regarding makeup, common makeup looks, makeup products for a particular race of a person and/or a particular skin type (skin tone, dry skin, skin texture). there is. Indexed database 1005 may be populated with makeup information extracted from external databases or web sites, such as product review information including ratings and public comments.

Indexed database 1005 may further include categories of cosmetics, including a category of makeup that provides a combination of functions such as appearance and skin care. Makeup products in this category may include foundations with anti-aging qualities or sun protection, and makeup products blended with drugs, or other skin treatment products. Another category could be skin care products for a skin care routine to be used before putting on makeup.

The recommender system 1000 includes a recommendation engine 1007 that retrieves and ranks recommended makeup filters. In the case of applying a specific virtual makeup, the recommended makeup filter may be for the look input by the user in step 501 and the virtual makeup. In some embodiments, recommended makeup filters may be retrieved based on user preferences or favorites. Personal user preferences may be makeup characteristics entered by the user when the app is first set up. An individual user preference may be one or more makeup filters previously established during a custom tutorial or consultation with the digital makeup artist 520 . A favorite may be a makeup feature that the user has flagged as a favorite. Individual user preferences and favorites may be for specific parts of the face or for the entire face.

In one or more embodiments, recommendation engine 1007 may use a look-feature matrix. 11 illustrates a non-limiting look-feature matrix according to an exemplary aspect of the present disclosure. The look-feature matrix shown in FIG. 11 is a partial matrix representing two types of virtual makeup for brevity. Other types of virtual makeup may be included in the look-feature matrix, including but not limited to foundation, mascara, concealer, cheek powder, and eyebrow pencil, to name a few. The look-feature matrix may be stored in an App of the mobile device to be compared with a vector of desired features. Desired features may be a vector of current user preferences, taking into account the user's current experience level and desired look (501). Recommendation engine 1007 may rank recommendations using one or more similarity metrics and scoring algorithms. In one embodiment, the recommendation engine 1007 may create a set of features that enhance the recommendation to encourage creativity by changing certain characteristics of the virtual makeup from being recommended. For example, if recommendation engine 1007 ranks a recommendation highly among retrieved recommendations, it may change one or more characteristics to increase the similarity score. Alternatively, recommendation engine 1007 may change one or more characteristics of the retrieved recommendation, such as shade or finish, one up or one down (e.g., change the shade one level up or one level down from a stored shade). can be changed). In one or more embodiments, the recommendation engine 1007 may more or less fine-tune the application gestures based on the user's experience level.

In one or more embodiments, recommendation engine 1007 may be implemented or supplemented by a machine learning model. A machine learning model can be trained to select a shade of foundation suitable for a particular skin undertone, skin type and/or ethnicity. A machine learning model can be trained to select a shade of lipstick based on the color of the user's lips. A machine learning model can be trained to select a shade of eyeglasses based on the user's eye color and skin undertone.

The machine learning model for recommender engine 1007 uses data in indexed database 1005 as well as data from external databases, especially external databases with images and videos that users can post their custom looks to. Can be trained using data. Exemplary external databases are for social media platforms that may be publicly available, such as Facebook, Instagram, Snapchat, TikTok, and video conferencing platforms such as Zoom, Microsoft Teams, Google Hangouts, or Google Meet, to name a few. contains database.

The machine learning model for the recommender engine 1007 can be based on decision trees, random forests, or algorithms on relatively small data sets including single layer neural networks - perceptrons. Given large data sets consisting of thousands of training data examples, a deep learning neural network can be used as a machine learning model. An architecture for a deep learning neural network can be a variant of a convolutional neural network for recognizing features in color images.

The recommendation engine 1007 may output 511 one or more recommendations to a recommendation user interface. The recommendation user interface may also display a sequence of video frames demonstrating application of the selected recommendation.

At 511 , one or more look options may be displayed on display screen 210 .

12 is an exemplary user interface in a mobile application. The digital makeup artist 520 may display custom recommendations 1201 of various looks. At 513, the user can select a recommended look. A look may be selected by using a pointing device to move the pointer on the display screen 210, a pointing device on the touch screen 221, or by inputting a voice command such as “Select LOOK1.” . There may be many more custom recommendations that may be displayed on display screen 210 . The user can view other recommended looks, for example, by selecting a function to display the next look's screen. In one embodiment, the user interface may provide a scroll bar 1203 that allows scrolling to view additional looks.

Once the user selects a look, at 515 system 510 may begin performing steps to create the selected look. 13 illustrates a user interface that may be displayed for starting a tutorial in a mobile application in accordance with an example aspect of the present disclosure. During the tutorial, at 517, information may be fed back to the system 510 to improve future interactions. Also, the tutorials are interactive. The tutorial can be controlled using commands such as SLOW DOWN, PAUSE, RE-DO A STEP, and SKIP A STEP. A message 1301 may be displayed to inform the user that commands may be used to control the tutorial. Commands may be selected on the user interface using a pointing device, or may be entered as verbal commands when microphone 241 is active (eg, via microphone icon 311 ). The tutorial being played can be modified to get a custom tutorial experience. For example, the RE-DO A STEP command may include an option to change the tutorial. System 510 may store changes made to a tutorial in database 107 as look preference data that may be used to improve future interactions. As mentioned above, the version of mobile application 111 may be for devices with smaller screens than laptop computers or desktop computers. 14 is an exemplary mobile application according to an exemplary aspect of the present disclosure. Given a smaller screen, interaction with the digital makeup artist 520 may be via audio and/or text. In this case, the command may be verbally input. During the tutorial, a facial image of user 1401 may be displayed, which includes applying makeup 1403a of color 1403b selected from makeup palette 1403 to portion 1405 of facial image 1401 . can do.

15, 16, 17 and 18 are flow charts for video control commands. As mentioned above, the tutorial can be controlled using commands including SLOW DOWN, PAUSE, RE-DO A STEP, SKIP A STEP. 15 is a flow diagram of a SLOW DOWN video control command. The SLOW DOWN command may perform a function of reducing the reproduction speed by a predetermined amount (X%) in S1501. Additionally, a call to a SLOW DOWN command may indicate that the user believes that the portion of the video being played is complex or requires careful viewing. Performing the function of reducing the playback speed may include, in S1503, storing information indicating the complexity of the above step.

16 is a flow chart of a PAUSE video control command. The PAUSE command, in S1601, may perform a function of stopping reproduction of a video at a specific point in time (position in seconds, or time code).

17 is a flow diagram of a RE-DO A STEP video control command. A RE-DO A STEP command may perform a function of utilizing a chapter. In some embodiments, the beginning of each chapter may include a starting image frame and a time point. Alternatively, the start of each chapter may be marked with a chapter frame. A RE-DO A STEP command can perform a function that can cause the video to jump back to the beginning of the current chapter and start playing again starting from the chapter's starting image. At S1701, the function may include reading the name of the current chapter. At S1703, the function may store information indicating the complexity of the steps associated with the current chapter. At 1705, the function can read an identifier indicating the starting position of a chapter, which can be a start point or a chapter frame.

As described below, in some embodiments, video frames may be dynamically generated, eg, prior to playback of a chapter, or while video frames are being played. By dynamically generating video frames, changes can be incorporated into the video, for example to use makeup of a different shade and/or finish. The beginning of a chapter includes a starting image, which may be a face image combined with a mask filter. Any makeup application performed during chapter playback can be saved as the next state of the face image combined with the mask filter.

In S1707, the video can be reproduced from the beginning of the chapter.

18 is a flow chart of a SKIP A STEP video control command. A SKIP A STEP command may perform a function that may cause the video to skip to the start of the next chapter. The function can be started by reading the name of the current chapter in S1801. Skipping a step may be an indication that the step is easy and does not require the user to be shown how to perform the step. The function may store information indicating that the step is easy in step S1803. In S1805, the function can read the name of the next chapter, and in S1807, it can read the time point associated with the next chapter. At S1809, the function may reset the video to start from the point of the next chapter.

19 is a block diagram of a video playback component in accordance with an exemplary aspect of the present disclosure. In some embodiments, video playback component 320 can generate a video for a tutorial before playing the video. In particular, the video playback component can perform actions on one or more frames in the tutorial video before rendering the frames for display. Also, a video of a tutorial may be divided into chapters. Chapters can be used to divide a video for a makeup tutorial into individual steps, where each step can be paused or stopped as well as played. Each chapter can be designed by a start frame. A chapter of the video may be for applying makeup to a specific part of the user's face, or may be for applying a specific type of makeup. A video may begin with an image of a user's face captured by a camera. The video may use information obtained through analysis of the facial image, such as location and labels of parts of the user's face image, color, texture, and lighting. The video may also use a look selected by the user. By default, the video may be a tutorial on how to create a selected look using an image of the user's face and information obtained from facial analysis.

As described above, a chapter of the video may be for applying makeup to a specific part of the user's face, or may be for applying a specific type of makeup. For example, a chapter may be a step of applying concealer to the eyelids of the user's face image. Concealer can be a makeup product provided for a selected look. In some embodiments, a makeup look can have a set of chapters, where each chapter can include one or more makeup products, where a makeup product can be divided into color, coverage, shade, and finish. have the same set of features. The video playback component 320 can create a selected look for a particular user's face and makeup products. A chapter may indicate cumulative results of makeup application starting with a face image of a user who has been made up in a previous chapter. In some embodiments, the cumulative result is one or more mask filters separate from the original face image.

Referring to FIG. 19 , at the beginning of a chapter, a face image or a mask filter 1903 for a face image may be provided as a start image to which additional digital makeup is applied for tutorial purposes. The beginning of the chapter is about the positional information 1907 of the part of the face (facial features) to which the makeup product is to be applied, as well as the makeup product, its characteristics, and the types of strokes that can be applied by a particular type of makeup applicator. information may be included. A chapter may include an accompanying audio component 1901 . A chapter can contain a sequence of mask filters 1905 of desired features that can be used to create video frames for the tutorial. Each video frame may be created by blending the previous frame 1911 with the desired feature 1913 to obtain the resulting feature 1915 . One or more feature masks may be used that represent applying specific makeup using a makeup applicator for specific facial features.

20 illustrates a blending process that can be used to create video frames based on desired and original features. Blending of facial features is accomplished as follows.

1. The desired feature (2001) is recolored (2003) to match the original feature's color(s) and obtain a recolored feature (2005).

2. The recolored features 2005 are multiplied by the feature mask filter 2007.

3. The original features 2009 are multiplied by the inverse 2011 of the feature mask filter (i.e. each mask value ranging from 1 minus 0 to 1).

4. The resulting images of 2 and 3 are summed pixel by pixel (2013) to create the final blended feature image (2015).

When the tutorial is complete, or when the user has completed as many tutorials as they wish to accomplish, the user interface may provide a choice to save the completed look. If the user chooses YES at 519 to save the finished look, then at 521 the look may be saved as a finished look, or the steps taken to apply makeup during the tutorial may be a custom filter or sequence of custom filters. can be stored as Also, at 523, the user interface may provide an option to send a custom look or to apply/migrate the custom filter as an image to other platforms (525).

Exemplary operation of the system is provided in FIGS. 21A-21F. 21A-21F are sequence diagrams for an example interactive use case of using a digital makeup artist 520 for a makeup tutorial in accordance with example aspects of the present disclosure. The sequence diagrams of FIGS. 21A-21F include operations and communications by system 510 , digital makeup artist 520 and user 530 . As noted above, system 510 may be a computer device such as a desktop computer, laptop computer 103, tablet computer, or mobile device 101, to name common types. System 510 may also take the form of a combination of a computing device and cloud service 105 (see FIG. 1). In either case, digital makeup artist 520 may be implemented as part of mobile application 111, and user 530 may interact with digital makeup artist 520 via voice, text, or a combination of voice and text. can

User 530 may first select a mobile application to be launched from the user interface of the computer device. Once the mobile application is launched, user 530 may be presented with the choice of using digital makeup artist 520 to teach in a tutorial or provide advice in a makeup consultation session (discussed below). If the user chooses to use the digital makeup artist 520 for the tutorial, in step 2101 the digital makeup artist 520 may ask "what look would you like to create?" At 2103, user 530 may mention the name of a look. The look may be an existing type of look or may be the name of a new look that the user 530 has in mind. Alternatively, system 510 may provide a list of existing looks for user 530 to select from. Existing looks may be stored in database 107 or provided locally to app 111 .

In some embodiments, at 2105, digital makeup artist 520 may ask user 530 to provide their level of experience in applying makeup. The experience level may be a category such as New, Experiences, Expert, and Professional. System 510 may present a list of experience levels for user 530 to select from. At 2107, user 530 indicates a level of experience.

At 2109, digital makeup artist 520 may ask user 530 how many parts of the face he would like to apply makeup in the tutorial to. A tutorial may be provided for part of the face or the entire face. At 2111, the user may select a tutorial provided for a part of the face, such as their eyes.

In some embodiments, at 2113, digital makeup artist 520 may ask user 530 how much time they have for the tutorial. A tutorial can be a shortened version if the user doesn't have much time, or a full version if the user has enough time for the full tutorial. User 530 may provide a qualitative response, or may provide a response that is time in minutes. At 2115, user 530 can provide a qualitative response, such as “not much time”. System 510 may interpret this response as an abbreviated version of the tutorial to be performed. However, user 530 may further express a willingness to save the tutorial in an intermediate state to be completed at a later time. In the latter case, the system 510 may store the tutorial at a point so that it can be played starting from an intermediate point.

In some embodiments, at 2117, digital makeup artist 520 wants user 530 to use any necessary makeup and makeup applicators already available, eg, makeup products purchased by user 530. User 530 may be asked to determine if desired, or if user 530 prefers that system 510 be able to provide makeup and makeup applicators that may be needed for the tutorial. At 2121, user 530 may provide a qualitative response indicating that they have some makeup products but system 510 is open to select makeup and makeup applicators.

In 2123, the digital makeup artist 520 may request the user 530 to take a picture of his/her face. A picture may be taken with a camera 231 built into the mobile device 101 or an external camera.

At 2125, system 510 may perform an analysis of the facial image. As mentioned above, facial analysis can be performed to obtain past look preferences as well as locations of parts of the face and facial features including skin color, skin texture, lighting, which can be used to create a video for a tutorial. This is information that can be used to

At 2127, system 2125 may select makeup for application in the tutorial.

At 2129, digital makeup artist 520 may ask user 530 if she would like to select makeup. At 2131, user 530 may respond that they do not agree with the selection, but rather will have the tutorial performed using the makeup provided by the user.

At 2133, digital makeup artist 520 may indicate that a tutorial is about to begin. In some embodiments, at 2135, digital makeup artist 520 may provide a list of products that can be used during the tutorial if user 530 wishes to obtain products for applying makeup during the tutorial. there is.

At 2141, system 510 may begin creating and playing a video of the tutorial. In an exemplary tutorial, digital makeup artist 520 may begin the tutorial by referring to an initial makeup that may be applied. At 2143, digital makeup artist 520 may speak or provide a text command indicating that concealer is to be applied first followed by primer.

The video of the tutorial may be played by system 510 with commands by digital makeup artist 520 via audio output, text output, or both audio and text. At S1845, the digital makeup artist 520 may provide a command for selecting an eye shadow. At 2147, digital makeup artist 520 may provide instructions for selecting a blending brush. At 2149, digital makeup artist 520 may provide instructions for applying eye shadow.

At points along the tutorial, user 530 may enter a video control command. For example, in 2151, the user 530 may input a command to RE-DO STEP. In some embodiments, the command may include a request to change in steps, such as Re-do with a one-up color. The digital makeup artist 520 may output a request for clarification by the user of what "one-up color" means. At 2153, system 510 may create new frames for the video starting with selecting an eye shadow in 2145 that uses the new color.

At 2155, digital makeup artist 520 may provide instructions for selecting a blending brush. At 2157, digital makeup artist 520 may provide instructions for applying eye shadow.

At 2159, user 530 may input a command such as PAUSE. System 510 may perform a pause function until user 530 inputs a PLAY command.

At 2161, digital makeup artist 520 may provide instructions for performing blending with other shades. At 2163, digital makeup artist 520 may provide a command to select an eyeshadow of the next shade. At 2165, digital makeup artist 520 may provide instructions for selecting a blending brush. At 2167, digital makeup artist 520 may provide instructions to perform blending.

At 2169, user 530 can enter a command such as "RE-DO A STEP" to rerun the blending. At 2171, system 510 may restart the video from the beginning of the blending step.

At 2173, digital makeup artist 520 may provide instructions for performing eyebrow sharpening. At 2175, digital makeup artist 520 may provide instructions for performing the wrinkle reducing step. At 2177, digital makeup artist 520 may provide instructions to perform the step of selecting a concealer with a pearl finish.

At 2179, digital makeup artist 520 may enter a statement that the concealer is too light and ask if the concealer can be toned down. At 2181, digital makeup artist 520 may respond by stating that it will try another concealer, which may be done by changing to another concealer and issuing a RE-DO A STEP.

At 2183, the system 510 may restart the video from the beginning of the wrinkle reduction step, depending on where the beginning of the chapter occurs. Then, at S1885, the system 510 may select a concealer with, for example, a mate finish, and play a video with the selected concealer. At 2187, user 530 may enter a PAUSE command to observe the results of the new concealer. At 2189, user 530 may input a PLAY command to start playing the video.

At 2191, digital makeup artist 520 may provide instructions for performing the step of shaping around the eyes. At 2193, digital makeup artist 520 may provide instructions for performing the eye lightening step.

At 2195, system 510 may select a pencil suitable for the user's skin tone. At 2197, system 510 may draw on the face image using the selected pencil.

At 2199, digital makeup artist 520 may provide instructions to perform the step of applying eyeliner. At this point, at 2001, user 530 may choose to pause the video to be resumed later.

At some time in the future after the video is complete, at 2203, digital makeup artist 520 may output a query as to whether the user wants to save the completed look. At 2205, the user may select to save the look and instruct system 510 to save the look. At 2207 , system 510 stores the final look in database 107 . In some embodiments, the final look may be stored in a database as a preferred look for a particular type of look. For example, a look created using a tutorial can be saved as a preferred look for the type of look initially described (see 503 in FIG. 5 ). The final look may also be transferred to other platforms such as social media platforms or video conferencing platforms. Some modern social media platforms on which users can post photos or videos include Facebook, LinkedIn, Instagram, TikToc, and Snapchat, to name a few. Some modern video conferencing platforms include Microsoft Teams, FaceTime, Google Hangouts or Google Meet, Zoom, GoToMeeting, and Skype. Additionally, changes made during the tutorial, such as color shade selection and color finish, can be stored as look preference data.

In some embodiments, at 2207 , system 510 may store one or more mask filters associated with creation of a final look in database 107 . One or more mask filters can be ported to other platforms and used to create custom looks for other platforms.

Social media and video conferencing have created a demand for makeup on-the-fly, social subcultures such as VSCO Girls, and aesthetic appeals such as eGirls and Soft girls. It also affected the look. There is a demand for virtual makeup looks for social media and video conferencing, namely augmented reality.

An example operation of a system for creating a virtual try-on for social media or videoconferencing is provided in FIGS. 22A-22E. 22A-22E are sequence diagrams for example interactive use cases using digital makeup artist 520 for makeup tutorials in accordance with example aspects of the present disclosure.

If the user chooses to use the digital makeup artist 520 for a tutorial on creating a virtual try-on for video conferencing, at 2221 the digital makeup artist 520 asks the question "what type of look do you want to create?" can do. At 2223, user 530 may state that they would like to apply the eGirl look. An eGirl look could be the look a user wants to post on social media or when videoconference with a friend or colleague. Existing eGirl looks may be stored in database 107 or served locally in app 111 .

In some embodiments, at 2225, digital makeup artist 520 may ask user 530 to provide their level of experience in applying makeup. Levels of experience can be categories such as New, Experienced, Expert, and Professional. System 510 may present a list of experience levels for user 530 to select from. At 2227, user 530 indicates a level of experience of being experienced, but new to applying the eGirl look.

At 2229, digital makeup artist 520 may ask user 530 how much of the face he/she would like to have the makeup of the tutorial applied to. A tutorial may be provided for part of the face or the entire face. At 2231, the user may select a tutorial presented for the full face.

In some embodiments, at 2233, digital makeup artist 520 may ask user 530 how much time they have for the tutorial. A tutorial can be a shortened version if the user doesn't have much time, or a full version if the user has enough time for the full tutorial. User 530 may provide a qualitative response, or may provide a response that is time in minutes. At 2235, user 530 can provide a qualitative response, such as “not much time”. System 510 may interpret this response as an abbreviated version of the tutorial to be performed. In some embodiments, at 2237, digital makeup artist 520 provides any necessary digital makeup and makeup applicators already available to user 530, e.g., digital makeup products purchased by user 530. User 530 may be asked to determine if he or she would like to use, or if user 530 would prefer that system 510 be able to provide digital makeup and makeup applicators that may be needed for the tutorial. At 2241, user 530 can provide a qualitative response to indicate that they have some digital makeup products.

At 2243, digital makeup artist 520 may request user 530 to take a picture of his/her face. A picture may be taken with a camera 231 built into the mobile device 101 or an external camera.

At 2245, system 510 may perform an analysis of the facial image. As mentioned above, facial analysis can be performed to obtain past look preferences as well as locations of parts of the face and facial features including skin color, skin texture, lighting, which can be used to create a video for a tutorial. This is information that can be used to

At 2247, system 510 may select digital makeup to apply in the tutorial.

At 2249, digital makeup artist 520 may ask user 530 if they would like to select digital makeup. At 2251, user 530 may respond that they do not agree with the selection, but rather will have the tutorial performed using the digital makeup provided by the user.

At 2253, digital makeup artist 520 may indicate that a tutorial is about to begin. In some embodiments, at 2255, digital makeup artist 520 may provide a list of digital makeup products that can be used during the tutorial if user 530 wishes to obtain products for applying makeup during the tutorial. there is.

At 2261, system 510 may begin creating and playing a video of the tutorial. In an exemplary tutorial, digital makeup artist 520 may begin the tutorial by stating initial makeup that may be applied. At 2263, digital makeup artist 520 may speak or provide a text command indicating to smooth the primer to prepare the foundation.

The video of the tutorial may be played by system 510 with commands by digital makeup artist 520 via audio output, text output, or both audio and text. At 2265, digital makeup artist 520 may provide commands to select a foundation with light coverage and natural finish. At 2267, digital makeup artist 520 may provide commands to sweep blush across the bridge of the nose and slightly extend it onto the cheeks. At 2269, digital makeup artist 520 may provide instructions to apply Highlighting Power to the tip of the nose using a tapered brush.

At points along the tutorial, user 530 may enter a video control command. For example, at 2271, the user 530 may input a RE-DO A STEP command. In some embodiments, the command may include a request to change in steps, such as Re-do with a one-step-up blush color. The digital makeup artist 520 may output a request for clarification on what the user means by “one step-up blush color”. At 2273, the system 510 may generate new frames for the video that begin the blush sweep at 2267 using the new blush color.

At 2275, digital makeup artist 520 may provide instructions to apply the eyebrow mechanical pencil in light strokes. At 2277, digital makeup artist 520 may provide a command to comb the eyebrows.

At 2279, user 530 may input a command such as PAUSE. System 510 may perform a pause function until user 530 inputs a PLAY command.

At 2281, digital makeup artist 520 may provide instructions to select an eye shadow of a pink shade. At 2283, digital makeup artist 520 may provide instructions to select a foot applicator for applying the eye shadow. At 2285, digital makeup artist 520 may provide instructions for applying eye shadow to the eyelid. At 2287, digital makeup artist 520 may provide instructions to perform blending on the crease with a fluffy blending brush.

At 2289, user 530 can enter a command such as "RE-DO A STEP" to redo the blending. At 2291, system 510 may restart the video from the beginning of the blending step.

At 2293, digital makeup artist 520 may provide instructions for creating wings across each eyelid using a wing stencil. At 2295, digital makeup artist 520 may provide instructions to perform the step of applying mascara to lashes using several swipes. At 2297, digital makeup artist 520 may provide instructions to apply a barely-there lip gloss.

At 2299, user 530 may enter a statement that the lip gloss is too light and ask if the lip gloss can be toned up.

At some time in the future after the video is complete, at 2301, the digital makeup artist 520 may output a query as to whether the user wants to save the completed look. At 2303, user 530 may select to save the look and instruct system 510 to save the look. At 2305 , system 510 stores the final look in database 107 . In some embodiments, the final look may be stored in a database as a preferred look for a particular type of look, such as a personal eGirl look. For example, an eGirl look created using the tutorial can be saved as a preferred look for the eGirl look initially described (see 503 in FIG. 5 ). The final look may also be transferred to other platforms such as social media platforms or video conferencing platforms. Some modern social media platforms on which users can post photos or videos include Facebook, LinkedIn, Instagram, TikToc, and Snapchat, to name a few. Some modern video conferencing platforms include Microsoft Teams, FaceTime, Google Hangouts or Google Meet, Zoom, GoToMeeting, and Skype. Also, changes made during the tutorial, such as color shade selection and color finish, may be stored as look preference data.

In some embodiments, at 2305 , system 510 may store one or more mask filters associated with creation of a final look in database 107 . One or more mask filters can be ported to other platforms and used to create a custom look for other platforms.

The digital makeup artist 520 may provide personal makeup consultation in addition to providing a tutorial service for creating a custom look. Digital makeup artist 520 may provide advice on how to enhance the user's makeup look, or may provide advice on improvements that may be made to the user's makeup application technique. The digital makeup artist 520 can provide advice on makeup application techniques that can better address problem areas, or make particular facial features stand out. The digital make-up artist 520 is based on the time of day (morning, afternoon, evening), where the user is expected to go (mainly indoors with artificial lighting, outdoors with primarily sun exposure and lighting), or current skin condition (dry skin). We can give you advice on makeup applications that may be suitable for you. The digital makeup artist 520 can provide advice on how to bring out the user's personality.

23 is a sequence diagram for interaction between a digital makeup artist and a user for a makeup consultation according to an exemplary aspect of the present disclosure.

As noted above, user 530 may be presented with the option of using digital makeup artist 520 to teach in a tutorial or provide advice in a makeup consultation session. At 2351, user 530 may request makeup advice via user interface window 310 (see FIG. 3). At 2353, system 510 may request user 530 to take a photo or video of their face. In a manner similar to the above for the tutorial, system 510 may analyze an image of the user's face to identify facial parts, detect skin tone, lip color, hair color, and skin texture. At 2355, system 510 conducts an interactive conversation with user 530 to obtain additional information related to the user's needs, including skin conditions, indoor/outdoor looks, favorite facial features, and any facial feature interests. can do. At 2357, user 530 may enter information of skin condition, eg, dry skin, indoor/outdoor look, favorite features, and facial feature interests.

Given user input information and preference information that may have previously been stored in database 107, and mask filters for the various types of looks stored in database 107, at 2359, system 510 provides makeup routines. Create one or more custom recommendations for As discussed above, custom recommendations may be obtained using a recommender system. Recommender system 1000 may be used to retrieve makeup filters to be used to create custom recommendations for makeup routines, and may be used to retrieve makeup routines, such as for skin care. At 2361, system 510 may display the retrieved one or more recommended makeup routines. A makeup routine can be created using the blending process shown in FIG. 20 .

The recommender system 1000 includes a recommendation engine 1007 that retrieves and ranks recommended makeup filters. Recommendation engine 1007 also retrieves from external storage for additional information including one or more of frequently asked questions and answers, common makeup looks, cosmetics for specific skin types and conditions, and ethnicity. can withdraw Recommendation engine 1007 may further fetch from external storage of cosmetic categories including skin care, and makeup with skin care qualities, where makeup with skin care properties is foundation with SFP to protect from sunlight, It may include makeup with anti-aging properties. In the case of a particular virtual makeup application, the recommended makeup filter may be for the preferred features and facial feature interests entered in step 2357 by the user.

In one or more embodiments, as described above, recommendation engine 1007 may be supplemented with a machine learning model to recommend a shade of makeup based on skin undertone.

In some embodiments, at 2363, user interface window 310 may take input from user 530 to refine and adjust the recommended makeup routine. Additional user input may be in the form of refinement or adjustment of specific features shown in the recommended routine. For example, the user may input that the lipstick color is too bold. Additional user input may be in the form of refinements or adjustments in the overall makeup routine. For example, the user may enter that the makeup routine is too bold, or that the user prefers longwear for makeup. At 2365, system 510 may perform adjustments to the recommended makeup routine according to the user input and the makeup look data that matches the type of adjustment. For example, system 510 can make adjustments by retrieving mask filters for long-wear makeup stored in database 107 . At 2367, system 510 may generate and display a modified makeup routine for the user's facial image using the retrieved mask filter. At 2369, system 510 may use the makeup routine to provide recommendations for makeup products that can be used to create the finished facial image.

Further, at 2371 , system 510 may store adjustments to the finished facial image and the makeup routine used to create the finished facial image in database 107 as makeup look preferences. For example, a mask filter for long-wear makeup may be stored together with a label of makeup look preference (make-up filter) for a makeup routine (look) in a feature matrix, as shown in FIG. 11 .

In one or more embodiments, at 2373, user 530 may choose to send/post the completed facial image created to a platform that presents live video or still images that the user wishes to attend. Platforms that provide live video include video conferencing platforms and social media platforms including Facebook, Linked-in, Google Hangouts or Google Meet, FaceTime, Microsoft Teams, TikToc, Zoom, to name a few.

To illustrate digital makeup artist 520, example operations are provided in FIGS. 24A-24D. 24A-24D are sequence diagrams of example interactions using a digital makeup artist for a makeup consultation in accordance with example aspects of the present disclosure. The sequence diagrams of FIGS. 24A-24D include operations and communications by system 510 , digital makeup artist 520 and user 530 .

At 2401, user 530 may select a mobile application for digital makeup artist. If user interface window 310 is presented, at 2403 user 530 may request to seek makeup advice. At 2405, digital makeup artist 520 may ask whether the user has a preferred look they would like advice on. At 2407, user 530 may provide an answer further defining the desired look. The digital makeup artist 520 may further narrow the type of advice to provide by asking the user at 2409 if they have makeup look preferences. User 530 may provide an answer indicating that it does not have a preference, but would rather let system 510 select a look at 2411 .

At 2413, system 510 may request that the user take a photo or video of their face, and will perform analysis on the image of the user's face. Analysis results may include the position of parts of the user's face, and features such as skin color, skin texture, lighting, as well as previous look preferences.

At 2415, digital makeup artist 520 may ask whether the user has favorite facial features. User 530 may respond at 2417 with one or more favorite facial features, such as lips. At 2421, system 510 can generate custom recommendations for one or more makeup routines. At 2423, system 510 may display the makeup routine and makeup product characteristics. At 2425, digital makeup artist 520 may ask the user if the user would like to have adjustments made to the recommended makeup routine. At 2427, user 530 may provide a response such as that the makeup look is too bold.

At 2429, digital makeup artist 520 may respond by asking if there is a particular part or parts of the face that stand out, or if the entire face is too bold. At 2431, user 530 may respond that the overall look is too bold. At 2433, digital makeup artist 520 can notify user 530 that adjustments to the makeup look will be made.

At 2435, system 510 performs adjustments to the makeup look, which may include retrieving mask filters from database 107, and selection of mask filters may consider past look preferences. At 2437, system 510 may display the adjusted makeup look. At 2439, user 530 can review the adjusted makeup look, provide additional feedback, such as that the eyes look slightly too dark, and request that the eyes be brightened.

At 2441, system 510 can make adjustments to the makeup look, and at 2443, it can display the additional adjusted look. At 2445, digital makeup artist 520 may notify the user that the system has increased the shade of the eye shadow, and ask the user if the adjustment sufficiently improved the brightness of the eye. At 2447, user 530 may respond with a statement that the adjustments are in the right direction, but that it would be better if slightly more adjustments were made.

At 2449, system 510 may further adjust the shade, and at 2451, display the further adjusted look. At 2453, digital makeup artist 520 may notify user again that the system has increased the shade, and may again ask user 530 if the adjustment was sufficient. At 2455, user 530 may respond that the adjustment looks good.

Once the makeup look is complete, at 2461, system 510 may display the final makeup look. Further, at 2463, system 510 may display makeup products that may be used to create a final makeup look, and at 2465, as user look preferences in database 107, makeup routine, final makeup look, and Adjustments made can be saved.

Through interaction with digital makeup artist 520, user 530 can improve their makeup look and improve their makeup application technique. The digital makeup artist 520 provides makeup advice tailored to the user. The digital makeup artist 520 continues to refine its recommendations through accumulation of user look preferences and custom looks. Digital makeup artist 520 can try on makeup for the user before the user applies makeup to their face, and teaches the user how to apply makeup to create a custom look. Additionally, the digital makeup artist 520 may create a custom makeup look based on stored look preferences and information about the user's facial features.

In one embodiment, the words "a", "an", etc. generally mean "one or more" unless stated otherwise.

Many modifications and variations of the present invention are possible in light of the above teachings. For example, data collected from skin tones and textures of various consumers will allow scaling of the artificial neural network beyond a single consumer. Artificial neural networks will be able to predict the presentation of new cosmetic formulas for each product shade. Accordingly, it is to be understood that within the scope of the appended claims, the invention may be practiced otherwise than as specifically described herein.

Accordingly, it is to be understood that within the scope of the appended claims, the invention may be practiced otherwise than as specifically described herein.

The disclosure also includes the embodiments listed below.

(1) Digital makeup artist system. The digital makeup artist system includes a mobile device having a display device, computation circuitry and memory; a database system for storing user's makeup routine information, common makeup looks, cosmetics for skin type and ethnicity, and user look preferences; a machine learning system for analyzing images of faces; and the mobile device comprising a user interface for interacting with the digital makeup artist, wherein the digital makeup artist determines the type of makeup look, indoor or outdoor look, skin condition, facial problem areas, and favorite facial features. perform an interactive dialog with a user to capture a user's request including one or more of, wherein the computer operation circuitry inputs the user's face image, and the user's face through the machine learning system; analyzing the image to identify a facial portion, analyzing the facial image to determine a facial characteristic including one or more of skin tone, eye color, hair color, lip color, and skin texture; , Synchronized with the interaction with the digital makeup artist and configured to generate image frames to be displayed on the display device, wherein the image frames are the analyzed face image of the user, the user's image obtained through the interaction with the user It is created based on one or more of requests, stored makeup routine information, typical makeup looks, cosmetics for skin type and race, and user look preferences.

(2) In feature (1), the computer operation circuitry is configured to reproduce a video by displaying the generated image frames at a predetermined playback speed, the video including chapters, each chapter comprising: step of a makeup routine, and input from the user includes video control commands to control playback of the video.

(3) The digital makeup artist system according to feature (2), wherein the video control command includes one of: slow playback speed, pause playback, restart from the start of a step, skip to a next step in a video.

(4) In feature (2) or (3), when the input from the user is a slow playback speed video control command, the computer operation circuitry reduces the playback speed by a predetermined percentage of the playback speed, and the command and storing in the database an indication that the current chapter being played is complex when entered.

(5) In feature (2) or (3), the input from the user is a restart video control command from the start of a step, and the computer operation circuitry when the command is input is associated with the start of the current chapter being reproduced. A digital makeup artist system configured to read a viewpoint and play a video starting from a frame at the viewpoint.

(6) The method of feature (2) or (3), wherein the input from the user is a restart video control command from the beginning of the step, further comprising a command for adjusting the generation of image frames, wherein the computer operating circuit comprises: A digital makeup artist system configured to read a point in time associated with the beginning of a current chapter being played back and to create adjusted image frames for playing video starting from a frame at the point in time.

(7) In feature (6), the adjustment to the generation of the image frame includes a change to a feature of a color of makeup for a face part in the face image, and the feature of the color is coverage, shade (shade), and finish (finish), digital makeup artist system.

(8) The digital makeup artist system of feature (6), wherein the user input includes a type of look, and wherein adjustments to the creation of the image frames take into account user look preferences associated with the type of look.

(9) The digital makeup artist system according to feature (6), wherein a makeup look created based on the adjustment to the image frame generation is stored in the database as a preferred makeup look.

(10) The method of any one of features (1) to (9), wherein the computer operation circuitry analyzes the user's face image through the machine learning system and selects a face shape, a lip shape, an eyelid shape, and a hairstyle. and further configured to determine one or more of the face image of the user to determine one or more of skin tone, eye color, hair color, and skin texture by analyzing the user's face image, wherein the image frames include the face shape, the lip shape, the The digital makeup artist system, which is created based on stored makeup routines related to eyelid shape, the hair style, the skin tone, the eye color, the hair color, and skin texture.

(11) The method of any one of features (1) to (10), wherein the interactive conversation with the user performed by the digital makeup artist includes a voice input, and the mobile device responds to the voice input with a natural language processing, wherein the digital makeup artist outputs a voice response.

(12) The digital makeup artist system according to feature (11), wherein the interactive conversation with the user includes the digital makeup artist speaking a question to obtain information about a condition of the user's skin.

(13) The digital makeup artist system according to feature (11), wherein the interactive conversation with the user includes the digital makeup artist speaking a question to obtain information about problem areas on the face of the user.

(14) The method of feature (12), wherein the interactive interaction with the user comprises the digital makeup artist receiving a voice input indicating that the user's skin is dry, and wherein the image frames are configured to indicate that the user's skin is dry. A digital makeup artist system that is created based on stored makeup routines associated with an indication that is dry.

(15) The method according to any one of features (1) to (14), wherein the makeup routine information and cosmetics stored in the database system are for skin care, and the image frames are stored in the makeup routine information and cosmetics for skin care. A digital makeup artist system created on the basis of

(16) In any one of features (1) to (15), the makeup routine information and cosmetics stored in the database system are for skin care makeup, and the image frames are the makeup routine for skin care makeup. A digital makeup artist system, created based on information and cosmetics.

(17) The method of feature (16), wherein the skin care type makeup includes cosmetics with anti-aging properties, and the image frames are stored in association with the application of the cosmetics with anti-aging properties. A digital makeup artist system, created based on routines.

(18) Analysis of the user according to any one of features (1) to (17), wherein the image frames include the skin tone to provide makeup routines using a shade optimal for the skin tone. A digital makeup artist system, which is created based on the image of the face.

(19) The digital makeup artist system according to any one of features (1) to (18), further comprising a skin undertone machine learning model for selecting a shade of cosmetic product suitable for a specific skin undertone of the face image.

(20) The digital makeup artist system according to feature (19), wherein the cosmetic is a foundation, and the skin undertone machine learning model is for selecting a foundation suitable for a specific skin undertone of the facial image.

(21) Digital makeup artist system. A digital makeup artist may include a mobile device having a display device, computer operating circuitry and memory; a database system that stores user's makeup routine information, common makeup looks, cosmetics for skin type and race, and user look preferences; a machine learning system for analyzing images of faces; and the mobile device includes a user interface for interacting with a digital makeup artist, wherein the digital makeup artist requests a makeup consultation, a type of makeup look, an indoor or outdoor look, a skin condition, a facial problem area, a favorite Conducting an interactive conversation with the user to provide advice including obtaining initial information including one or more of the facial features. The computer manipulation circuit unit inputs the user's face image, analyzes the user's face image through the machine learning system to identify a face part, and analyzes the face image to determine skin tone, eye color, lip color, determine facial features, including one or more of hair color, and skin texture, and generate image frames to be displayed on the display device synchronized with interaction with the digital makeup artist to provide advice; The image frames include one or more of the analyzed face image of the user, initial information obtained through interaction with the user, stored makeup routine information, a normal makeup look, cosmetics for skin type and race, and user look preferences. It is generated in synchronization with the interaction based on.

(22) The method of feature (21), wherein the computer-operated circuitry is configured to conduct the makeup consultation through the interactive dialogue between the user and the digital makeup artist, and the computer-operated circuitry is tailored to a favorite facial feature. A digital makeup artist system that performs interactions that create custom recommendations for makeup routines.

(23) The method of feature (21) or (22), wherein the computer operating circuitry is configured to conduct the makeup consultation through the interactive dialogue between the user and the digital makeup artist, and the interaction is configured to perform the digital makeup consultation. an artist prompting the user to enter at least one problem facial area of interest, wherein the computer-operated circuitry interacts with the user to generate a custom recommendation for a makeup routine tailored to the problem facial area of interest; , a digital makeup artist system.

(24) The method of any of features (21) to (23), wherein the computer operating circuitry is configured to perform the interactive conversation to generate a custom recommendation for the makeup routine, and to provide instructions for adjusting the recommended makeup routine. configured to receive additional input comprising a request, and to perform adjustments based on the requested adjustments of the recommended makeup routine, the saved makeup looks, and the user look preferences to create a refined makeup routine; Digital makeup artist system.

(25) The method of feature (24), wherein the computer operating circuitry is configured to store the adjustment along with the refined makeup routine in the database as a user look preference included in the user look preferences. , digital makeup artist system.

(26) The digital makeup artist system of feature (24), wherein the computer operating circuitry is configured to output recommended makeup and skin care products for the refined makeup routine.

(27) The digital makeup artist system of feature (24), wherein the request to adjust the recommended makeup routine comprises a request to change makeup features of the full face image.

(28) The method of feature (27), wherein the request to change the makeup feature comprises changing one or more of coverage, shade, and finish of each makeup color in the facial image. , digital makeup artist system.

(29) The method of feature (24), wherein the request to adjust the recommended makeup routine comprises a request to change a makeup feature of a facial portion in the facial image, and wherein the makeup feature change request includes a request to change a makeup feature in the facial image. A digital makeup artist system comprising altering one or more of coverage, shade and finish of makeup color on a part of the face.

(30) The method of feature (22), wherein the interactive interaction with the user comprises the digital makeup artist receiving an audio input indicating that the user's skin is dry, and the image frames are configured to indicate that the user's skin is dry. A digital makeup artist system, created based on stored makeup routines associated with indications of being dry.

(31) The method according to any one of features (21) to (30), wherein the makeup routine information and cosmetics stored in the database system are for skin care, and the image frame is based on the makeup routine information and cosmetics for skin care. created by the digital makeup artist system.

(32) The method according to any one of features (21) to (31), wherein the makeup routine information and cosmetics stored in the database system are for skin care makeup, and the image frames are the makeup cosmetics for skin care makeup. A digital makeup artist system, created based on information and cosmetics.

(33) The method of feature (32), wherein the skin care makeup includes cosmetics with anti-aging properties, and wherein the image frames are created based on stored makeup routines relating to application of cosmetics with anti-aging properties. , digital makeup artist system.

(34) The computer-controlled circuitry of any of features (21) to (33), wherein the computer-operated circuitry is operable to generate a custom recommendation for a makeup routine tailored to the user's skin tone, and an image of the makeup routine. frames are created based on the user's analyzed facial image to create a custom recommendation that uses a shade of cosmetic product that is optimal for a skin tone.

(35) The digital makeup artist system of feature (34), further comprising a skin undertone machine learning model for selecting a shade of the cosmetic product suitable for a particular skin undertone of the facial image.

(36) The digital makeup artist system according to feature (35), wherein the cosmetic product is foundation and the skin undertone machine learning model is for selecting a shade of foundation appropriate for a particular skin undertone in the facial image.

(37) The computer-operated circuitry of any of features (21) to (37), wherein the computer-operated circuitry performs interactions to create a custom recommendation for a makeup routine tailored to the lip shape and lip color of the user, and The image frames of the routine are generated based on the user's face image analyzed to create a custom recommendation that uses a cosmetic shade and/or finish for the user's face that is optimal for the user's lip shape and lip color. , digital makeup artist system.

(38) according to any of features (21) to (37), wherein the computer operating circuitry performs the interaction to create a custom recommendation for a makeup routine comprising obtaining the user's skin type and ethnicity; and , and image frames of the make-up routine are created based on the user's skin type and race and user look preferences to create the custom recommendation using cosmetics stored in the database specified according to the user's skin type and race. Make-up artist system.

(39) In feature (38), the makeup routine information and cosmetics stored in the database system are for skin care type makeup for skin type and race, and the image frames are for skin type and race, A digital makeup artist system that is created based on makeup routine information and cosmetics for the skin care type makeup.

(40) The digital makeup artist system according to feature (38), further comprising an ethnic machine learning model for selecting a shade of the cosmetic product suitable for a specific ethnicity of the facial image of the user.

Claims (40)

In the digital makeup artist system,
a mobile device having a display device, computation circuitry and memory;
a database system for storing a user's makeup routine information, a common makeup look, cosmetics for skin type and ethnicity, and user look preferences;
a machine learning system for analyzing images of faces; and
The mobile device includes a user interface for interacting with a digital makeup artist, wherein the digital makeup artist selects one of a type of makeup look, an indoor or outdoor look, a skin condition, a facial problem area, or a favorite facial feature. Conducting an interactive dialog with the user to capture the user's request including anomalies;
The computer operating circuit part,
Enter the user's face image;
Analyzing the user's face image through the machine learning system to identify a face part;
analyzing the facial image to determine a facial characteristic including one or more of skin tone, eye color, hair color, lip color, and skin texture; and
configured to generate image frames to be displayed on the display device in synchronization with interaction with the digital makeup artist;
The image frames include the analyzed face image of the user, the user's request obtained through interaction with the user, stored makeup routine information, a normal makeup look, cosmetics for skin type and race, and user look preferences. A digital makeup artist system, created based on one or more. According to claim 1,
the computer operating circuitry is configured to reproduce the video by displaying the generated image frames at a predetermined playback speed;
The video includes chapters, each chapter being a step of a makeup routine, and
wherein the input from the user includes video control commands for controlling playback of the video. According to claim 2,
wherein the video control command comprises one of: slow playback speed, pause playback, restart from the start of a step, skip to the next step in a video. According to claim 3,
If the input from the user is a slow playback speed video control command, the computer operation circuitry reduces the playback speed by a percentage of a predetermined playback speed, and displays an indication that the current chapter being played is complex when the command is input. A digital makeup artist system that stores in a database. According to claim 3,
The input from the user is a restart video control command from the start of a step, and the computer operating circuitry reads a point in time associated with the start of the current chapter being played when the command is input, and the video starting from the frame at the point in time. A digital makeup artist system, configured to play. According to claim 3,
wherein the input from the user is a restart video control command from the beginning of a step, further including a command for adjusting the creation of image frames, wherein the computer operating circuitry reads a point in time associated with the start of the current chapter being played and the A digital makeup artist system configured to generate adjusted image frames for replaying a video starting from a frame at a viewpoint. 7. The method of claim 6, wherein the adjustment to the generation of the image frame includes a change to a feature of a color of makeup for a facial part in the facial image,
The feature of the color is one or more of coverage, shade, and finish, the digital makeup artist system. 7. The method of claim 6, wherein the user input includes a type of look, and
and the adjustment to the creation of the image frames takes into account user look preferences associated with the type of look. 7. The digital makeup artist system according to claim 6, wherein makeup looks created based on adjustments to the image frame creation are stored in the database as preferred makeup looks. The method of claim 1 , wherein the computer manipulation circuit unit determines one or more of a face shape, a lip shape, an eyelid shape, and a hairstyle by analyzing the user's face image through the machine learning system, and determines the user's face image. to determine one or more of skin tone, eye color, hair color, and skin texture; and
The image frames are created based on makeup routines stored in relation to the face shape, the lip shape, the eyelid shape, the hair style, the skin tone, the eye color, the hair color, and the skin texture. Make-up artist system. 2. The method of claim 1, wherein the interactive conversation with the user performed by the digital makeup artist includes voice input, and
The mobile device performs natural language processing on the voice input, and the digital makeup artist outputs a voice response. 12. The digital makeup artist system of claim 11, wherein the interactive conversation with the user comprises the digital makeup artist speaking a question to obtain information about the condition of the user's skin. 12. The digital makeup artist system of claim 11, wherein the interactive conversation with the user includes the digital makeup artist speaking a question to obtain information about problem areas of the user's face. 13. The method of claim 12, wherein the interactive conversation with the user comprises the digital makeup artist receiving a voice input indicating that the user's skin is dry, and
wherein the image frames are generated based on makeup routines stored in association with an indication that a user's skin is dry. The method of claim 1, wherein the makeup routine information and cosmetics stored in the database system are for skin care, and
The image frames are generated based on the makeup routine information and cosmetics for skin care, digital makeup artist system. The method of claim 1, wherein the makeup routine information and cosmetics stored in the database system are for skin care type makeup, and
The image frames are generated based on the makeup routine information and cosmetics for skin care type makeup. The method of claim 16, wherein the skin care type makeup includes cosmetics having anti-aging properties, and
wherein the image frames are created based on stored makeup routines related to application of the cosmetic product having anti-aging properties. 2. The digital makeup according to claim 1, wherein the image frames are created based on the analyzed user's facial image including the skin tone to provide makeup routines using a shade optimal for the skin tone. artist system. The digital makeup artist system of claim 1, further comprising a skin undertone machine learning model for selecting a cosmetic shade suitable for a particular skin undertone of the facial image. 20. The method of claim 19, wherein the cosmetic is a foundation, and
wherein the skin undertone machine learning model is for selecting a foundation suitable for a specific skin undertone of the face image. In the digital makeup artist system,
a mobile device having a display device, computer operating circuitry and memory;
a database system that stores user's makeup routine information, common makeup looks, cosmetics for skin type and race, and user look preferences;
a machine learning system for analyzing images of faces; and
The mobile device includes a user interface for interacting with a digital makeup artist, wherein the digital makeup artist requests a makeup consultation, the type of makeup look, indoor or outdoor look, skin condition, facial problem area, favorite face conducting an interactive conversation with the user to provide advice comprising obtaining initial information including one or more of the features;
The computer operating circuit part,
Enter the user's face image;
Analyzing the user's face image through the machine learning system to identify a face part;
analyzing the facial image to determine facial features including one or more of skin tone, eye color, lip color, hair color, and skin texture;
configured to generate image frames to be displayed on the display device in synchronization with interaction with the digital makeup artist to provide advice;
The image frames include one of the analyzed face image of the user, initial information obtained through interaction with the user, stored makeup routine information, a normal makeup look, cosmetics for skin type and race, and user look preferences. A digital makeup artist system, which is created based on the above and synchronized with the interaction. 22. The method of claim 21, wherein the computer operating circuitry is configured to perform the makeup consultation through the interactive conversation between the user and the digital makeup artist, and
wherein the computer-operated circuitry performs interactions to generate custom recommendations for makeup routines tailored to favorite facial features. 22. The method of claim 21, wherein the computer operating circuitry is configured to perform the makeup consultation through the interactive conversation between the user and the digital makeup artist,
the interaction comprising the digital makeup artist prompting the user to enter at least one problem facial region of interest; and
wherein the computer-operated circuitry performs interactions to generate custom recommendations for makeup routines tailored to the problem facial regions of interest. 22. The method of claim 21, wherein the computer operation circuitry comprises:
conducting the interactive conversation to create a custom recommendation for the makeup routine;
receive additional input comprising a request for adjustment of the recommended makeup routine; and
and perform adjustments based on the requested adjustments of the recommended makeup routine, the stored makeup looks, and the user look preferences to create a refined makeup routine. 25. The digital make-up artist of claim 24, wherein the computer-operated circuitry is configured to store the adjustment along with the refined makeup routine in the database as a user look preference included in the user look preferences. system. 25. The digital makeup artist system of claim 24, wherein the computer operating circuitry is configured to output recommended makeup and skin care products for the refined makeup routine. 25. The digital makeup artist system of claim 24, wherein the request to adjust the recommended makeup routine includes a request to change makeup features of the full face image. 28. The digital makeup artist of claim 27, wherein the request to change the makeup feature comprises changing one or more of coverage, shade, and finish of each makeup color in the face image. system. 25. The method of claim 24, wherein the request for adjustment of the recommended makeup routine includes a request for changing a makeup feature of a facial portion in the facial image, and
The makeup feature change request includes a change of one or more of a makeup color coverage, shade, and finish of a face part in the face image. 23. The method of claim 22, wherein the interactive interaction with the user comprises the digital makeup artist receiving a voice input indicating that the user's skin is dry, and
wherein the image frames are generated based on makeup routines stored in association with an indication that a user's skin is dry. The method of claim 21, wherein the makeup routine information and cosmetics stored in the database system are for skin care, and
The image frames are generated based on the makeup routine information and cosmetics for skin care, digital makeup artist system. The method of claim 21, wherein the makeup routine information and cosmetics stored in the database system are for skin care type makeup, and
The image frames are generated based on the makeup cosmetic information and cosmetics for skin care type makeup. 33. The method of claim 32, wherein the skin care type makeup includes cosmetics having anti-aging properties, and
wherein the image frames are created based on stored makeup routines relating to application of cosmetic products having anti-aging properties. 22. The computer-controlled circuitry of claim 21, wherein the computer-operated circuitry interacts to generate custom recommendations for a makeup routine tailored to the user's skin tone; and
wherein the image frames of the makeup routine are created based on the user's facial image analyzed to create a custom recommendation that uses a shade of cosmetic product that is optimal for the skin tone. 35. The digital makeup artist system of claim 34, further comprising a skin undertone machine learning model for selecting a shade of the cosmetic product suitable for a particular skin undertone of the facial image. 36. The method of claim 35, wherein the cosmetic is a foundation,
wherein the skin undertone machine learning model is for selecting a shade of foundation appropriate for a specific skin undertone of the face image. 22. The computer-aided circuitry of claim 21, wherein the computer-operated circuitry interacts to create custom recommendations for makeup routines tailored to the user's lip shape and lip color; and
Image frames of the make-up routine are based on the analyzed user's face image to create a custom recommendation that uses a cosmetic shade and/or finish for the user's face that is optimal for the user's lip shape and lip color. Created, digital makeup artist system. 22. The computer-aided circuitry of claim 21, wherein the computer-controlled circuitry performs the interaction of generating a custom recommendation for a makeup routine comprising obtaining the user's skin type and ethnicity, and
wherein image frames of the make-up routine are created based on the user's skin type and race and user look preferences to create the custom recommendation using cosmetics stored in the database specified according to the user's skin type and race. system. 39. The method of claim 38, wherein the makeup routine information and cosmetics stored in the database system are for skin care type makeup for skin type and race, and
The image frames are generated based on makeup routine information and cosmetics for the skin care type makeup for the skin type and race. 39. The digital makeup artist system of claim 38, further comprising an ethnic machine learning model for selecting a shade of the cosmetic product suitable for a specific ethnicity of the facial image of the user.

Images