JP2024083217A - Information processing device, information processing method, and program - Google Patents

Abstract

To allow intuitive selection of a person that is to be a processing object.SOLUTION: An information processing device performs the following: recognizing characteristics of captured images each including an attribute of a person who appears as a subject; switching displays of selection buttons which are used for selecting the person to be a processing object according to the characteristics of a captured image being selected as a processing object, the selection button being included in a processing screen which is used for selecting a processing content for a captured image; and performing processing for part of the person selected using the selection button in the captured image being selected as the processing object. The present technique can be applied to a computer installed in a photographing studio.SELECTED DRAWING: Figure 10

Description

Application for exception to loss of novelty has been filed

This technology relates to an information processing device, an information processing method, and a program, and in particular to an information processing device, an information processing method, and a program that enable intuitive selection of a person to be processed.

In photo taking devices such as passport photo booths and print sticker machines installed in game centers, users can adjust the appearance of the image after it has been taken. Users can adjust the appearance of each image that they receive printed on sticker paper or the like to their liking.

Patent Document 1 describes a technology that, when a couple course is selected as the photography course, changes the editing tools displayed on the editing screen depending on the positions of the man and woman when taking the photo. For example, in the area assigned to a male user, buttons used to select relatively simple editing tools are displayed.

JP 2014-074914 A

When multiple people appear in the image you have selected for adjustment, you may want to apply different image processing to one person than to the others.

This technology was developed in light of these circumstances, and allows users to intuitively select the person they want to edit.

An information processing device according to one aspect of the present technology includes a recognition unit that recognizes the characteristics of each captured image including the attributes of a person appearing as a subject, a display control unit that configures a processing screen that is used to select the content of processing for the captured image and that switches the display of a selection button used to select the person to be processed in accordance with the characteristics of the captured image that is selected as the processing target, and an image processing unit that applies processing to the part of the person selected by the user using the selection button in the captured image that is selected as the processing target.

In one aspect of the present technology, the characteristics of each captured image, including the attributes of the person appearing as the subject, are recognized, and the display of a selection button used to select the person to be processed, which constitutes a processing screen used to select the content of processing for the captured image, is switched according to the characteristics of the captured image selected as the processing target. Also, processing is applied to the part of the person selected by the user using the selection button in the captured image selected as the processing target.

This technology allows you to intuitively select the person you want to edit.

FIG. 1 is a diagram showing a workflow in a photography studio. FIG. 13 is a diagram showing an example of providing a captured image. FIG. 13 is a diagram showing an example of a retouch screen. FIG. 13 is an enlarged view showing a display of a retouch selection area. FIG. 13 is a diagram illustrating an example of application of retouching parameters. FIG. 13 is a diagram showing an example of a selection screen for an operation mode. FIG. FIG. 11 is another diagram showing the work flow. FIG. 13 is an enlarged view showing a display of a person selection button. FIG. 13 is a diagram illustrating an example in which a person selection button is operated. FIG. 13 is an enlarged view showing a display of a person selection button. FIG. 13 is an enlarged view showing a display of a person selection button. FIG. 13 is a diagram illustrating an example in which a person selection button is operated. FIG. 13 is an enlarged view showing a display of a person selection button. FIG. 13 is an enlarged view showing a display of a person selection button. FIG. 2 is a block diagram showing an example of the configuration of a retouching PC. FIG. 2 is a block diagram showing an example of the functional configuration of a retouching PC. FIG. 11 is a diagram illustrating an example of a management file. FIG. 1 is a diagram showing an example of a captured image to be retouched. FIG. 13 is a diagram illustrating an example of image processing for style up. FIG. 13 is a diagram showing examples of captured images before and after a style up. FIG. 13 is a diagram illustrating an example of image processing for style up. FIG. 13 is a diagram showing an example of a stretching direction. FIG. 13 is a diagram showing a display example of a retouch screen. FIG. 11 is a diagram showing another display example of the retouch screen. FIG. 11 is a diagram showing another display example of the retouch screen. FIG. 13 is a diagram showing a display example of a frame image. FIG. 13 is a diagram showing an example of selection of whether or not to retouch a subject; FIG. 13 is a diagram showing an example of presentation of feature recognition results. FIG. 11 is a diagram showing another example of presentation of feature recognition results. FIG. 13 is a diagram showing an example of an image selection screen.

<<Work flow in the photography studio>>
FIG. 1 is a diagram showing the flow of work in a photography studio.

This technology is used, for example, in photography studios that take commemorative photos for weddings. This technology can also be applied to taking commemorative photos for various events, such as entrance/graduation photos, coming-of-age ceremonies, and Shichi-Go-San (celebration of the 365-3 festival).

As shown on the left side of Figure 1, photography in a photography studio is performed by a photographer. In the example of Figure 1, the subjects of the photography are a bride and groom standing in front of a background screen. The groom is wearing a tuxedo, and the bride is wearing a wedding dress. Around the bride and groom, equipment such as a strobe device that emits light in accordance with the photographer's photography is placed. Photography of the bride and groom may also be performed outside the photography studio.

The photographer takes many still images (photographs), varying the composition, such as 50 or 100. The composition of the still images is determined by the poses of the bride and groom, the relative positions of the bride and groom and the photographer, and camera parameters such as focal length. The still images taken by the photographer are transferred to and saved on a PC (not shown) connected to the camera.

After the photo shoot is completed, the bride and groom move to the retouching space prepared in the photography studio and use retouching PC 1 to check the images taken by the photographer, as shown on the right side of Figure 1.

The bride and groom, as users of the retouching PC 1, select the number of images they ultimately want printed, and adjust the appearance of each image by retouching them themselves. Retouching is performed on a portion of the images selected from those taken by the photographer. For example, a predetermined number of images, such as five or ten, are selected as images to be retouched, depending on the plan the user has signed up for. Retouching (processing) includes various types of image processing to adjust the appearance of the images.

In the retouching space, one or more retouching PCs 1 are prepared. The captured images stored in the storage PC are transferred to the retouching PC 1 and displayed on the display 11 of the retouching PC 1. A retouching application 1A, which is an application used to retouch images taken by a photographer, is installed on the retouching PC 1.

In this way, in a photography studio where a retouching PC 1 to which the present technology is applied is installed, the users (the bride and groom) themselves retouch the images taken by the photographer. Using the functions of the retouching application 1A, the users can retouch the images to their liking. The retouching application 1A may be installed on a device other than a PC, such as a tablet device, and used for retouching.

Figure 2 shows an example of providing captured images.

The captured images retouched by retouching app 1A are printed by the photography studio's management company and later given to the user in the form of, for example, an album, as shown by arrow #1.

The retouched photographed images are uploaded from the retouching PC 1 to the image management server 2, as shown by arrow #2, and are managed by the image management server 2. The image management server 2 is, for example, a server on the Internet. The image management server 2 is managed by the management company of the photography studio, or by a business entity different from the management company of the photography studio.

By operating their own mobile terminal 3 such as a smartphone and operating the image management app 3A, users can view and download images that have been taken of them as the subject, as shown by arrow #3. The image management app 3A is an application for viewing, etc., the captured images managed in the image management server 2. The image management app 3A is provided, for example, by the business entity that manages the image management server 2.

The images taken in the photography studio will be provided in the form of an album and, where appropriate, in the form of data. It may be possible to provide the images in the form of data for a fee separate from the fee for the album.

The provision of photographed images in the form of data is carried out so that not only photographed images that have been retouched by the user, but also photographed images that have not been selected by the user as subjects for retouching are provided. Photographed images that have not been selected by the user as subjects for retouching are automatically retouched by the retouching application 1A without any user operation, as described below.

The user can receive in the form of data even captured images that were not selected for retouching. Instead of using the image management app 3A, the user can use a browser to access a specific site and receive captured images managed in the image management server 2. For example, by registering as a member of an image providing service provided by the administrator of the image management server 2, the user can receive captured images managed in the image management server 2 using the image management app 3A or a browser.

Instead of the mobile terminal 3, it may be possible to receive captured images using other terminals such as a tablet terminal or a PC.

<<About the retouching app>>
<Screen display>
Retouch Screen FIG. 3 is a diagram showing an example of a retouch screen.

A display area 31 is formed in the upper center of the retouch screen, which is a screen used for retouching photographed images. The display area 31 is an area in which one photographed image currently being retouched is enlarged and displayed. A thumbnail image of the photographed image selected by the user is displayed in a thumbnail area 32 formed on the left side of the display area 31. In the example of FIG. 3, thumbnail images of five photographed images P1 to P5 are displayed. The photographed image P1, which is selected using the thumbnail image, is enlarged and displayed in the display area 31.

A horizontally long retouching selection area 33 is formed below the retouching screen. The buttons displayed in the retouching selection area 33 are used to select the retouching content. Each time a retouching content is selected using a button displayed in the retouching selection area 33, the selected content is reflected in the display in the display area 31. While viewing the display in the display area 31, the user can use the display in the retouching selection area 33 to select the retouching content to be applied to the captured image currently being retouched.

Figure 4 shows an enlarged view of the retouch selection area 33.

The retouching selection area 33 is made up of area 33-1, which is a display area for information used to adjust the entire face and body, area 33-2, which is a display area for information used to adjust the eyes and makeup, and area 33-3, which is a display area for information used to adjust brightness.

In area 33-1 formed to the left of retouch selection area 33, adjustment button 51 used to adjust the appearance of a smaller face, adjustment button 52 used to adjust the shape of the face, adjustment button 53 used to adjust the skin type, adjustment button 54 used to adjust the length of the chin, and adjustment button 55 used to adjust the style (style up).

Five levels (degree of adjustment) are available for the small face effect. A small face effect of "0" indicates that the size of the face is not changed and remains at its original size. A small face effect of "1" indicates that the size of the face is reduced by one level based on the parameter corresponding to "1." By selecting one of the levels "1" to "4," the user can adjust the size of their own face in the captured image displayed in the display area 31.

Face shapes available are "round," "egg," and "slim." By selecting one of these shapes, the user can adjust the shape of their face that appears in the captured image displayed in the display area 31. By selecting "no retouching," it is also possible to leave the shape of the face unchanged.

Five levels of skin texture are available. Skin texture "0" indicates that the skin texture is not changed and remains the original skin texture. Skin texture "1" indicates that the skin texture level is increased by one level based on the parameter corresponding to "1." By selecting one of the levels "1" to "4," the user can adjust the degree of smoothness of their own skin texture as it appears in the captured image displayed in the display area 31.

Three levels of chin length are available. A chin length of "0" indicates no processing, i.e. the chin length is left unchanged at its original length. A chin length of "short" indicates that processing is performed based on the parameters corresponding to "short", shortening the chin length by one level. By selecting one of the levels, the user can adjust the length of their chin that appears in the captured image displayed in the display area 31.

Three levels of style improvement are available. Style improvement "0" indicates no processing, i.e., the style is not changed and remains the original style. Furthermore, style improvement "weak" indicates that processing such as making the legs longer is performed based on the parameters corresponding to "weak", improving the overall balance by one level. By selecting one of the levels, the user can adjust their own style as it appears in the captured image displayed in the display area 31.

In area 33-2 formed in the center of retouch selection area 33, adjustment button 56 used to adjust eye size, adjustment button 57 used to adjust eye shape, adjustment button 58 used to adjust makeup intensity, adjustment button 59 used to adjust nose bridge, and adjustment button 60 used to adjust facial hair removal are displayed.

Five levels of eye size are available. Eye size "0" indicates no processing, i.e., the eye size is left unchanged at its original size. Eye size "1" indicates that the eye size is increased by one level based on the parameter corresponding to "1." By selecting one of the levels "1" through "4," the user can adjust the size of their own eyes as they appear in the captured image displayed in the display area 31.

Eye shapes available are "round eyes," "droopy eyes," and "cat eyes." By selecting one of these shapes, the user can adjust the shape of their eyes that will appear in the captured image displayed in the display area 31. By selecting "no processing," it is also possible to leave the shape of the eyes unchanged.

Five levels of makeup intensity are available. A makeup intensity of "0" indicates that the makeup intensity is not changed and remains at the original intensity. A makeup intensity of "1" indicates that the makeup intensity is increased by one level based on the parameter corresponding to "1." By selecting one of the levels "1" through "4," the user can adjust the intensity of their own makeup in the captured image displayed in the display area 31.

It is possible to select ON/OFF for the nose bridge. By selecting ON or OFF, the user can choose whether to emphasize the highlights of the bridge of their nose that appear in the captured image displayed in the display area 31, or to leave it in its original state.

Three levels of beard erasure are available. Beard erasure "0" means that the beard is not changed and remains the original beard. Furthermore, beard erasure "weak" means that processing is performed based on the parameters corresponding to "weak" to make the beard one level thinner. By selecting one of the levels, the user can adjust the darkness of their own beard in the captured image displayed in the display area 31. The beard erasure function is enabled when the person being retouched is the groom.

In area 33-3 formed to the right of retouch selection area 33, adjustment button 61 used to adjust the brightness of the skin, adjustment button 62 used to adjust the overall brightness, and adjustment button 63 used to adjust the brightness of only the background are displayed.

Three levels of skin brightness are available. Skin brightness "0" indicates that the skin brightness is not changed and remains at the original brightness. Skin brightness "1" indicates that the skin brightness level is increased by one level based on the parameter corresponding to "1." By selecting one of the levels, the user can adjust the brightness of their own skin in the captured image displayed in the display area 31.

Three levels of overall brightness are available. An overall brightness of "0" indicates that the overall brightness is not changed and remains at the original brightness. An overall brightness of "1" indicates that the overall brightness level is increased by one level based on the parameter corresponding to "1." By selecting one of the levels, the user can adjust the overall brightness of the captured image displayed in the display area 31.

Three levels of background brightness are available. A background brightness of "0" indicates that the background brightness is not changed and remains at its original brightness. A background brightness of "1" indicates that the background brightness level is increased by one level based on the parameter corresponding to "1." By selecting one of the levels, the user can adjust the background brightness shown in the captured image displayed in the display area 31.

Returning to the explanation of FIG. 3, a batch retouch button 34 used for batch retouching is displayed on the left side of the display area 31. Batch retouching is a function for selecting a combination of various retouching contents as described above all at once.

In the example of FIG. 3, the user can select one of three levels of batch retouching, each of which has a different combination of retouching content levels. By using the batch retouching function, the user can select the retouching content with a single operation, without having to select each retouching content using the display in the retouching selection area 33.

A Before-processing button 35 is displayed on the right side of the display area 31. The Before-processing button 35 is a button that is operated when switching the display of the display area 31 to the state before retouching. While the Before-processing button 35 is pressed, the captured image in the state before retouching is displayed in the display area 31. By using the Before-processing button 35, the user can compare the state after retouching with the state before retouching and check whether the degree of retouching is too strong, for example.

Below the Before button 35, a person selection button 36 is displayed, which is used to select a person to be retouched. By using the person selection button 36, the user can select to retouch only the person appearing on the left side of the captured image, only the person appearing on the right side, or both people. Face recognition and other processes are performed on the captured image, and the person to be the target of image processing for retouching is selected in response to the user's operation.

Using these various functions, the user can select the retouching content to be applied to the captured image displayed in the display area 31. In this way, retouching that can be performed using the retouching application 1A includes at least adjustments to the face and body as a whole, adjustments to the eyes and makeup, and adjustments to brightness.

The captured image to be retouched is switched using the thumbnail image, and retouching is performed sequentially on each of the captured images P1 to P5. When retouching of each of the captured images P1 to P5 is completed, retouching using the retouch screen is completed.

<All items reflected>
FIG. 5 is a diagram showing an example of application of retouching parameters.

As shown in FIG. 5, the retouching contents applied to each of the captured images P1 to P5 by the user's operation are reflected in all unselected images. An unselected image is a captured image that was not selected as a subject for retouching. A captured image selected as a subject for retouching on the image selection screen becomes a selected image. For example, if five captured images are selected as selected images from among 100 captured images, the remaining 95 captured images become unselected images.

In other words, the number of captured images selected as retouching targets according to the plan (selected images) are retouched according to the content selected directly by the user using the retouching screen, whereas unselected images that were not selected as retouching targets are retouched so that the content selected for the selected image, which is a different image, is automatically reflected. For example, the retouching content of the selected image is reflected on all unselected images.

In this way, the retouching application 1A is provided with a full reflection function that reflects the retouching content of a selected image to unselected images after the user has retouched the selected image.

When photographing an event such as a wedding, many shots are usually taken, often hundreds of shots. It is a heavy burden for the user to manually retouch all of the images taken by the photographer. By making it possible for the retouching of a selected image to be reflected in all unselected images, the user can receive all of the images taken with the image adjusted to the user's liking.

Which of the retouched images P1 to P5 should have its retouching contents reflected in which unselected image is selected based on the characteristics of the selected image and the unselected image, such as whether the same person is in the image and whether the faces are of similar size. For example, the retouching contents of a selected image that contains the same person's face of a similar size as the person in the unselected image is reflected in a certain unselected image.

In this case, the number of people appearing in each captured image (selected image and unselected image), the size of their faces, and other characteristics are recognized. The retouching details of the selected image that has the same characteristics are reflected in each unselected image.

The retouching application 1A is also equipped with a function for recognizing the characteristics of such captured images. In addition to size, other facial conditions such as the direction of the face may be recognized. Other types of characteristics such as the person's gender, age, and the location where the image was taken (outdoors/indoors) may also be recognized.

<Customer service mode and export mode>
The retouching application 1A has two operation modes, a customer service mode and a write mode. The customer service mode is an operation mode that is selected when the user performs retouching using the retouching screen. The write mode is an operation mode that is selected when performing full reflection.

Figure 6 shows an example of an operation mode selection screen.

The selection screen displays a button 81 that is operated to select the customer service mode, and a button 82 that is operated to select the export mode. When asking a user to perform retouching, for example, a staff member at a photography studio selects the customer service mode using a selection screen such as that shown in FIG. 6, and has the user perform retouching. After the user has finished retouching, the staff member at the photography studio selects the export mode and reflects the changes in all images.

Figure 7 shows the workflow for a specific user.

As shown in FIG. 7, a photographer takes a photograph of the user as a subject, and then the user retouches the photograph using the customer service mode of the retouching application 1A. After the user has finished retouching, the staff member uses the export mode to reflect all the photographs.

As mentioned above, several hundred photos are taken in one shooting session, so full reflection places a large processing load on the retouching PC 1 and is a time-consuming process. By providing an export mode and setting it so that full reflection occurs after the export mode is selected, it is possible to prevent a situation in which full reflection starts automatically immediately after retouching is finished, preventing the next user from starting retouching.

That is, as shown in FIG. 8, it is possible to reflect all the retouching for multiple users (multiple pairs of brides and grooms) after all users have finished retouching.

In the example of Figure 8, a photograph of a bride and groom pair by user #1 is taken and then retouched by user #1, and then a photograph of a different bride and groom pair by user #2 is taken and retouched by user #2. After retouching by user #2, a photograph of a different bride and groom pair is taken and retouched by that user. The work of one user is carried out in parallel with the work of another user as appropriate, such that retouching by user #1 and the photographing by user #2 are carried out in parallel.

For example, after all users have finished taking and retouching photos, export mode is used to reflect all photos for each user. In the example of Figure 8, after all photos for user #1 are reflected, all photos for user #2 are reflected.

In this way, by making it possible to execute a full update for multiple users at once, it is possible to improve the efficiency of the overall operation compared to executing a full update every time retouching is performed.

<Display Switching of Person Selection Button 36>
3, the retouching screen is provided with a person selection button 36 used to select a person to be retouched. By using the person selection button 36, even if multiple people appear in the selected image selected as the retouching target, the user can individually select the people to be retouched and adjust the appearance of each person.

The display of the person selection button 36 is switched depending on the characteristics of the selected image selected to be retouched. Depending on the characteristics of the selected image, the display of the person selection button 36 is switched each time the selected image to be retouched is switched.

Example of Switching Display According to the Number of People FIG. 9 is an enlarged view of the display of the person selection button 36. As shown in FIG.

When a captured image P1 showing the bride and groom side by side is being selected as a subject to retouching as shown on the left side of Fig. 9, as shown on the right side of Fig. 9, button 41A is operated to individually select only the person on the left side as a subject to retouching, button 41B is operated to individually select only the person on the right side as a subject to retouching, and button 41C is operated to select both people as subjects to retouching, are displayed as person selection buttons 36. Circular buttons 41A, 41B, and 41C display text explaining the contents of each selection.

The retouching PC 1 is equipped with a function for identifying the number of people appearing in a selected image by recognizing the faces of people appearing in the selected image.

In this state, when button 41A is operated as shown on the right side of FIG. 10A, only the groom on the left side is selected as the subject of retouching, and a frame is displayed around the groom's face. The user (for example, the groom) can adjust the appearance of the groom's face by using the display of retouch selection area 33.

On the other hand, when button 41B is operated as shown on the right side of FIG. 10B, only the bride on the right side is selected as the subject of retouching, and a frame is displayed around the bride's face. The user (for example, the bride) can adjust the appearance of the bride's face by using the display of retouching selection area 33.

When button 41C is operated, the groom on the left and the bride on the right are selected as subjects for retouching, and two frames are displayed around their respective faces.

In this way, when a photographed image containing multiple people is selected as a subject for retouching, the person selection button 36 is displayed in an operable state (on state) for the button used to individually select the left and right people.

Figure 11 is another diagram showing an enlarged view of the person selection button 36.

When photographed image P3, which shows a close-up of the bride, is selected as the subject of retouching as shown on the left side of FIG. 11, person selection button 36 is displayed grayed out and buttons 41A, 41B, and 41C constituting person selection button 36 are inoperable as shown on the right side of FIG. 11. Because the only person appearing in photographed image P3 is the bride, the subject of retouching is fixed to the bride while photographed image P3 is selected as the subject of retouching. A frame surrounding the bride's face may or may not be displayed.

When a photographed image containing one person is selected as the subject of retouching, the person selection button 36 is displayed in a disabled state (off state) for the button used to individually select the left and right persons.

For example, if photographed image P3 is selected as the subject of retouching after photographed image P1, and then photographed image P5 (Figure 3) depicting the bride and groom is selected as the subject of retouching, the state of person selection button 36 will be switched between on, off, and on each time the photographed image to be retouched is switched.

The on/off state of the person selection button 36 can be switched depending on the number of people appearing in the photographed image to be retouched, allowing the user to easily select the people to be retouched.

The person selection button 36 may be hidden when all five selected images are photographs in which one person appears. In this case, the display area for the person selection button 36 may be blank or may display a decorative image.

In this way, if there is at least one selected image that shows multiple people, the person selection button 36 is displayed in an on or off state, and if there is no selected image that shows multiple people, the person selection button 36 is not displayed.

Example of Switching Display According to Gender FIG. 12 is an enlarged view of the display of the person selection button 36. As shown in FIG.

When a photographed image P5 showing a close-up of the bride and groom side by side is selected as a retouching target as shown on the left side of FIG. 12, as shown on the right side of FIG. 12, a button 42A operated when selecting only the male as a retouching target, a button 42B operated when selecting only the female as a retouching target, and a button 42C operated when selecting both of them as retouching targets are displayed as person selection buttons 36. Buttons 42A, 42B, and 42C display characters explaining the respective selections. Buttons 42A to 42C display characters different from the characters displayed on buttons 41A to 41C. The selections may be presented using other information such as symbols instead of characters. Button 42A may display the characters "Groom only" and button 42B may display the characters "Bride only".

In this state, when button 42A is operated as shown on the right side of FIG. 13A, only the groom, who is a man, is selected as the subject of retouching, and a frame is displayed around the groom's face. The user (for example, the groom) can adjust the appearance of the groom's face by using the display of retouch selection area 33.

On the other hand, when button 42B is operated as shown on the right side of FIG. 13B, only the bride, who is a woman, is selected as the subject of retouching, and a frame is displayed around the bride's face. The user (for example, the bride) can adjust the appearance of the bride's face by using the display of retouching selection area 33.

When button 42C is operated, the male groom and female bride are selected as subjects for retouching, and two frames are displayed around their respective faces.

In this way, when a photographed image containing a man and a woman is selected as a subject for retouching, person selection button 36 is displayed in a state in which the button used to select the man and the woman individually is operable (on state).

Figure 14 shows an enlarged view of the person selection button 36.

When photographed image P3, which shows a close-up of the bride, is selected as the subject of retouching as shown on the left side of FIG. 14, person selection button 36 is displayed grayed out and buttons 42A, 42B, and 42C constituting person selection button 36 are inoperable as shown on the right side of FIG. 14. Because the only person appearing in photographed image P3 is the bride, the subject of retouching is fixed to the bride while photographed image P3 is selected as the subject of retouching. A frame surrounding the bride's face may or may not be displayed.

When a photographed image containing one person is selected as the subject of retouching, the person selection button 36 is displayed in a disabled state (off state) for the button used to select a man and a woman separately.

The on/off state of the person selection button 36 can be switched depending on the gender of the person appearing in the photographed image to be retouched, allowing the user to easily select the person to be retouched.

The person selection button 36 may be hidden when all five selected images are photographs in which one person appears. In this case, the display area for the person selection button 36 may be blank or may display a decorative image.

In this way, if there is at least one selected image that shows a man and a woman, the person selection button 36 is displayed in an on or off state, and if there is no selected image that shows a man and a woman, the person selection button 36 can be made not to be displayed.

When two women or two men are in the photographed image selected as the subject of retouching, person selection buttons 36 are displayed, which are button 41A operated to individually select only the person on the left side as the subject of retouching, button 41B operated to individually select only the person on the right side as the subject of retouching, and button 41C operated to select both people as the subject of retouching.

Example of Switching Display According to Age FIG. 15 is an enlarged view of the display of the person selection button 36. As shown in FIG.

When a captured image P11 in which the child and bride are side by side is selected as the subject to retouching as shown on the left side of Fig. 15, as shown on the right side of Fig. 15, button 43A, which is operated when selecting only the adult as the subject to retouching individually, button 43B, which is operated when selecting only the child as the subject to retouching individually, and button 43C, which is operated when selecting both persons as the subject to retouching, are displayed as person selection buttons 36. Buttons 43A, 43B, and 43C display text explaining the respective selections.

When button 43A is operated in this state, only the bride, who is an adult, is selected as the subject of retouching, and a frame is displayed around the bride's face. The user (for example, the bride) can adjust the appearance of the bride's face by using the display of retouching selection area 33.

On the other hand, when button 43B is operated, only the child is selected as the subject of retouching, and a frame is displayed around the child's face. The user (e.g., the child) can adjust the appearance of the child's face by using the display of retouch selection area 33.

When button 43C is operated, the child and the adult bride are selected as subjects for retouching, and two frames are displayed around each of their faces.

In this way, when a captured image that includes an adult and a child is selected as a subject for retouching, the person selection button 36 is displayed in an operable state (on state) for the button used to select the adult and child individually.

When photographed image P3, for example, which shows a close-up of the bride, is selected as the subject to be retouched, person selection button 36 is displayed in a grayed-out state, and buttons 43A, 43B, and 43C constituting person selection button 36 are rendered inoperable. Since the only person appearing in photographed image P3 is the bride, the subject to be retouched is fixed to the bride while photographed image P3 is selected as the subject to be retouched.

When a photographed image containing one person is selected as the subject of retouching, the person selection button 36 is displayed in a disabled state (off state) that is used to select adults and children individually.

The on/off state of the person selection button 36 can be switched depending on the age of the person appearing in the photographed image to be retouched, allowing the user to easily select the person to be retouched.

<<Configuration and operation of retouching PC 1>>
<Configuration of retouching PC 1>
FIG. 16 is a block diagram showing an example of the configuration of the retouching PC 1.

CPU (Central Processing Unit) 101, ROM (Read Only Memory) 102, and RAM (Random Access Memory) 103 are interconnected by bus 104.

An input/output interface 105 is further connected to the bus 104. An input unit 106, an output unit 107, a recording unit 108, a communication unit 109, and a drive 110 are connected to the input/output interface 105.

The input unit 106 is composed of a keyboard, mouse, etc.

The output unit 107 is composed of a display 11, etc.

The recording unit 108 is composed of a hard disk, non-volatile memory, etc. The recording unit 108 records various information such as programs such as the retouching application 1A executed by the CPU 101, captured images, etc.

The communication unit 109 is an interface to networks such as the Internet and a LAN. For example, the communication unit 109 communicates with a storage PC connected to a camera in a photography studio, and receives a captured image sent from the storage PC, in which the user about to begin retouching appears as a subject. The communication unit 109 also communicates with the image management server 2 via the Internet, and sends the retouched captured image to the image management server 2.

The drive 110 controls writing of data to the removable medium 111 and reading of data from the removable medium 111. Import of captured images from a PC for storage may be performed via the removable medium 111.

Figure 17 is a block diagram showing an example of the functional configuration of retouching PC 1.

As shown in FIG. 17, in the retouching PC 1, the retouching application 1A is executed to realize an information processing unit 121. The information processing unit 121 is composed of an image acquisition unit 131, a display control unit 132, a recognition unit 133, a retouching processing unit 134, and a file management unit 135.

In customer service mode, the image acquisition unit 131 reads out and acquires all captured images from the recording unit 108. When shooting is finished, all captured images (e.g., several hundred images) of a group of users are transferred from the PC for storage and recorded in the recording unit 108. The captured images acquired by the image acquisition unit 131 are supplied to the display control unit 132 and the recognition unit 133.

In addition, in the write mode, the image acquisition unit 131 reads and acquires all captured images, including unselected images, from the recording unit 108 and outputs them to the recognition unit 133.

In customer service mode, the display control unit 132 displays an image selection screen based on the captured image supplied from the image acquisition unit 131. After a captured image to be a selected image is selected using the image selection screen, the display control unit 132 displays a retouching screen and controls the display of the retouching screen. While retouching is being performed using the retouching screen, information on the results of image processing is supplied from the retouching processing unit 134.

For example, the display control unit 132 switches the display of the person selection buttons 36 depending on the characteristics of the photographed image selected as the target for retouching. As described above, the display control unit 132 switches the display of the person selection buttons 36 consisting of three types of buttons: a button used to individually select only one person, a button used to individually select only the other person, and a button used to select both people.

In customer service mode, the recognition unit 133 recognizes the features of each selected image selected using the image selection screen. Specifically, the recognition unit 133 recognizes the face of the user appearing as a subject in the selected image, and recognizes attributes such as gender and age for each recognized face (each person). Recognition of faces and attributes is performed using an inference model generated by machine learning.

For example, the recognition unit 133 is provided with an inference model for face recognition that takes a photographed image as input and outputs information on recognized facial feature points. Facial feature points include not only feature points that indicate the facial outline, but also feature points that indicate the position of each part of the face. In addition, the recognition unit 133 is provided with an inference model for attribute recognition that takes a photographed image as input and outputs information on the attributes of each person.

Recognition by the recognition unit 133 may be performed by analyzing the image rather than using an inference model.

Information on the characteristics of the selected image is supplied to the retouching processing unit 134 and the file management unit 135 together with the selected image that was the subject of recognition. The characteristics of the selected image include information on the face ID, facial feature points, face size, and face direction. The face ID is identification information for the recognized face. The face size and face direction are determined, for example, based on the facial feature points.

The characteristics of the selected image also include attributes of each person, such as gender and age.

In addition, in the export mode, the recognition unit 133 recognizes the features of each unselected image in the same manner as in the customer service mode. Information on the recognition results by the recognition unit 133 is supplied to the retouching processing unit 134 and the file management unit 135 together with the unselected image that was the recognition target.

In customer service mode, the retouching processor 134 performs various types of image processing related to retouching in response to user operations on the retouching screen. For example, when an operation related to adjusting the size or shape of the eyes is performed, the retouching processor 134 performs image processing such as shifting the position of the eye outline in response to the content selected by the user.

Specifically, when one of levels "1," "2," "3," or "4" is selected using the adjustment buttons 56 (FIG. 4), the retouching processor 134 extracts the eye area of the person to be retouched by performing masking or other processing. The retouching processor 134 enlarges the image of the extracted eye area according to the selected level, and performs eye enlargement processing by pasting the enlarged image of the eye area at the original position of extraction.

In addition, when an operation to adjust the intensity of makeup is performed, the retouching processing unit 134 performs image processing such as compositing an image of blush of a specified color onto the cheek area, or compositing an image of lipstick of a specified color onto the lips.

When an operation related to adjusting the skin texture is performed, the retouching processing unit 134 performs image processing such as adjusting the brightness of the skin. When an operation related to adjusting the face size (feeling of a small face) or face shape is performed, the retouching processing unit 134 performs image processing such as shifting the position of the face outline according to the content selected by the user.

Specifically, when one of levels "1," "2," "3," or "4" is selected using the adjustment buttons 51 (Figure 4), the retouching processor 134 performs image processing to shift each position of the facial contour of the person being retouched toward the center of the face according to the selected level, thereby making the face smaller.

Information representing the results of image processing by the retouching processing unit 134 is supplied to the display control unit 132 and used to display the retouched screen. When retouching of all selected images is completed, each of the retouched selected images generated by the image processing by the retouching processing unit 134 is supplied to the file management unit 135 together with information indicating the content of the retouching.

In addition, in the export mode, the retouching processing unit 134 reflects the retouching content applied to the retouched selected image to all unselected images. The unselected image to which the retouching content applied to the selected image is to be reflected is selected, for example, based on characteristics recognized by the recognition unit 133. When retouching of all unselected images is completed, each of the retouched unselected images generated by image processing by the retouching processing unit 134 is supplied to the file management unit 135 together with information indicating the retouching content.

In customer service mode, the file management unit 135 saves the selected image that has been retouched in response to user operations. A management file is linked to the file of the retouched selected image based on information supplied from the recognition unit 133, etc.

Figure 18 shows an example of a management file.

As shown in FIG. 18, the management file contains information on the image size, selection flag, and retouched flag. The selection flag is a flag that indicates whether the captured image has been selected as the selected image. The retouched flag is a flag that indicates whether the captured image has been retouched.

The management file also contains face IDs, and linked to each face ID, information on retouching parameters, gender, age, face size, contour points, and face direction. The management file not only records retouching parameters that indicate the retouching content for individual people, but also retouching parameters that indicate the retouching content for the entire image (parameters for style improvement, image brightness, etc.).

In addition, in the write mode, the file management unit 135 reads out the management file of the retouched selected image and outputs it to the retouching processing unit 134. The management file output to the retouching processing unit 134 is used to identify the content of the retouching applied to the selected image. The file management unit 135 also saves unselected images that have been retouched by reflecting all of them. A management file is also linked to the file of the retouched unselected image based on information supplied from the recognition unit 133, etc.

<<Modification of Retouch Screen Display>>
Display of face frame image Although a frame is displayed to surround the face of the person selected as the retouching target using the person selection button 36, the color of this frame may be linked to the color of the button. In this case, each button constituting the person selection button 36 is displayed in, for example, a different color. As shown in Fig. 10A, when button 41A is operated to select the groom on the left side, the frame surrounding the groom's face is displayed in the same color as button 41A. Also, as shown in Fig. 10B, when button 41B is operated to select the bride on the right side, the frame surrounding the bride's face is displayed in the same color as button 41B.

This allows the user to more intuitively recognize which person has been selected as the subject of retouching by operating the person selection button 36.

Display According to the Reliability of Attribute Recognition Results Button 42A, which is operated when selecting only males, and button 42B, which is operated when selecting only females, are displayed based on the gender recognized by the recognition unit 133. However, if the reliability of the recognition result is lower than a threshold value, instead of buttons 42A and 42B, button 41A, which is operated when selecting only people on the left, and button 41B, which is operated when selecting only people on the right, may be displayed.

Similarly, button 43A, which is operated when selecting only adults, and button 43B, which is operated when selecting only children, are displayed based on the age recognized by recognition unit 133. However, if the reliability of the recognition result is lower than a threshold, button 41A, which is operated when selecting only people on the left side, and button 41B, which is operated when selecting only people on the right side, may be displayed instead of buttons 43A and 43B.

In this case, the reliability of the attribute recognition result by the recognition unit 133 is calculated and compared with a threshold value. The reliability output from the inference model for attribute recognition may be used for comparison with the threshold value, or the reliability calculated by calculation based on the face size, face orientation, etc. may be used for comparison with the threshold value.

Examples of Attributes Although the number of people, gender, and age are recognized as person attributes in the above, other attributes may be recognized using an inference model generated by machine learning or analyzed by image processing, so long as they are recognizable attributes, and may be used to display the person selection button 36. Examples of recognizable attributes include the following:
Height: Tall people/Short people Costume color: White clothes people/Black clothes people Hair length: Long hair people/Short hair people

Button Display Position When the user is to select which person to retouch based on attributes such as gender and age, the button display position may be changed depending on the user's position on the captured image.

For example, when a photographed image in which the groom appears on the left and the bride appears on the right (e.g., photographed image P5 in FIG. 12) is selected as the subject of retouching, button 42A, which is operated when selecting only the male as an individual subject to retouching, is displayed on the left in accordance with the positions of the groom and bride, and button 42B, which is operated when selecting only the female as an individual subject to retouching, is displayed on the right.

On the other hand, when a photographed image (not shown) in which the bride appears on the left and the groom appears on the right is selected as the subject of retouching, button 42B, which is operated when selecting only the female subject individually as the subject of retouching, is displayed on the left in accordance with the positions of the bride and groom, and button 42A, which is operated when selecting only the male subject individually as the subject of retouching, is displayed on the right.

Similarly, when a captured image in which a child appears on the left side and a bride appears on the right side (for example, captured image P11 in FIG. 15) is selected as the subject of retouching, button 43B, which is operated when selecting only the child individually as the subject of retouching, is displayed on the left side in accordance with the positions of the child and bride, and button 43A, which is operated when selecting only the bride individually as the subject of retouching, is displayed on the right side.

On the other hand, when a captured image (not shown) in which the bride is on the left and the child is on the right is selected as the subject of retouching, button 43A, which is operated when selecting only the bride individually as the subject of retouching, is displayed on the left in accordance with the positions of the bride and child, and button 43B, which is operated when selecting only the child individually as the subject of retouching, is displayed on the right.

This also allows users to intuitively select the person they want to retouch.

<<About the Style Up feature>>
As described with reference to Fig. 4, style up is provided as one of the retouching functions. The user can adjust the style of a person appearing in a photographed image by operating the adjustment button 55 and selecting a level of "weak" or "strong." As a style adjustment, image processing is performed to make parts of the body other than the face, such as the torso and limbs, appear longer.

<When there is only one subject>
FIG. 19 is a diagram showing an example of a photographed image to be retouched.

In the example of FIG. 19A, a photographed image P21 is shown that shows a close-up of the bride. In the photographed image P21, the bride's upper body is captured from around the waist up.

When captured image P21 is selected as the selected image, feature points p1 to p5 are recognized by the recognition unit 133, as shown in FIG. 19B. Feature points p1 to p5 indicate the center of the face, the top of the head, the chin, the position on the right contour, and the position on the left contour, respectively. In reality, even more feature points are recognized.

Figure 20 shows an example of image processing for style up.

As shown on the left side of Figure 20, the positions of feature points p1 to p5 on captured image P21 are represented by positions (x1, y1) to (x5, y5). When a specific level of style up is selected in this state, captured image P21 is stretched vertically (image processing is performed to stretch it) as shown by the tip of arrow #21. The amount of stretching is shown exaggerated in Figure 20, but in reality, image processing is performed using a slight amount of stretching that corresponds to the level selected by the user.

In the example of Fig. 20, the captured image P21 is stretched in the vertical direction with the center as the reference. The position of the feature point p1 is expressed as the position (x1, y1') by changing only the position in the vertical direction. The position of the feature point p1 in the captured image P21 after stretching is shifted slightly upward from the position of the feature point p1 in the captured image P21 before stretching.

Similarly, the position of the feature point p2 is represented as the position (x2, y2'), the position of the feature point p3 is represented as the position (x3, y3'), and the position of the feature point p4 is represented as the position (x4, y4').
The position of the feature point p5 is represented as position (x5, y5').

In the enlarged captured image P21, the bride appears stretched vertically overall. Note that the enlarged captured image P21 is not the image that the user who is retouching the image sees (it is not displayed on the retouching screen).

In addition, the captured image may not be stretched based on the center, but may be stretched based on another position, such as the top or bottom of the captured image. The position that serves as the basis for stretching may be selected depending on the content of the captured image, such as stretching based on the center of a person's face.

Next, as indicated by the tip of arrow #22, the top and bottom of the stretched captured image P21 are trimmed, and image processing is performed to return each facial feature point to the same position it was in before stretching. As shown on the right side of Figure 20, the positions of feature points p1 to p5 are represented as positions (x1, y1) to (x5, y5). A styled-up captured image P21 is generated by image processing that returns the facial feature points to the same positions they were in before stretching.

Figure 21 shows examples of images taken before and after styling.

By returning only the facial feature points to the same positions as before enlargement, a captured image P21 is generated as an image after style enhancement, in which the face looks the same as before enlargement, but the body appears slimmer, as shown on the right side of Figure 21. The size of the captured image is the same before and after style enhancement.

In this way, by stretching the entire captured image vertically and then stretching only the facial feature points and returning them to their original positions, it is possible to easily generate an image in which the body appears slimmer.

The positions to which each facial feature point is moved can be various positions other than the same positions as before stretching, as long as they are close to the positions before stretching. In other words, image processing after stretching is performed by moving each facial feature point in the direction of the positions before stretching.

If adjustments to the face size or face shape have been made before the appearance adjustments for the style enhancement, the image processing for the style enhancement described above is performed using the facial feature points after the adjustments for the face size or the facial feature points after the shape adjustments. The face that appears in the captured image after the style enhancement will be one that reflects the adjustments to the face size and face shape.

<When there are multiple subjects>
FIG. 22 is a diagram showing another example of image processing for style-up.

In FIG. 22, a photographed image P31 showing a close-up of the bride and groom is shown as the subject of image processing. When a photographed image showing multiple people is targeted, the same processing is carried out as when a photographed image showing one person is targeted.

In other words, when a specific level of style up is selected, the captured image P31 is stretched vertically according to the level selected by the user, as indicated by the tip of arrow #31. After stretching, the captured image P31 shows the groom and bride stretched overall in the vertical direction.

Next, as indicated by the tip of arrow #32, the top and bottom of the stretched captured image P31 are trimmed, and image processing is performed to return the facial feature points of the bride and groom to the same positions they were in before stretching. By performing image processing that returns the facial feature points of the bride and groom to the same positions they were in before stretching, a styled-up captured image P31 is generated.

In this way, when multiple people appear in a captured image that is the subject of style-up image processing, the image processing that returns facial feature points in the stretched captured image to the same positions as they were before stretching is performed on each person's face.

This makes it easy to generate an image that makes people's bodies look slimmer, even when multiple people are in the image.

Even when multiple people appear in a captured image, it is possible to move each facial feature point to a different position other than the same position as before enlargement. In this case, the amount of movement of the facial feature points may be switched between the bride and groom. For example, by moving the facial feature points of the bride, who is a woman, by an amount greater than the amount of movement of the facial feature points of the groom, who is a man, it is possible to generate an image in which the woman's face appears smaller.

The amount of movement of facial feature points may be changed according to other attributes such as age, rather than according to gender. The attributes of each person are identified based on the information recorded in the management file.

The amount of movement of facial feature points may be changed depending on the appearance of the face, such as the size of the face and the direction of the face.

<Modifications for improving style>
Variations in stretching direction Although the stretching direction (extension direction) of the captured image during image processing for style-up is set to be vertical, the captured image may be stretched in a specified direction other than vertical depending on the condition of the subject appearing in the captured image to be processed.

Figure 23 shows examples of stretching directions.

In the example of Figure 23A, a photographed image P41 is shown that shows a close-up of the bride. In photographed image P41, the bride is photographed at an angle, with her head facing in the upper right direction of the image. In wedding photography, images are also taken that show people at various angles like this.

When the bride is photographed at an angle, as shown in FIG. 23B, the captured image P41 is stretched diagonally (upper right to lower left) with the center of the image as the reference point. For example, the same direction as the center line of the face is selected as the stretching direction. The dashed dotted line L1 in FIG. 23B indicates the center line passing through the center of the face. The center line of the face is set, for example, as a straight line connecting the feature point indicating the position of the top of the head and the feature point indicating the position of the chin. The stretching direction may also be set based on other feature points.

Even if the image is stretched diagonally, image processing to style it up is performed by returning only the facial feature points to the same positions they were in before stretching.

By setting the stretching direction according to the state of the person, it is possible to process the image to style up without looking unnatural, even if the person is photographed at an angle. For example, if the captured image P41 shown in FIG. 23 is stretched vertically, the bride's shoulders may appear unnaturally wide in the captured image after the style up, resulting in an unnatural image, but it is possible to prevent this from happening.

Relationship with operation using person selection button 36 Since the entire captured image is stretched, even if multiple people appear in the captured image to be processed, the style-up image processing is not capable of targeting individual people. When a single person to be retouched is selected using the person selection button 36, the adjustment button 55 may be operated to enable or disable the use of the style-up function.

Figure 24 shows an example of the retouch screen.

In the example of FIG. 24, a photographed image P1 showing a bride and groom is displayed as an object to be retouched. In addition, button 41A of person selection buttons 36 is operated, and only the groom, who appears on the left side, is selected as an object to be retouched.

In this state, for example, if the style up is selected to level "2" using the adjustment button 55 as shown in the lower left of FIG. 24, the above-mentioned image processing is performed on the photographed image P1 according to the parameters of level "2." After the style up, the photographed image shows the bride and groom with their bodies looking slimmer according to level "2."

After that, for example, when button 41B of person selection buttons 36 is operated and only the bride on the right side is selected as the subject of retouching, and when style up is selected as level "1" using adjustment button 55, image processing is performed on captured image P1 according to the parameters of level "1." After style up, the captured image will show both the groom and the bride, whose bodies look slimmer according to level "1."

In this way, even if the person to be retouched is selected using the person selection button 36, when style up is selected, a captured image is generated that shows multiple people whose styles have been adjusted using the same parameters (levels).

Figure 25 shows another example of the retouch screen.

In the example of FIG. 25, captured image P1, which shows the groom and bride, is also displayed as the subject of retouching. In addition, button 41A of person selection buttons 36 is operated, and only the groom, who appears on the left side, is selected as the subject of retouching.

When a captured image contains multiple people, and only one of the people is selected as the subject of retouching, the adjustment button 55 may be displayed in an inoperable state, as shown in the lower left of FIG. 25. The adjustment button 55 in FIG. 25 is colored to indicate that the adjustment button 55 is grayed out and inoperable.

This will prevent the Style Up feature from making unintended adjustments to the appearance of people who are not selected for retouching.

<<Modification of Face Frame Image Display>>
15, the child is selected as a retouching target and the appearance of the face can be adjusted, but the child may be excluded from the retouching target. In bridal photography, the bride and groom may be photographed including the child.

Figure 26 shows another example of the retouch screen.

Thumbnail images of the photographed images P1 to P4 and the photographed image P11 are displayed in the thumbnail area 32 of the retouch screen shown in FIG. 26, and the photographed image P11 selected using the thumbnail image is enlarged and displayed in the display area 31. The photographed image P11 is an image in which a child is captured between the bride and groom.

A frame image F1 is displayed around the bride's face on the right, and a frame image F2 is displayed around the child's face. Frame images F1 and F2 are frame images of different colors. No frame image is displayed around the groom's face on the left.

Frame image F1 is UI information indicating that the bride is the subject of retouching. In the example of FIG. 26, button 41B is operated, and the bride, who is the person on the right side, is selected as the subject of retouching.

On the other hand, frame image F2 is UI information indicating that retouching cannot be applied to children (retouching is not possible). That is, in the example of FIG. 26, children are automatically excluded from the subjects to be retouched without any user operation, and frame image F2 indicating that retouching cannot be applied is displayed on image P11 together with frame image F1 showing the person to be retouched.

There is a need to avoid retouching children's faces because it can look very unnatural. By displaying frame image F2 and clearly indicating that retouching of children is not possible, the bride and groom can adjust their appearance without worry.

Figure 27 shows an example of how a frame image is displayed.

For example, when button 41A is operated in the state shown in FIG. 26 and the groom, who is the person on the left, is selected as the subject to be retouched, a frame image F3 is displayed surrounding the groom's face, as shown in FIG. 27A. A frame image F2 continues to be displayed around the child's face. No frame image is displayed around the face of the bride, who can be selected as a subject to be retouched but has not been selected as a subject to be retouched.

In addition, when button 41C is operated to select the bride and groom as subjects for retouching, frame images F1 and F3 are displayed to surround the faces of the two people, as shown in FIG. 27B. Frame image F2 continues to be displayed around the child's face.

Display of the frame image F2 begins when, for example, a photographed image that includes a child is selected using a thumbnail image, and continues while the same image is selected for retouching. If, for example, multiple children appear in the photographed image to be retouched, a frame image is displayed around each child's face to indicate that retouching is not possible.

Figure 28 shows an example of how to select whether or not to retouch an image.

As shown in FIG. 28, the retouching possible/not possible selection model M1 is prepared, for example, in the recognition unit 133 (FIG. 17). The retouching possible/not possible selection model M1 is an inference model that takes a photographed image as input and outputs information indicating whether each person appearing in the photographed image can be retouched. The information indicating whether retouching is possible is linked to the face ID of each person's face. The retouching possible/not possible selection model M1 is generated, for example, by machine learning using photographed images in which a label indicating whether retouching is possible or not is set for each person as training data.

The recognition unit 133 inputs each photographed image selected using the image selection screen into the retouching possibility selection model M1, and selects whether or not each person can be retouched. Whether or not each person can be retouched is selected based on the characteristics of the photographed image.

When the retouching screen is displayed and a captured image containing a person who cannot be retouched is selected as the subject of retouching, the display control unit 134 displays a frame image superimposed on the captured image indicating that retouching cannot be applied.

The selection of whether or not retouching is possible may be made based on the results of image analysis, rather than using an inference model generated by machine learning. The selection of whether or not retouching is possible may also be made based on the age recognized by the recognition unit 133. For example, the recognized age is compared with an age set as a threshold, and people who are below the threshold age are selected as people who cannot be retouched.

An inference model that outputs a determination result of whether the person is an adult or a child may be prepared in the recognition unit 133, and the selection of whether or not retouching is possible may be made based on the output of this inference model.

The user may be allowed to select people who should not be retouched. For example, the user may select people not to be retouched by designating the face of a person on the captured image displayed in the display area 31.

<<About feature recognition>>
<Type of features>
Although the recognition unit 133 recognizes features including human attributes such as gender and age, the following features of the captured image may be recognized (determined). Each feature is recognized, for example, by using an inference model generated in advance by machine learning, or based on the analysis result of the image.

・Face angle (direction)
The orientation of the person's face is recognized. For example, one of three orientations, i.e., facing forward, diagonally, or sideways, is recognized.

・Facial expressions A person's facial expressions, such as closing their eyes or smiling, are recognized.

Angle of view: The angle of view of the captured image (the range in which a person is captured) is recognized. For example, depending on the size of the face relative to the size of the captured image, the angle of view may be recognized as a close-up, a full body, etc.

- Presence or absence of catchlights This function recognizes whether or not there is a catchlight in a person's eyes.

・Facial brightness The brightness of the face, such as luminance and illuminance, is recognized.

Skin color: It is recognized whether or not the skin color is within an appropriate range. A specific color range is set in advance as the appropriate range.

Degree of Facial Overlapping Based on the positional relationship between the facial region of one person and the facial region of another person, it is recognized whether a face is hidden or whether the faces are touching each other.

- Hair appearance The state of the hair appearance, such as hair color and hair brightness, is recognized.

- Whether or not the whole body is captured Whether or not a person's whole body is captured is recognized.

-Body direction: If a person's body is in the photo, its direction will be recognized.

- Type of costume The type of costume, such as dress, kimono, uchikake, etc., is recognized.

Photographing scene Photographing scenes such as the location of photography are recognized. Optical conditions at the time of photography such as backlighting, blown-out highlights, etc. may be recognized as the photographing scene.

Whether or not rules are violated When rules are set for each type of photography course, such as rules for bridal photos, rules for coming-of-age ceremony photos, etc., it is recognized whether or not the rules are violated. In a course for taking school ID photos, for example, it is recognized whether or not school rules are violated.

It is possible for the recognition unit 133 to recognize at least one of the various features described above.

<Use of recognition results>
- Example of use in retouching image processing It is possible to change the content of retouching based on the result of feature recognition. For example, when the type of clothing is recognized, if it is recognized that the type of clothing is an uchikake, processing to improve the style is automatically performed, and image processing is performed to make the whole body look slimmer. Also, when retouching to make the upper arms (upper arms) slimmer is prepared, it is possible to prevent the retouching from being performed when the type of clothing is recognized as a traditional Japanese garment.

In this way, it is possible to change the content of the image processing performed as retouching based on the results of recognizing the characteristics of the captured image.

Example of using recognition results for presentation The feature recognition results may be presented to the user.

Presentation on Retouch Screen FIG. 29 is a diagram showing an example of presentation of the feature recognition results.

The captured image P11 shown in FIG. 29A is an image that has been enlarged and displayed on the retouch screen as described with reference to FIG. 26. A frame image F1 is displayed surrounding the face of the bride on the right side, and a frame image F2 is displayed surrounding the face of the child.

In addition, next to the frame image F1 that surrounds the face of the bride to be retouched, information is displayed indicating that the gender of the person to be retouched has been recognized as "female" and that the orientation of the face has been recognized as "diagonal." In the example of Figure 29, the recognition results of the child's features are not displayed.

When the person to be retouched is switched to the groom, as shown in FIG. 29B, a frame image F3 is displayed surrounding the face of the groom to be retouched, and next to it is information indicating that the gender of the person to be retouched has been recognized as "male" and that the orientation of his face has been recognized as "frontal."

In this way, by displaying information showing the recognition results, the user can check whether the features have been recognized correctly.

Some of the retouching functions described above have different image processing intensities depending on whether the subject is a woman or a man, even if the same level of parameters are selected. There are also functions prepared for women and functions prepared for men (for example, a "beard remover" function). By presenting the feature recognition results, the user can use those functions to perform retouching after confirming that the recognition results are correct.

Figure 30 shows another example of the presentation of recognition results.

In the example of FIG. 29, only the features of the person selected as the subject of retouching are displayed, but as shown in FIG. 30, the recognition results of the features of all people appearing in the captured image may be displayed. In the example of FIG. 30, information indicating the recognition result is displayed next to the face of the groom selected as the subject of retouching, and information indicating the recognition result is also displayed next to the face of the bride not selected as the subject of retouching. Information indicating the recognition result is also displayed next to the face of a child not to be retouched. As the recognition result of the child's features, information indicating that the person's age has been recognized as "child" and that the orientation of the face has been recognized as "frontal" is displayed.

Instead of displaying the feature recognition results as text, other information such as icons or marks may be displayed. The feature recognition results may be presented as audio or using animation.

Information indicating the result of feature recognition may be displayed in a specified area outside the photographed image, rather than being displayed at a position on the photographed image to be retouched. Information indicating the result of feature recognition may be displayed when the cursor is placed over a face on the photographed image.

In the examples of Figures 29 and 30, information indicating gender, facial orientation, and age is displayed, but the type of features to be displayed as the recognition result may be selected by, for example, the staff of a photography studio. The user may also be allowed to select the type of features while retouching the image.

Presentation on Image Selection Screen The feature recognition results may be presented on an image selection screen that is used to select a captured image to be retouched.

Figure 31 shows an example of an image selection screen.

The number of selected images is set, for example, by a staff member at the photography studio, before selecting images. If a user applies for a plan to receive five images printed as an album, five is set as the number of selected images.

A thumbnail display area 201 is formed on the right side of the image selection screen. The thumbnail display area 201 is an area in which thumbnail images of the captured images are displayed. For example, if 50 captured images have been taken, thumbnail images of each of the 50 captured images are displayed. In the example of FIG. 31, it is possible to display thumbnail images of nine captured images at once. By pressing the switching buttons ("NEXT" and "BACK"), the user can switch the thumbnail images displayed in the thumbnail display area 201.

A captured image selected using a thumbnail image is enlarged and displayed in a display area 202 formed to the left of the thumbnail display area 201. In the example of FIG. 31, the thumbnail image displayed at the bottom right edge of the thumbnail display area 201 is selected, and the captured image is enlarged and displayed in the display area 202.

In addition, a frame image is displayed around the faces of the bride and groom, and information showing the recognition results of the bride and groom's features is displayed on top of it.

A confirm button 203 is displayed at the bottom right of the display area 202. The confirm button 203 is a button that is pressed when deciding on the captured image displayed in the display area 202 as the image to be retouched. The user selects thumbnail images to sequentially enlarge and display each captured image in the display area 202, and when the user finds an image that the user likes, the user presses the confirm button 203 to select it as the image to be retouched.

Below the display area 202, a selected image display area 204 is formed in which thumbnail images of the captured images selected by the user are displayed. When four captured images P1 to P4 are selected, the respective thumbnail images are displayed side by side as shown in FIG. 31.

When the selection of five captured images has been completed, the image selection process is complete. After the selection of the captured images to be retouched has been completed, retouching of each of the selected captured images begins.

In this way, by displaying information showing the recognition results, the user can check whether the characteristics of each captured image have been correctly recognized, and can select captured images without erroneous recognition as targets for retouching. For example, by displaying information showing that the eyes are closed in the photograph, the user can exclude such captured images from targets for retouching. In addition, by displaying information showing the recognition results of the angle of view, the user can exclude captured images with undesirable angles of view from targets for retouching.

As described above, the reflection of all retouched images is performed by selecting which of the retouched images to reflect the retouching of, based on the characteristics of the selected and unselected images. In other words, the retouching applied to the image selected to be retouched is applied to unselected images that have the same characteristics as the image selected. For example, the retouching applied to the bride is reflected to the bride who appears in a similar state in the unselected images, and the retouching applied to the groom is reflected to the groom who appears in a similar state in the unselected images.

When full reflection is performed in this manner, if there is a misrecognition of the characteristics of a retouched photographed image, the retouching applied to the bride may be incorrectly reflected on the groom in an unselected image, or the retouching applied to the groom may be incorrectly reflected on the bride in an unselected image. By selecting photographed images that are not misrecognized as the images to be retouched, it is possible to prevent such erroneous reflections from occurring when full reflection is performed.

The presentation of the feature recognition results on the image selection screen is realized, for example, by performing feature recognition by the recognition unit 133 on all captured images. When a thumbnail image is selected in the thumbnail display area 201, the features of the captured image corresponding to that thumbnail image may be recognized, and the recognition results may be presented.

<<Others>>
Regarding the program: The above-mentioned series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed in a computer incorporated in dedicated hardware, or in a general-purpose personal computer.

The program to be installed is provided by being recorded on removable media 111 shown in FIG. 16, which may be an optical disk (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), etc.) or a semiconductor memory. It may also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting. The program can be pre-installed in ROM 102 or recording unit 108.

The program executed by the computer may be a program in which processing is performed chronologically in the order described in this specification, or it may be a program in which processing is performed in parallel or at the required timing, such as when called.

The effects described in this specification are merely examples and are not limiting, and other effects may also exist.

The embodiment of this technology is not limited to the above-mentioned embodiment, and various modifications are possible without departing from the gist of this technology.

For example, this technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices over a network.

In addition, each step described in the above flowchart can be executed by a single device, or can be shared and executed by multiple devices.

Furthermore, when a single step includes multiple processes, the multiple processes included in that single step can be executed by a single device, or can be shared and executed by multiple devices.

1 Retouching PC, 2 Image management server, 3 Mobile terminal, 11 Display, 121 Information processing unit, 131 Image acquisition unit, 132 Display control unit, 133 Recognition unit, 134 Retouching processing unit, 135 File management unit

Claims (16)

a recognition unit that recognizes features of each captured image including attributes of a person appearing as a subject;
a display control unit that switches display of a selection button used to select the person to be processed, which constitutes a processing screen used to select the content of processing for the photographed image, in accordance with a feature of the photographed image being selected as the processing target;
an image processing unit that processes a portion of the person selected by a user using the selection button in the captured image that is being selected as a processing target. The information processing device according to claim 1 , wherein the display control unit switches the display of the selection button every time the selected captured image is changed. The information processing device according to claim 1 , wherein the display control unit turns on the display of the selection button when a plurality of people are depicted in the captured image selected as the processing target, and turns off the display of the selection button when one person is depicted in the captured image. The information processing device according to any one of claims 1 to 3, wherein the display control unit displays three types of buttons as the selection buttons: a first button used to select one of the people appearing in the captured image selected as the subject of processing, a second button used to select the other person, and a third button used to select both people. The information processing device according to claim 4 , wherein the display control unit displays the selection buttons including the first button used to select the person appearing on the left side and the second button used to select the person appearing on the right side. The recognition unit recognizes the gender of each of the persons appearing in the captured image as the attribute,
The information processing device according to claim 4 , wherein the display control unit displays the selection buttons including the first button used to select a male and the second button used to select a female. The recognition unit recognizes an age of each of the persons appearing in the captured image as the attribute,
The information processing device according to claim 4 , wherein the display control unit displays the selection buttons including the first button used to select an adult and the second button used to select a child. An information processing device,
It recognizes the characteristics of each captured image, including the attributes of the people in the image,
switching display of a selection button used to select the person to be processed, which constitutes a processing screen used to select the content of processing for the photographed image, in accordance with a feature of the photographed image being selected as the processing target;
the information processing method further comprising: processing a portion of the person selected by a user using the selection button in the captured image selected as a processing target. On the computer,
It recognizes the characteristics of each captured image, including the attributes of the people in the image,
switching display of a selection button used to select the person to be processed, which constitutes a processing screen used to select the content of processing for the photographed image, in accordance with a feature of the photographed image being selected as the processing target;
a program for executing a process of applying processing to a part of the person selected by a user using the selection button in the captured image selected as a processing target. a display control unit that displays a processing screen used for selecting the processing content of the captured image;
an image processing unit that processes a portion of the photographed image selected as a processing target, the portion being selected by a user to process the person;
The display control unit displays first UI information for a first person selected by the user, indicating that the first person is a target for processing, and displays second UI information for a second person different from the first person, indicating that processing cannot be applied to the second person. The information processing device according to claim 10 , wherein the display control unit displays frame images of different colors as the first UI information and the second UI information. A recognition unit that recognizes features of each of the persons appearing in the captured image is further provided,
The information processing device according to claim 10 , wherein the display control unit displays the second UI information for the second person selected as a person not to be processed based on a result of the recognition of the features. The information processing device according to claim 12 , wherein the display control unit displays the second UI information for a child who is the second person. The information processing device according to claim 10 , wherein the display control unit causes the second UI information to be displayed for the second person selected by the user. a display control unit that displays a processing screen used for selecting the processing content of the captured image;
an image processing unit that processes a part of a person selected by a user in the captured image that is being selected as a processing target,
displaying first UI information indicating that a first person selected by the user is a processing target;
The information processing method further comprises displaying second UI information indicating that processing cannot be performed on a second person different from the first person. a display control unit that displays a processing screen used for selecting the processing content of the captured image;
an image processing unit that processes a part of the photographed image selected as a processing target, the part being selected by a user,
displaying first UI information indicating that a first person selected by the user is a processing target;
a program for executing a process of displaying second UI information indicating that processing cannot be performed on a second person different from the first person.

Images