JP2013003412A - Automatic photograph creation device, automatic photograph creation method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To change a background image in a manner that a user can sufficiently obtain amusement such as a stereoscopic effect and the sense of presence.SOLUTION: An automatic photograph creation device performs the steps of: recognizing a face area of a user included in a photographed image (S400); when there are a plurality of users, calculating a center coordinate on the basis of each face area (S410); calculating the amount of deformation corresponding to the displacement amount of the center coordinate from a predetermined reference position (S420); and executing processing for deforming a background image according to the calculated amount of deformation and a displacement direction (S430). The background image is composed with the user's image, and continuously displayed while being outputted as a photographic sticker. Thus, the user can sufficiently obtain amusement such as a stereoscopic effect and the sense of presence in the displayed image and the photographic sticker.

Description

  The present invention relates to an automatic photo creation device, an automatic photo creation method, and a program, and more specifically, an automatic photo creation device that takes a user with a camera and outputs a composite image generated based on the taken image as a photograph, The present invention relates to an automatic photo creation method and program.

  2. Description of the Related Art Conventionally, there is known a photo creation device for a game in which a user is photographed with a camera and the photographed image is output as a photo sticker or a photo card. Such a play photo creation device is highly playable or entertaining, and therefore, an image to be combined with a photographed image is prepared from a variety of background images and foreground images prepared in advance according to the user's preference. Many are configured so that a user can select or draw freely using a touch pen. Since such an operation by the user is performed on the photographed image, it is called “graffiti”.

  In addition, the background image used in such a game photo creation device is usually a single (non-changing) still image prepared in advance, but a user's image with a moving image changing with time as a background. There is also an automatic photo creation device that combines and displays a still image corresponding to a moving image displayed at the time of shooting with a user's image as a background image (see, for example, Patent Document 1). By using the background image that changes in this way, the playability of the device is further improved.

JP 2005-142788 A

  However, in the configuration described in the above publication, since a background is generated by reproducing a moving image, the background basically changes only in advance with time. Therefore, the playability such as a sense of presence that the user can obtain from the background remains within the range obtained by being caused (experienced) by the content of the moving image.

  In this regard, if the background image is configured to move according to the user's operation (typically horizontal scrolling), it does not change in advance with time, so the playability of the device is More improved. However, since the range in which the image changes remains within the range based on the predetermined data, it cannot be said that the unexpectedness is large. In particular, the user can be obtained by changing the background image according to a mode such as horizontal scrolling. It cannot be said that the playability such as three-dimensionality and presence is great.

  Accordingly, an object of the present invention is to provide an automatic photo creation device, an automatic photo creation method, and a program that change a background image in a manner in which a user can sufficiently obtain playability such as a three-dimensional effect and a sense of reality.

The present invention captures a user, outputs a composite image including a photographed image acquired by photographing as a photograph, detects the position or orientation of the user, and displays the image.
A predetermined background image is changed based on the detected position or orientation with respect to the user image included in the captured image continuously acquired by shooting, and continuously with the user image. Is displayed in a composite. Then, based on the position or orientation detected when the captured image to be included in the composite image is acquired, the composite image obtained by combining the background image changed in the same manner as the change in the composite display with the captured image acquired by the capture Is generated.

  According to the present invention, the position or orientation of the user is detected, a predetermined background image is changed based on the detected position or orientation, and control is performed so as to continuously synthesize and display the image of the user. Based on the position or orientation detected at the time when the captured image to be included in the composite image is acquired, a composite image obtained by combining the changed background image with the captured image is generated and output as a photograph, so that it is displayed. In the synthesized image and the output photo, the user can sufficiently obtain playability such as a three-dimensional effect and a sense of reality.

It is a figure which shows the external appearance of the photo production apparatus for play which is an automatic photo production apparatus which concerns on the 1st Embodiment of this invention. It is a front view of the imaging unit in the embodiment. It is a front view of the edit unit in the said embodiment. It is a front view of the output unit in the said embodiment. It is a schematic diagram which shows the display structure of the touchscreen for imaging operations in the said embodiment. It is a block diagram which shows the structure which looked at the principal part of the photo production apparatus for play concerning the said embodiment from the functional surface. It is a flowchart which shows the procedure of the imaging | photography process in the said embodiment. It is a flowchart which shows the procedure of the graffiti edit process in the said embodiment. It is a flowchart which shows the procedure of the background image deformation | transformation process in the said embodiment. It is a figure which shows the picked-up image in the said embodiment simply. It is a figure for demonstrating the center coordinate calculation in the said embodiment. It is a figure which illustrates the background image prepared in advance in the said embodiment. It is a figure which shows the synthesized image which synthesize | combined without deform | transforming the background image shown in FIG. 12 in the said embodiment. In the said embodiment, it is a figure which shows simply the picked-up image when a user moves to the left direction of an image. In the said embodiment, it is a figure for demonstrating center coordinate calculation when a user moves to the left direction of an image. It is a figure which shows the background image obtained by image-transforming the background image shown in FIG. 12 in the said embodiment. It is a figure which shows the synthesized image which synthesize | combined the background image shown in FIG. 16 in the said embodiment, and the user's image shown in FIG. 14 in this embodiment. In the said embodiment, it is a figure for demonstrating the image deformation | transformation in the upper surface of the rectangular parallelepiped which comprises a background. In the said embodiment, it is a figure for demonstrating the image deformation | transformation in the left side surface of the rectangular parallelepiped which comprises a background. In the said embodiment, it is a figure which shows simply the picked-up image when a user moves to the downward direction of an image. In the said embodiment, it is a figure which shows a synthesized image when a user moves below as shown in FIG. In the said embodiment, it is a figure which shows simply the picked-up image when a user moves to the near side of an image. In the said embodiment, it is a figure which shows the synthetic | combination image which synthesize | combined the background image deform | transformed so that the camera might be arrange | positioned far away, and the user's image shown in FIG. In the 2nd Embodiment of this invention, it is a figure which shows the synthesized image synthesize | combined without changing a background image. In the said embodiment, it is a figure which shows simply the method of producing | generating the background image shown in FIG. 24 based on three-dimensional data. In the said embodiment, it is a figure which shows a synthesized image when a user moves to the left direction. In the said embodiment, it is a figure which shows simply the method of producing | generating the background image shown in FIG. 26 based on three-dimensional data. In the modification of the said 1st Embodiment, it is a figure which shows a synthesized image in case the direction of a user's face is facing forward. In the modification of the said 1st Embodiment, it is a figure which shows a synthesized image in case the direction of a user's face is facing the right direction. In the modification of the said 1st Embodiment, it is a figure which shows the synthesized image synthesized without deform | transforming a background image and without moving a foreground image. In the modification of the said 1st Embodiment, it is a figure which shows the synthesized image which deform | transformed the background image and moved the foreground image and synthesize | combined.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.
<1. First Embodiment>
<1.1 Overall configuration>
FIG. 1 is a diagram showing an appearance of a play photo creation apparatus that is an automatic photo creation apparatus according to a first embodiment of the present invention. More specifically, FIG. 1 (a) is an external side view of the play photo creation apparatus as viewed from the side, and FIG. 1 (b) is an external plan view as viewed from above. This play photo creation apparatus accepts an editing operation including a shooting room 2 in which a user enters, a shooting unit 3 for shooting a user to obtain a shot image, and a graffiti (drawing operation) by the user for the shot image. The editing unit 4 generates a composite image combined with the captured image, and the output unit 5 outputs the composite image. FIG. 2 is a front view of the photographing unit 3, FIG. 3 is a front view of the editing unit 4, and FIG. 4 is a front view of the output unit 5. Hereinafter, the overall configuration of the game photo creation device according to the present embodiment will be described with reference to FIGS. 1 to 4.

  The photographing room 2 has a substantially rectangular parallelepiped shape, and the photographing unit 3 is arranged along the front surface of the photographing room 2 which is a front surface when viewed from the user entering the inside. Note that an opening for a user to enter and exit, and a light-shielding curtain that covers a part or all of the opening are provided on a part of each of the left and right side surfaces of the photographing room 2. In addition, a single color (in this case, blue or green) for chroma key composition processing is attached to the rear surface, which is the rear surface as viewed from the user who enters the photographing room 2. In addition, these colors may be attached | subjected also to the floor surface etc. which are included in the imaging | photography range of the imaging | photography unit 3, for example.

  The photographing unit 3 includes a camera 10 as an imaging unit for photographing a user, strobes 11, 12, 13 L, 13 R, and 14 that are arranged at the top, bottom, left and right positions of the camera 10 and emit flash light, and below the camera 10. It is provided with a shooting operation touch panel 20 that is arranged and receives operations from a user, displays shot images, and the like.

  The camera 10 is typically a digital camera that generates a digital image signal using a CCD (Charge Coupled Device), captures a user, and outputs an image signal representing the captured image. The flashes 11 to 14 emit a flash toward the user in order to obtain sufficient light for photographing. The shooting operation touch panel 20 is configured to provide an operation screen for accepting various operations by the user during shooting, and to display an image based on the image signal in real time.

  FIG. 5 is a schematic diagram showing a display configuration of the touch panel 20 for shooting operation. As shown in FIG. 5, at the time of shooting, the shooting operation touch panel 20 has a real-time preview area 201 for displaying a shot image in real time, a pose sample display area 202 for displaying a pose sample, and a shooting. The graffiti target image display area 203 for displaying the graffiti target image obtained by the above is displayed. In the real-time preview area 201, as will be described in detail later, a background image deformed according to the position of the user is displayed in a chroma key composition display on the captured image.

  The photographing unit 3 is built around a computer and incorporates a control device for controlling each part, an I / O control device, a network adapter for communicating with the editing unit 4, and the like. The photographing unit 3 includes a coin insertion slot 26 at the lower front side.

  The editing unit 4 is configured around a computer similar to the photographing unit 3, and includes a control device that controls each part and the like, and a network adapter that communicates with the photographing unit 3 and the like. Further, as shown in FIG. 1B, the editing unit 4 is divided into two units 4a and 4b so that two sets of users can play. One of the units 4a includes an editing operation touch panel 400 that includes a graffiti area, an area for displaying a menu for graffiti, a tool, and the like and functions as a GUI (Graphical User Interface) display means, and the editing operation touch panel 400 Touch pens 49L and 49R are provided as pointing devices used for the operation of. The other unit 4b has the same configuration. Here, the unit 4a seen in the foreground in FIG. 1A is referred to as “graffiti booth A”, and the unit 4b on the opposite side (the back side in the figure) is referred to as “graffiti booth B”.

  The editing operation touch panel 400 according to the present embodiment has a display configuration in which each of the units 4a and 4b allows two or three users to simultaneously doodle. In this description, components used by the user on the left side are given reference signs including “L”, and components used by the user on the right side are assigned reference signs including “R”. ing. In addition, when three users use, in addition to this, reference numerals including “C” are given to components used by the central user.

  In addition, the editing operation touch panel 400 according to the present embodiment is configured to be able to simultaneously detect the operation positions of the plurality of touch pens 49L and 49R, and each of the detected operation positions is any of the touch pens 49L and 49R. It is also possible to detect whether it corresponds to the above operation. For example, when a capacitive touch panel is used as the editing operation touch panel 400, simultaneous detection of such a plurality of operation positions and identification of the operated touch pen are possible.

  As shown in FIG. 4, the output unit 5 is typically provided with a take-out port 33 for taking out a photo sticker, a photo card, or the like on which a composite image that has been edited by the editing unit 4 is printed. . Further, the output unit 5 transfers, for example, a material image such as a Deco-mail (registered trademark) image or a composite image obtained by grabbing a photographed image to the user's mobile phone terminal by using an infrared port built in the mobile phone terminal, for example. Output operation touch panel 30 that is operated by the user, and an infrared port (non-contact) that is arranged below the output operation touch panel 30 and directly transmits the material image or the like as an infrared signal to the mobile phone terminal. In some cases, a communication port) 31 and a speaker 32 that informs the user by voice of operation methods and sound effects necessary for the communication may be provided.

  The output operation touch panel 30 transmits the material image or the like to the mobile phone terminal having the non-contact communication function when the user wants to view the composite image not only on the photo sticker on which the composite image is printed but also on the mobile phone terminal. It is configured to provide an operation screen for accepting various operations necessary for this. This operation is mainly performed using the time until the photo sticker of the composite image is printed. Therefore, the user should use the time effectively until the photo sticker etc. is printed. Can do. Further, without providing the output operation touch panel 30 in the output unit 5, it is possible to perform the operation on the editing operation touch panel 400 of the editing unit 4.

  Such an output unit 5 is also configured around a computer similar to the photographing unit 3, and includes a control device for controlling each part, a network adapter for communicating with the editing unit 4, etc. A network printer 35 for printing an image as a photo sticker or the like is provided.

  In the configuration as described above, after shooting in the shooting room 2, the user uses the graffiti booth A of the editing unit 4 or the touch panel 400 for the editing operation of the graffiti booth B to generate based on the shot image. Graffiti is performed on the graffiti target image. Then, the user prints the composite image generated by the graffiti by the network printer, transmits the image to the mobile phone terminal having the infrared communication function, and displays the image received by the mobile phone terminal on the terminal screen. .

<1.2 Functional configuration>
FIG. 6 is a block diagram showing the configuration of the main part of the play photo creating apparatus according to the present embodiment as seen from the functional aspect. 6 is functionally according to a photographing processing unit 7 for mainly performing a process of photographing a user (photographing process) and a graffiti operation of the user mainly on a graffiti target image. Edit processing unit 8 for performing the editing process of the graffiti target image, and output the graffiti target image subjected to the editing process as a photo sticker, etc. And an output processing unit 9 that performs processing (output processing) for outputting to a mobile phone terminal.

  The imaging processing unit 7 includes a first control unit 70, an imaging unit 71, a first display / operation unit 72, an I / O control unit 73, an illumination unit 74, and a first communication unit 75. It is configured. The edit processing unit 8 includes a second control unit 80, second display / operation units 81 and 82, and a second communication unit 83. The output processing unit 9 includes a third control unit 90, a third display / operation unit 91, a print output unit 92, an audio output unit 93, a non-contact communication unit 94, and a third communication unit 95. And a wireless communication unit 96. The first, second, and third communication units 75, 83, and 95 that are network adapters can communicate with each other via a network 6 that is a LAN (Local Area Network).

  The imaging unit 71 corresponds to the camera 10 configured using an imaging element such as a CCD, and captures an image in real time and outputs an image signal representing the image (captured image). This image signal is input to the first control unit 70 and temporarily stored as photographed image data in its internal memory. The captured image data is supplied as a captured image signal from the first control unit 70 to the first display / operation unit 72, and a captured image based on the captured image signal is displayed in real time. Actually, the captured image data stored as the captured image is high-resolution still image data, and the captured image data (also referred to as “through image data”) for real-time display is low-resolution moving image data. However, since the subject to be photographed is the same, the following description will be made without distinguishing them in principle.

  The first display / operation unit 72 corresponds to the shooting operation touch panel 20 and accepts an operation for selecting a background image (and foreground image) to be added to the shot image, a shutter operation, and the like. Signals indicating these operations are input to the first control unit 70 as operation signals. Here, when a predetermined process for photographing the user (corresponding to the selected photographing menu) is started, guidance for the user is displayed on the first display / operation unit 72, and thereafter Based on an instruction from the first control unit 70, a flash is emitted from the flashes 11 to 14 in the shooting direction of the camera 10 after a predetermined time of about several seconds. At that time, an image signal output from the imaging unit 71 as a signal representing a photographed image of the user is input to the first control unit 70, a memory in the first control unit 70, a hard disk device as an auxiliary storage device, or the like. Stored as captured image data.

  The illumination unit 74 corresponds to the strobes 11, 12, 13 </ b> L, 13 </ b> R, 14 disposed at the top, bottom, left, and right positions of the camera 10, and is turned on / off by the I / O control unit 73 based on an instruction from the first control unit 70. Turning off and dimming are controlled. The I / O control unit 73 corresponds to an I / O control device built in the photographing unit 3, and controls the illumination unit 74 based on an instruction from the first control unit 70. Further, an input signal such as a detection signal from a coin detection unit (not shown) described later is transferred to the first control unit 70. The first communication unit 75 corresponds to a network adapter built in the photographing unit 3 and functions as an interface for data transmission / reception via the network 6.

  The first control unit 70 is built in the photographing unit 3 and corresponds to a control device mainly composed of a computer including a CPU, a memory, a frame buffer, a timer, an auxiliary storage device and the like, and is stored in a predetermined memory stored in an internal memory. When the CPU executes the program, an instruction is given to each unit as described above in order to control each unit based on the operation signal or the like input as described above. A graffiti target image is generated based on the photographed image. The generated graffiti target image is written in the frame buffer and displayed on the first display / operation unit 72. Further, the first control unit 70 generates a composite image that is an image in which a predetermined background image or foreground image is drawn on the graffiti target image. As will be described in detail later, this graffiti target image has a background color that is a key color used to create a mask, and based on the well-known chroma key composition processing method, the image includes only the user's image. An image part is extracted and synthesized so that the photographed image part is inserted into the background image. The composite image generated in this way is displayed on the first display / operation unit 72 by being written in the frame buffer.

  In addition, the first control unit 70 performs a known face recognition process (face detection process) on the photographed image, thereby detecting the position of the user's face included in the photographed image, and according to the position. The background image (and the foreground image in some cases) deformed in this manner is continuously synthesized and displayed on the first display / operation unit 72, and is similarly transformed when a captured image to be included in the synthesized image is acquired. A synthesized image is generated by synthesizing the background image and the captured image. This point will also be described in detail later. When the shooting and the generation of the composite image are completed in this way, the generated composite image is appropriately selected according to the input operation of the user, and then the second corresponding to the editing unit 4 via the first communication unit 75. It is sent to the control unit 80. The face recognition process is not limited as long as the position of the user can be detected, and the user need not be individually identified, but may be identified. As will be described later, the face orientation may be detected. Furthermore, if the user's position can be detected, various well-known position detection methods that do not use face recognition processing can be employed. For example, a floor (typically in a matrix) is used to detect the position of the user. B) A configuration may be provided in which a pressure sensor, a plurality of optical sensors, an ultrasonic sensor, and the like are provided. However, since it is expensive to newly provide these sensors, it is typically preferable to use a face recognition process that can detect the position only by software processing.

  In addition to the above-described components, the photographing unit 3 is further provided with a coin detection unit (not shown) for detecting coins inserted into the coin insertion slot 26 in the photographing unit 3, and the first control unit 70 includes Based on the detection result of the coin detection unit, each unit is controlled so as to allow the user to play with the present photo creation device, such as shooting, selection of the background image and foreground image, and graffiti for a predetermined time. Since the detection operation by the coin detection unit and the control operation by the first control unit 70 based on the detection result are the same as those of the conventional game photo creation device and are well known, detailed description thereof will be omitted.

  The second control unit 80 is a built-in editing unit 4 and corresponds to a control device that is configured around a computer including a CPU, a memory, a frame buffer, an auxiliary storage device, and the like. Is executed by the CPU to perform overall control related to the editing process. That is, the second control unit 80 functions as a GUI control unit that controls the second display / operation unit. In addition, the second control unit 80 adds a predetermined image to the graffiti target image based on an operation signal for performing graffiti processing on the graffiti target image (an image including a captured image) sent from the first control unit 70. A composite image that is an image in which is drawn is generated. When the generation of such a composite image is completed, an instruction to select a divided layout from the user is accepted, and information indicating the selected divided layout and the generated composite image are sent to the output unit 5. If the print output unit 92 is outputting another composite image, a message to that effect is displayed and sent after waiting for completion.

  The second display / operation units 81 and 82 correspond to the editing operation touch panel 400 functioning as a graffiti GUI display means, and accept user operations using a touch pen. The second communication unit 83 corresponds to a network adapter built in the editing unit 4 and functions as an interface for data transmission / reception via the network 6.

  The third control unit 90 is built in the output unit 5 and corresponds to a control device mainly composed of a computer including a CPU, a memory, a frame buffer, a timer, an auxiliary storage device, and the like, and is stored in an internal memory. When the CPU executes a predetermined program, overall control relating to output processing is performed. The third control unit 90 stores the composite image sent from the second control unit 80 in the memory as composite image data. The print output unit 92 corresponds to the network printer 35 built in the output unit, and prints the composite image data stored in the memory as a photo sticker (or photo card) (which is appropriately laid out). The printed photo sticker or the like is taken out from an outlet 33 provided at the lower front of the output unit 5.

  Further, the third control unit 90 starts printing processing such as a photo sticker based on the divided layout information and the composite image sent from the second control unit 80, and at the same time accepts a user input operation. An operation screen to be described later is displayed on the third display / operation unit 91. The third display / operation unit 91 corresponds to the output operation touch panel 30 and functions as an input unit. The output operation touch panel 30 includes a monitor that functions as a display unit such as a liquid crystal display or a CRT (Cathode Ray Tube), and a touch panel for one person that is stacked on the upper surface and can recognize input coordinates. The monitor can display an operation screen divided into a plurality of screens, and the touch panel accepts an input operation using a user's touch pen for each of the plurality of divided screens, and the received input operation is used as an operation signal as a third signal. To the control unit 90.

  The third communication unit 95 corresponds to a network adapter built in the output unit 5 and functions as an interface for data transmission / reception via the network 6.

  Here, the third control unit 90 writes the demo image (demonstration image) stored in advance in the auxiliary storage device in the frame buffer from the end of the graffiti until the operation is started. Displayed on the display / operation unit 91. The audio output unit 93 corresponds to the speaker 32. The audio output unit 93 explains the input operation method to the user in conjunction with the operation screen displayed on the third display / operation unit 91, and the demo image is displayed on the third display / operation unit 91. When playing, play music etc. according to the demo image. The description of the input operation method, music, and the like are stored in advance in a hard disk device as an auxiliary storage device.

  Thereafter, the third control unit 90 provides the user with a mini-game to be described later until printing of the composite image is completed by the display of the third display / operation unit 91 or the voice or sound effect of the voice output unit 93. to continue. The contents of this mini game are similarly stored in advance in a hard disk device or the like.

  Here, the predetermined program executed in each control device is provided by, for example, a DVD-ROM which is a recording medium on which the program is recorded. That is, a DVD-ROM as a recording medium for the predetermined program is attached to a DVD-ROM driving device built in the control device as an auxiliary storage device, and the predetermined program is read from the DVD-ROM as an auxiliary storage device. Installed on the hard disk drive. The predetermined program may be provided via a recording medium (CD-ROM or the like) other than a DVD-ROM or a communication line. Then, when an operation for starting the present photo creating device is performed, the predetermined program installed in the hard disk device is transferred to the memory in the control device and temporarily stored therein, It is executed by the CPU. Thereby, the control process of each part by the control device is realized.

  The first to third control units 70, 80, 90 have been described as corresponding to devices including different computers built in different units, but such a configuration is an example, and The first to third control units 70, 80, 90 may be realized by two or less devices or four or more devices. In this case, a program corresponding to the function to be realized is executed in each device. In addition, the photographing unit 3, the editing unit 4, and the output unit 5 may be configured by two or less units or four or more units.

<1.3 Processing procedure in the photo creation device for play>
As described above, the play photo creating apparatus includes the photographing unit 3, the editing unit 4, and the output unit 5. The photographing unit 3 performs photographing processing, the editing unit 4 performs later-described graffiti editing processing, and the output unit 5 performs output processing. It should be noted that when a certain user is playing with the photographing unit 3, another user can play with the editing unit 4, and yet another user can output a composite image with the output unit 5. ing. In other words, this game photo creation device can perform shooting processing, graffiti editing processing, and output processing in parallel. Below, the outline of the procedure of the graffiti editing process and the output process will be described last, focusing on the shooting process.

<1.3.1 Shooting process>
FIG. 7 is a flowchart showing the procedure of the photographing process in the present embodiment. When the play photo creation device is not used (when no play is performed), a demonstration image is displayed on the shooting operation touch panel 20. When the user inserts a coin into the coin insertion slot 26 while displaying the demo image, play is started (step S100).

  When the play is started, the first control unit 70 accepts selection of the shooting mode by the user (step S105). In step S110, for example, image quality (specifically, one of clear image quality with high contrast, soft image quality with softness, or cool image quality with transparency) is selected, brightness is selected, Whether to shoot automatically or manually is selected. When shooting automatically, a shooting theme is selected. In this case, the first control unit 70 displays on the shooting operation touch panel 20 a screen for allowing the user to select one or more shooting themes from a plurality of shooting themes prepared in advance. The selection operation by the person is accepted. Then, the first control unit 70 acquires selection information based on a user's selection operation, and determines a combination of a frame and a background to be used for shooting based on the selected shooting theme. When shooting manually, the user can freely determine the frame and background.

  Subsequently, in step S110, the first control unit 70 performs a process of deforming the background included in the frame image according to the position of the user (background image deformation process). Since this background image deformation process has a characteristic configuration, it will be described in detail later. In addition, the second background change processing that does not deform the background image but partially adds a new background image in accordance with the position of the user and partially erases the displayed background image is second. This embodiment will be described in detail. In the following description, for convenience, it is assumed that the frame image includes only the background image.

  Next, in step S120, the first control unit 70 performs a chroma key composition display process of an image to be graffiti and a frame image (here, a background image). This chroma key composition display process generally includes a process for creating a chroma key mask and a process for synthesizing a background image. The chroma key mask is, for example, a well-known composite mask (alpha channel), and an area having a key color (in this case, blue or green) in a target photographed image, that is, an area occupied by an image on the back of the photographing room 2. Is data for making transparent and making other areas non-transparent. In this composite mask, a value α indicating transparency of 0 (completely transparent) or 255 (completely non-transparent) is generally set for each pixel of the target image. By using such a chroma key mask, the background image region corresponding to the non-transparent region (α = 255) of the photographed image is hidden (masked), and the background image corresponding to the transparent region (α = 0). This area is synthesized so that it can be seen. Therefore, a composite image can be obtained as if a non-transparent area portion of the target captured image is cut out and the cut-out portion is pasted on the background image. The composite image obtained in this way is displayed on the shooting operation touch panel 20. The captured image here is not actually (high-resolution) still image data stored as a captured image, but through-image data for real-time display. This through image data is continuously acquired from the camera 10 (as a broadly captured image) from the start point to the end point of the entire shooting process (S100 to S170). With such chroma key processing, it is possible to create a photo with high playability. The background image is appropriately selected by the user before the start of shooting or during shooting.

  Thereafter, the process proceeds to step S125, where it is determined whether or not it is a shooting timing at which one set of three shootings should be performed. When the shooting timing is not reached (No in step S125), the background image is displayed until the shooting timing is reached. The deformation process and the chroma key composition display process are repeated (S125 → S110 → S120 → S125), and if it is the shooting timing (Yes in step S125), the process proceeds to the shooting process in step S130.

  In the shooting process in step S130, as described above, after a predetermined time of about several seconds has passed, a flash is emitted in the shooting direction of the camera 10, and the shot image data acquired by the camera 10 is stored in the first control unit 70. Stored in memory.

  Next, in step S140, a combined image obtained by chroma key combining the captured image obtained in step S130 with the background image obtained in step S110, that is, the same processing as that performed in step S120 (strictly, a low-resolution image). The composite image generated by the chroma key composition process (compositing the captured image, which is a high-resolution still image with the background image), instead of compositing the through image with the background image) is used as the graffiti target image. Is displayed. In the process of step S140, the same background image deformation process as the background image deformation process in step S110 is performed. The details of the background image deformation process here are detailed processes in step S110 described later. Instead of description, description is omitted here.

  Subsequently, in step S150, the first control unit 70 determines whether or not photographing of a predetermined number (set number) has been completed. As a result of the determination, if the number of images has been shot, the process proceeds to step S160. If the number of images has not been shot, the process returns to step S110. In practice, a time limit for photographing (for example, 3 minutes) is provided.

  In step S160, an image to be actually graffitied is selected (by the user) from a plurality of graffiti subject images. Specifically, the first control unit 70 displays a list of graffiti target images on the shooting operation touch panel 20 in order to allow the user to select an image to be used for graffiti and printing, and performs a selection operation by the user. Accept. Then, the first control unit 70 sends the image selected by the user to the second control unit 80 as an actual graffiti target image. When the process in step S160 ends, the process proceeds to step S170. In step S170, a guidance screen is displayed. Specifically, the first control unit 70 displays a screen for guiding the user to any one of the editing units 4 (4 a or 4 b) on the photographing operation touch panel 20. Thereby, the photographing process is completed. Note that the present embodiment has a feature in the contents of the background image deformation process in step S110 described above. This point will be described later with reference to FIG. 9, and the contents of the graffiti editing process will be described next.

<1.3.2 Graffiti editing process>
FIG. 8 is a flowchart showing the procedure of graffiti editing processing in the present embodiment. The graffiti editing process is realized by the second control unit 80 operating as shown in FIG. 8 based on a predetermined program. In this process, after the above-described shooting process is completed, the second control unit 80 acquires a graffiti target image sent from the first control unit 70 via the network (LAN) 6 (step S210).

  Next, the second control unit 80 receives from the first control unit 70 a plurality of graffiti target images acquired by the photographing process performed by the first control unit 70 and selected by the user (step 210). Subsequently, the timer 46 is set to a predetermined time (specifically, a time during which graffiti is permitted), and countdown is started (step S220).

  After the countdown of the timer 46 starts, a doodle editing operation screen is displayed on the editing operation touch panel 400 of each of the units 4a and 4b constituting the editing unit 4, and a graffiti operation (drawing operation) by the user is accepted (step S230). ~ S250). At the time of this graffiti editing process, the editing unit 4 may accept an editing operation for creating a material image such as a decomail image in addition to an editing operation on the graffiti target image based on the photographed image.

  Specifically, when one of the buttons and tools in the editing operation screen is touched with a touch pen, the coordinate value is input (S230), and the corresponding graffiti screen processing (S240) is performed. Thereafter, in step S250, the remaining time of the timer 46 becomes 0 (or the user performs an operation to end), and it is determined whether or not the graffiti has ended. If not (if No in step S250), Returning to step S230, the process is repeated until the graffiti ends. If the graffiti ends (Yes in step S250), the process proceeds to step S260.

  In step S260, the division pattern of the output photo is selected. Specifically, the second control unit 80 displays an operation screen on the editing operation touch panel 400 for causing the user to select one of a plurality of division patterns prepared in advance, and uses the operation screen. The selection operation by the person is accepted. And the 2nd control part 80 acquires selection information based on a user's selection operation. After the process of step S260 is completed, the process proceeds to step S270.

  In step S270, a guidance screen is displayed. Specifically, the second control unit 80 displays a screen for guiding the user to the output unit 5 on the editing operation touch panel 400. Thereby, the graffiti editing process ends. As described above, when the graffiti editing process is completed, the output unit 5 starts a printing process such as a photo sticker based on the composite image sent from the editing unit 4. Next, detailed contents of the background image deformation process in step S110 will be described with reference to FIG.

<1.3.3 Background Image Deformation Process>
FIG. 9 is a flowchart showing the procedure of the background image deformation process in the present embodiment. In step S400 shown in FIG. 9, the first control unit 70 performs a recognition process of a face area that is an image area occupied by a user's face image included in a captured image (strictly, a through image) (step S400). S400).

  FIG. 10 is a diagram simply showing a captured image. In the hatched portion shown in FIG. 10, an area occupied by an image on the back of the photographing room 2 with a single color (typically blue or green) for chroma key composition (hereinafter, this area is referred to as “ The background image is referred to as a “background area”, and a background image, which will be described later, is chromakeyed in this area. Further, an area other than the background area in the photographed image is an area occupied by the user's image as shown in FIG. 10 (hereinafter, this area is referred to as a “user area”). In step S400, processing for recognizing two areas occupied by the face images of the two users shown in FIG. 10 (hereinafter, these areas are referred to as “face areas”) is performed.

  Here, the recognition process of the face area may adopt a well-known method. For example, a template image is generated by enlarging / reducing and rotating a model image stored in advance, and the face image is used as a reference based on the template image. When an image having features is included in a captured image, a method for recognizing the region as a face region, a well-known recognition method that does not use a template, such as recognizing a face region based on the shape of a skin color region It is possible to adopt a recognition method using the pattern recognition method. Detailed description of these known methods is omitted. By executing the program for recognizing the face area in this way, the position of the user can be measured easily and inexpensively without providing a sensor or the like. If hardware such as a well-known face detection LSI chip is used, the recognition of the face area can be performed faster than in the case of software processing. As described above, the face area recognition process can be performed independently of the above-described chroma key composition process (S120), but the user area (the shape of the user area) obtained by performing the chroma key mask creation process in the chroma key process is described. Etc.) may be used to perform face area recognition processing.

  Next, in step S410, center coordinates that are the midpoints of the line segments connecting the centroids of the two face regions are calculated. FIG. 11 is a diagram for explaining the center coordinate calculation. The two face areas F1 and F2 shown in FIG. 11 have centroid points Pc1 and Pc2, respectively. Since the coordinates of the centroid points Pc1 and Pc2 in the captured image can be easily calculated based on a known centroid calculation method, the coordinates of the center coordinate Pm, which is the midpoint of the line segment connecting the centroid points Pc1 and Pc2. Can also be calculated easily.

  The reason for calculating the center coordinates is to determine the positional relationship of the user area in the captured image. That is, if the center coordinate is present near the center in the left-right direction of the captured image, it can be seen that the user is generally reflected near the center of the captured image, and if the center coordinate is on the left side or the right side of the center, , It can be seen that the user is biased to the left or right side of the captured image. Therefore, any calculation may be performed as long as the positional relationship in the captured image of the user can be determined. For example, the center of gravity position of the user area may be set as the central coordinate without performing the face recognition process. Further, even when face recognition processing is performed, it is not always necessary to calculate the center of gravity of the face area. For example, facial feature parts such as eyes and mouth may be recognized and the coordinates of the part may be calculated. For example, the coordinates of the uppermost part (the top of the head) and the lowermost part (the chin part) of the outer periphery of the face area may be calculated. Further, center coordinates or reference coordinates corresponding thereto may be calculated based on only one of the two face areas. For example, a user may be specified by face recognition processing, and one of the specified (here) two faces may be determined as a reference face, and reference coordinates such as center coordinates in the face area may be calculated. . Note that the reference face may be changed at every shooting or at a predetermined timing. Further, the reference face may be determined by the face size, for example, the larger face or the smaller face. The size of this face is used to calculate the amount of deformation (for image deformation) when the user moves in the front-rear direction (perspective direction) with respect to the camera as will be described later. This configuration is particularly suitable.

  However, in the configuration for calculating the positions of the plurality of users (typically, the center position) based on the process of recognizing the face areas of the plurality of users, a configuration in which the positions of the plurality of users are detected by sensors or the like. The position can be detected more accurately. This is because a plurality of users set their positions so that their faces are not hidden so that each of them can be surely captured in a photograph, and therefore each of the plurality of user's face areas may overlap in the captured image. Or at least not significantly overlap. For this reason, the position of the user is not erroneously calculated (or reduced) due to erroneous detection of the face area.

  In the above example, the center coordinates of the face areas of two users are calculated, but even if there are three or more persons, the center coordinates of the face areas of each user can be calculated in the same way. . When there is only one user, the center point Pc1 of the user's face area F1 is typically calculated as the center coordinate Pm. Further, instead of the center coordinate Pm, a point other than the midpoint on the line segment connecting the centroid points Pc1 and Pc2 (for example, a dividing point that divides the line segment at a predetermined ratio) may be used. Points outside the center of gravity Pc1 and Pc2 and having a predetermined positional relationship may be used. This is the same even when there are three or more users, and the same applies even when a point other than the center of gravity is used.

  Subsequently, in step S420, the deformation amount of the background image (used in step S420 to be described later) corresponding to the positional relationship in the photographed image of the user is calculated based on the center coordinates calculated in step S410. This amount of deformation is zero when the user is in the vicinity of the center of the photographed image, and the position where the user is photographed is more biased to the right than the center of the photographed image (the sign is in the positive direction). It is determined to become larger and to become smaller in the negative direction as the position where the user appears is more deviated leftward from the center of the captured image. If comprised in this way, a background image can be easily changed with the deformation amount according to the movement amount only by a user moving the position (to the left-right direction).

  Specifically, it is calculated based on the face area recognized by the face recognition process with the center coordinates of the face area of the user standing at the center position associated with the unmodified (original) background image as the reference position. The amount of deformation is calculated based on the amount of deviation of the center coordinate from the reference position. Note that the specific deformation amount (absolute value) takes into account how much the deformation of the background image according to the position of the user is suitable for giving the user a three-dimensional effect or a sense of reality. It is determined appropriately in advance.

  Subsequently, in step S430, image deformation processing for deforming a predetermined background image is performed according to the deformation amount calculated in step S420. The background image deformed by this process is displayed after the chroma key composition of the user image by the chroma key composition display process in step S120 described above. Hereinafter, the image deformation process will be described with reference to FIG. 12 to FIG.

<1.4 Screen configuration example>
FIG. 12 is a diagram illustrating a background image prepared in advance. As shown in FIG. 12, this background image is drawn so that a depth is felt in a direction perpendicular to the display surface. For example, when a person stands in front of this background image, the person is ) I feel like I'm in the room. However, in FIG. 12, since only a part of the side of the rectangular parallelepiped is drawn, nothing is drawn on the inner surface (visible) of the rectangular parallelepiped. However, a pattern indicating a room wallpaper, a graphic indicating a window, a graphic indicating furniture (three-dimensionally drawn), or the like may be drawn on the surface. Hereinafter, a figure showing a part of a rectangular parallelepiped for constituting a background as if in a room as described above is simply referred to as a “cuboid”. The background image is not limited to a rectangular parallelepiped shape, and may be a different polyhedron or sphere (hemisphere) shape, or a plurality of planes drawn so that different depths can be felt in the vertical direction with respect to the display surface. It may be. When the above-described user image (user area) shown in FIG. 10 is chromakey-combined so that the background image shown in FIG. 12 becomes the background, a composite image shown in FIG. 13 is obtained.

  FIG. 13 is a diagram illustrating a composite image obtained by combining the background image illustrated in FIG. 12 without deformation. As described above, the background image shown in FIG. 13 is not deformed by the image deformation processing in step S430. This is because the deformation amount calculated in step S420 is zero. That is, the position of the center coordinate Pm shown in FIG. 11 is located slightly to the right of the center line Lm that is the center line in the left-right direction of the image, but is located to the left of the center right end line Lmr. The center right end line Lmr and the center left end line Lml are provided at positions separated from the center line Lm by a predetermined short distance to the left and right, and the center coordinate Pm exists between the center right end line Lmr and the center left end line Lml. For example, the deformation amount is set to zero in step S420. With this configuration, when the user does not want to perform image deformation, the image deformation can be easily canceled by adjusting the center coordinate Pm (adjusting the standing position). Of course, as long as the center coordinate Pm is not on the center line Lm, the image may be deformed according to the amount of deviation. Further, the deformation amount may be changed stepwise every time the displacement amount changes by a predetermined amount instead of uniquely corresponding to the displacement amount. With this configuration, since the image is not continuously deformed, the change is lacking in smoothness, but the calculation load accompanying the image deformation can be reduced. Next, the case where the image is deformed by the user moving in the left direction of the image will be described with reference to FIGS.

  FIG. 14 is a diagram simply showing a captured image when the user moves to the left of the image. In FIG. 14, the part indicated by the dotted line represents the outer edge of the user image (user area) shown in FIG. 10, and the user moves by a distance ML1 to the left (toward the left) of the image. I understand that The center coordinate Pm at this time is the position shown in FIG.

  FIG. 15 is a diagram for explaining center coordinate calculation when the user moves to the left of the image. The calculation method of the center coordinate Pm shown in FIG. 15 is as described above as the processing contents of step S400 and step S410 with reference to FIG. Since the center coordinate Pm shown in FIG. 15 is further to the left of the center left end line Lml, the deformation amount is not zero, and an appropriate value is calculated by the processing in step S420 according to the distance from the center line Lm. .

  FIG. 16 is a diagram showing a background image obtained by transforming the background image shown in FIG. The background image (each side) shown in FIG. 12 is indicated by a dotted line in FIG. 16, and has a modified form in which the back side surface is moved in the right direction when viewing the image. If it deform | transforms in this way, it will feel as if the virtual viewpoint position with respect to the background image of the said rectangular parallelepiped moved to the right direction. Therefore, it is possible to generate a composite image having a stereoscopic effect or a sense of presence as if the camera 10 was moved in the right direction without moving the position of the camera 10 that is the actual viewpoint position.

  FIG. 17 is a diagram illustrating a combined image obtained by combining the background image illustrated in FIG. 16 and the user image illustrated in FIG. 14. As shown in FIG. 17, by transforming a three-dimensional background as if a virtual viewpoint has been moved and combining it with the user's image, the user can create a background image that changes due to his / her movement. It is possible to interactively generate a composite image according to the preference while watching. Therefore, more diverse playability can be obtained. Such image deformation will be described with reference to FIGS. 18 and 19.

  FIG. 18 is a diagram for explaining image deformation on the upper surface of a rectangular parallelepiped constituting the background, and FIG. 19 is a diagram for explaining image deformation on the left side of the rectangular parallelepiped constituting the background. The upper surface (top surface) of the background image shown in FIG. 18 is formed of a quadrilateral AECD. Compared with the quadrilateral ABCD indicating the upper surface (top surface) before image deformation, the vertex E is a distance Mr1 from the vertex B. It can be seen that the vertex F moves to the right by the same distance Mr1 from the vertex C, and the vertices A and D do not move.

  Similarly, the left side surface of the background image shown in FIG. 19 is composed of a quadrilateral AGJE. Compared with the quadrilateral AGHB indicating the left side surface before image deformation, the vertex E is on the right side by a distance Mr1 from the vertex B. The vertex J moves from the vertex H to the right by the same distance Mr1, and the vertices A and G do not move.

  If the positions of the vertices of the rectangular parallelepiped constituting the background are changed in this way, a background image having the above-described three-dimensional effect or presence can be generated with a simple calculation. Further, even when a pattern or a figure is drawn on each surface (each inner surface) of the rectangular parallelepiped, it can be easily deformed. That is, as can be seen from FIG. 18 and FIG. 19, the movement amount of each point included in the line segment connecting the moving vertices (for example, the line segment EF and the line segment EJ) is the same as the above-mentioned distance Mr1. Yes (and the direction of movement is the same), parallel to these line segments and parallel to these line segments (eg, line segment AD and line segment AG) connecting the vertices that do not move The movement amount of each point included in the infinite number of line segments increases, for example, in proportion to the distance away from the line segment connecting the above non-moving vertices in the vertical direction, so that after the movement of all points existing in each plane The coordinates of can be easily calculated. Since the calculation can be performed easily as described above, the deformation can be performed at high speed, and the composite image can be displayed at higher speed (smoothly).

  Also, here, the configuration has been described in which the background image is deformed along the direction opposite to the moving direction of the user (the left direction of the screen), but it may be deformed in the same direction as the moving direction. In this configuration, although depending on the moving amount of the image, it is possible to make it appear that the shooting direction of the camera 10 changes, and the playability can be further improved.

  Note that the above calculation method is a comparatively simplified example, and the back side of the rectangular parallelepiped that forms the background does not appear to translate left and right when the viewpoint is actually moved left and right. In other words, the inclination of the surface with respect to the viewpoint position changes. Therefore, although the calculation becomes more complicated, a background image with a more realistic or realistic feeling can be generated by performing such a calculation. Further, by using various well-known three-dimensional expression techniques such as ray tracing, a background image with a more realistic or realistic feeling can be generated.

  Further, here, the image is deformed to generate a composite image having a stereoscopic effect. However, when the actual viewpoint position changes, a part of the surface that has not been seen until now can be seen. Therefore, the background image may be changed not only to deform the image but also to add a graphic indicating the surface that was not visible. Next, a case where image deformation is performed by the user moving downward in the image will be described with reference to FIGS. 20 and 21. FIG.

  FIG. 20 is a diagram simply showing a captured image when the user moves downward in the image. In FIG. 20, the portion indicated by the dotted line represents the outer edge of the user image (user area) shown in FIG. 10, and it can be seen that the user has moved by a distance MD1 in the downward direction of the image. Even in such a movement mode, the calculation method of the center coordinates Pm and the deformation amount of the image can be calculated in the same manner as when the user moves to the left of the image.

  FIG. 21 is a diagram showing a composite image when the user moves downward as shown in FIG. As shown in FIG. 21, a three-dimensional background is transformed as if a virtual viewpoint has been moved and synthesized with a user's image to interactively generate a synthesized image according to the preference. It becomes possible.

  Similarly, a composite image can be generated when the user moves rightward or upward in the image. Furthermore, even when the user moves in an oblique direction such as the lower left direction of the image, a combined image can be easily generated by combining the above calculation methods. Next, a case where the background image is deformed by the user moving toward the front of the image (the direction approaching the camera) will be described with reference to FIGS. 22 and 23. FIG.

  FIG. 22 is a diagram simply showing a captured image when the user moves toward the front of the image. In FIG. 20, the part indicated by the dotted line represents the outer edge of the user's face image (face area) shown in FIG. 10, and the user moves in the troublesome direction of the image, that is, approaches the camera 10. You can see that it is moving. In the case of such a movement mode, the movement amount of the user cannot be calculated by the change of the center coordinate Pm. However, since the size of a person's face is not so different even if there is an individual difference, based on the average face size, the user's face size recognized by the face recognition process in step S400 shown in FIG. It is possible to associate the area of the face area with the position where the user stands in advance. Therefore, in step S410 shown in FIG. 9, instead of calculating the center coordinates, the area of the user's face area is calculated, and in step S420, the corresponding deformation amount is calculated from the calculated face area. it can.

  Specifically, the reference value of the area of the calculated face area, with the (general) area of the face area of the user standing at the position associated with the unmodified (original) background image as the reference value The amount of deformation is calculated on the basis of the amount of deviation with respect to. If there are multiple users, there are various methods such as obtaining the average value of the user's face area or using only the area of the user's face area near the center of the captured image. Conceivable.

  As a deformation mode of the background image here, for example, when the user approaches the camera, the background image is deformed so that it is arranged far away, and when the user moves away from the camera, the background image is arranged nearby. Deformation is possible. By transforming the background image in this way, if there is a user near the camera, it can appear as if the user is shooting with a wide-angle lens, and there is a user far away from the camera Can be seen as if the user is photographing with a telephoto lens. Further, with the configuration in which the background image is deformed as described above in accordance with the size of the face area, the user can easily deform the background image simply by moving the face closer to or away from the camera.

  FIG. 23 is a diagram illustrating a synthesized image obtained by synthesizing the background image that has been deformed so that the camera is disposed far away and the user image illustrated in FIG. 22. As shown in FIG. 23, a stereoscopic background is transformed from a virtual viewpoint as if it was photographed at a wide angle (or as if it was photographed at a telephoto position) and synthesized with a user image. It is possible to interactively generate a composite image according to the preference. In order to show the background image as if it is located far away, it is necessary to transform the image so as to reduce the inner surface of the rectangular parallelepiped, which is the background image, and to show the background image as if it is located nearby. In this case, the image may be deformed so as to enlarge the inner surface of the rectangular parallelepiped that is the background image.

  The background image deformation process as described above is performed in the same manner for the background image that should be included in the composite image to be graffiti as described above, but the description of the process content is omitted instead of the above description. .

  In addition, the background image deformation processing as described above can generate a background image having a three-dimensional effect or a sense of presence by simple calculation. For example, the virtual gaze direction is rotated around the position of the camera. When the virtual viewpoint position is rotated around the user's position, the image change mode is uncomfortable with simple image deformation. That is, in the image transformation, a new image is not added to the image before the transformation, but when the line-of-sight direction and the viewpoint position are rotated as described above, a new image that was not seen before the change is added. If it is not such a change mode, a sense of incongruity is produced. This may cause the same discomfort even if there is a difference in degree even when the virtual viewpoint position moves in the vertical and horizontal directions or in the front-rear direction as in the above embodiment. Therefore, in the second embodiment shown below, when the virtual line-of-sight direction is rotated (centered on the camera position), an example of generating a background image having a three-dimensional feeling or a sense of realism without a sense of incongruity is generated. Will be explained.

<2. Second Embodiment>
Since the basic configuration and operation of the automatic photo creation apparatus in this embodiment are the same as those in the first embodiment except for the background image deformation process described above, the same reference numerals are given and description thereof is omitted. To do. In the following, the content of the background image changing process in the present embodiment, which is different from the first embodiment, will be described.

  The background image change process according to the present embodiment is a configuration in which the background image is changed by adding a new background image instead of deforming the background image, and the face recognition process in the background image deformation process shown in FIG. The same processing as (S400) and center coordinate calculation (S410) is performed, and a change amount corresponding to the deformation amount is calculated by the same calculation as the deformation amount calculation (S420). Accordingly, in the second embodiment, the following image change process is performed instead of the image deformation process (S430) in the first embodiment.

  In this image change processing, a predetermined background image is not deformed, but a virtual image direction (or viewpoint position) is rotated so that a new image that was not seen before the change is added. The background image is changed (so that the image that was visible before the change is partially erased). Hereinafter, a detailed description will be given with reference to FIGS.

  FIG. 24 is a diagram showing a composite image that is synthesized without changing the background image as described above. The background image shown in FIG. 24 is a background image generated based on three-dimensional data (stereoscopic model data), and when the user stands at a reference position, that is, a change amount corresponding to the above-described deformation amount. This is a background image to be generated when the user stands at the center position where becomes zero. As shown in FIG. 24, this background image also feels as if the user is in a (cuboid) room, like the background image shown in FIG. 12 in the first embodiment. . As in the case of FIG. 12, even if a pattern showing a room wallpaper on the inner surface (visible) of the rectangular parallelepiped or a figure showing furniture three-dimensionally drawn is drawn on the surface. Good. However, unlike the case of FIG. 12, a quadrilateral representing a window having vertices W1, W2, W3, and W4 (hereinafter referred to as “quadrangle W”) is depicted. Specifically, the background image shown in FIG. 24 is generated based on three-dimensional data representing a rectangular parallelepiped, the virtual position of the user in the rectangular parallelepiped, and the virtual viewpoint position. This generation method will be described in detail with reference to FIG.

  FIG. 25 is a diagram simply showing a method for generating the background image shown in FIG. 24 based on the three-dimensional data. As shown in FIG. 25, the user is a rectangular parallelepiped having vertices C1, C2, C3, C4, B1, B2, B3, and B4, and is located near the center of the rectangular parallelepiped as a virtual three-dimensional background as described above. Given a virtual position like standing. And, when the user stands in a predetermined angle of view from the virtual viewpoint position Vp shown in the figure, among the rectangular parallelepipeds that should be the background, the rectangles made up of vertices P1, P2, P3, and P4 are shown. A rectangular parallelepiped as a background (specifically, a left side having vertices P1, P2, B2, and B1, a right side having vertices B4, B3, P3, and P4). The floor surface having vertices B2, P2, P3, B3, the top surface having vertices P1, B1, B4, and P4, and the back surface having vertices B1, B2, B3, and B4) and the quadrilateral drawn on the right side surface W or the like is generated as a background image. Here, the virtual viewpoint position Vp is located on a plane (here, the center of the plane for convenience of explanation) indicated by a rectangle made up of vertices C1, C2, C3, and C4 that are one side of the rectangular parallelepiped shown in FIG. It is stipulated to do. The viewpoint position Vp is determined so as to correspond to the actual arrangement position of the camera 10, and the angle of view is determined so as to coincide with the angle of view of the camera 10. Then, by combining with the user's image that is an actual photographed image, a sense of reality as if it actually exists as a background can be obtained.

  In FIG. 24, for convenience of explanation, the position where the user stands virtually is shown as being on the boundary position of the visible range determined by the angle of view. It is determined as appropriate so that a sense of reality can be obtained. Also, in FIG. 25, in order to show the shape of the rectangular parallelepiped in an easy-to-understand manner, the rectangular parallelepiped is shown as looking down from the upper right front, and is a portion displayed as a background image in a rectangular parallelepiped composed of a non-transparent wall (surface). And the side which is not hidden when it looks down from the upper right front as mentioned above is shown as the continuous line.

  Next, when the user moves in the right direction in the figure (left direction as viewed from the user), the background image changes according to the amount of movement, and as shown in FIG. A part of the background image that has not been seen until now (here, part of the right side of the rectangular parallelepiped) changes so as to newly appear. Hereinafter, a background image generation method in such a case will be described with reference to FIGS. 26 and 27. FIG.

  FIG. 26 is a diagram illustrating a combined image obtained by changing the background image as described above and combining the background images. The background image shown in FIG. 26 is generated when the user stands at a position moved by a predetermined distance in the right direction in the figure from the reference position in the vicinity of the position of the user image shown in FIG. It should be an image. As can be seen by comparing the background image shown in FIG. 26 with the background image shown in FIG. 24, the vertices P1, P2, which represent the left side of the rectangular parallelepiped located near the left end of the background image shown in FIG. The quadrilateral having B2 and B1 is erased as being invisible in the visible range in the background image shown in FIG. 26, and is a rectangular parallelepiped located near the right end of the background image that was not displayed in FIG. A part of the right side is newly displayed in FIG.

  When the background image changes in this way, the line-of-sight direction is rotated rightward (clockwise) around the virtual viewpoint position Vp by appropriately setting the amount of change according to the amount of movement of the user. It can be shown as if That is, although the camera 10 actually captures only the user (more precisely, it captures the background with a chroma key color for chroma key composition of the user and the background image). It is possible to generate a background image that feels as if the user is standing inside a rectangular parallelepiped room and the shooting direction of the camera 10 changes as the user moves to the right. A method for generating such a background image will be described in detail with reference to FIG.

  FIG. 27 is a diagram simply showing a method for generating the background image shown in FIG. 26 based on the three-dimensional data. As shown in FIG. 27, the user is given a virtual position such as standing in a position moved rightward from the vicinity of the center of the rectangular parallelepiped, which is a virtual solid background, and the virtual viewpoint position Vp. Of the rectangular parallelepiped serving as the background within the visible range indicated by the rectangle composed of the vertices P1, P2, P3, and P4 among the rectangular solids serving as the background when the user stands in a predetermined angle of view. Specifically, the right side surface having vertices B4, B3, P3, and P4, the floor surface having vertices B2, P2, P3, and B3, the top surface having vertices P1, B1, B4, and P4, and the vertices B1, B2, and B3 , B4) and the quadrilateral W drawn on the right side are generated as background images.

  Here, the virtual viewpoint position Vp shown in FIG. 27 is determined so as to correspond to the actual arrangement position of the camera 10 as described above with reference to FIG. Although it is determined to coincide with the angle of view, the line-of-sight direction (shooting direction) shown in FIG. 27 rotates clockwise by a predetermined angle from the line-of-sight direction shown in FIG. For this reason, the background image that should be visible within the visible range changes. In this way, a background image that should be visible within a visible range at a predetermined angle of view from the virtual viewpoint position Vp can be generated by a known calculation method based on the three-dimensional data from the rotated line-of-sight direction. Then, by synthesizing the generated background image and the user's image that is the actual captured image, the shooting direction of the camera 10 changes (rotates) and the background changes according to the movement of the user. Because it can be seen, you can get a sense of realism as if it actually existed as a background.

  Here, on the inner surface of the rectangular parallelepiped that is a three-dimensional background, as described above, an image such as a pattern imitating wallpaper or a picture of furniture is pasted (for example, using a texture mapping technique). The quadrilateral W representing the window shown in FIG. 27 is not a part of the right side of the rectangular parallelepiped, but is handled as an opening thereof, and an image showing a predetermined landscape is pasted as if the scenery can be seen outside the window. It has a configuration that can be. That is, an image of a landscape that exists (virtually) in the distance is prepared in advance, and the image portion of the landscape that should be seen within the quadrilateral W showing the window in a predetermined viewing direction from the virtual viewpoint position Vp is calculated. If it is calculated and displayed while recalculating the image portion in accordance with the change in the line-of-sight direction, it is possible to obtain a sense of presence as if the background actually exists as if the scenery is visible from the window.

  In this embodiment, the example in which the line-of-sight direction rotates clockwise in the left direction in the figure is described, but the same is true regardless of how the line-of-sight direction rotates, such as the right direction, the vertical direction, and the front-back direction. The rotational position does not have to be the virtual viewpoint position Vp as the central axis, and the visual line direction is changed by rotating about an appropriate position such as a position virtually determined as a position where the user stands. May be. By changing the rotation axis in this way, the appearance can be changed. Furthermore, as described above, the known stereoscopic display is not necessary even when the viewpoint position moves in the left-right direction, the up-down direction, and the front-rear direction as in the first embodiment, instead of rotating the line-of-sight direction. It can be displayed in a similar manner by a technique.

  Next, a modification of this embodiment will be described. In the following, for convenience of explanation, a description will be given as a modification of the first embodiment, but the same configuration can be applied as a modification of the second embodiment.

<3. Modification>
<3.1 First Modification>
In each of the above embodiments, the background image is deformed in a deformation amount and a deformation direction according to the recognized user face position by recognizing the position of the user's face area in the captured image by face recognition processing. However, the deformation of the background image may be performed not according to the position of the face area of the user but according to the orientation. Hereinafter, a description will be given with reference to FIGS.

  FIG. 28 is a diagram showing a composite image when the user's face is facing forward, and FIG. 29 is a diagram showing a composite image when the user's face is facing right. It is. As shown in FIG. 28, since the user's face included in this composite image is facing forward, the background image is not deformed as in the composite image shown in FIG. The orientation of the user's face can be determined by detecting a well-known face recognition process, for example, which of the eye positions the face is biased toward. Here, when the user turns his face to the right (left direction in the figure), it is determined based on the face recognition process that the user's face has turned to the right, and the background image is the same as in FIG. It becomes a deformation | transformation aspect that the surface of the back side moved to the right direction. The composite image obtained in this way is as shown in FIG. If it deform | transforms in this way, it will feel as if the virtual viewpoint position with respect to the background image of the said rectangular parallelepiped moved to the right direction. Therefore, it is possible to generate a composite image having a stereoscopic effect or a sense of presence as if the camera 10 was moved in the right direction without moving the position of the camera 10 that is the actual viewpoint position.

  Note that the amount of deformation at this time may be determined according to the angle of the face (to the left and right), but for example, if the user keeps the face facing leftward, a predetermined constant deformation The state may be such that the background image continues to be deformed by the amount, and the deformation stops when the user returns the face to the front. By doing so, the user can shoot with the face facing forward, and can easily deform the background image in a desired direction by a desired deformation amount.

  In this case, the background image may be changed so as to move (scroll) in a predetermined direction, instead of deforming the background image. That is, a part of the background image prepared in advance may be displayed, and the background image may be changed so that the part of the background image that should be displayed in the direction corresponding to the orientation of the user's face is changed. . Such a change aspect is applicable also in each said embodiment.

  Further, by applying the first modification to the second embodiment described above, for example, when rotation of the user's face is detected, the line-of-sight direction is rotated around the viewpoint position Vp (see FIG. 27), when movement of the face in the up / down / left / right or front / rear direction is detected, various configurations such as moving the viewpoint position Vp in the direction corresponding to the movement direction without changing the line-of-sight direction are conceivable. Each embodiment and each modification can be appropriately combined as described above.

<3.2 Second Modification>
In the first embodiment, the background image is deformed according to the position of the user. However, the foreground image may be further changed (here, the position is moved). Hereinafter, a description will be given with reference to FIGS. 30 and 31. FIG.

  FIG. 30 is a diagram illustrating a composite image that is synthesized without changing the background image and without moving the foreground image, and FIG. 31 is a composite image that is synthesized by deforming the background image and moving the foreground image. FIG. As shown in FIG. 30, the rectangular parallelepiped background image showing a part of the room shown in FIG. 31 is deformed in the direction opposite to the moving direction of the user as in the first embodiment. . In addition, the foreground image of the shape showing the fence shown in FIG. 31 moves in the same direction as the movement direction of the user, as can be seen from comparison with FIG.

  When the foreground image is moved in this way, it is felt as if the virtual viewpoint position with respect to the background image and the foreground image has moved in the right direction in combination with the deformation of the background image. Therefore, it is possible to generate a composite image having a stereoscopic effect or a sense of presence as if the camera 10 was moved in the right direction without moving the position of the camera 10 that is the actual viewpoint position. It should be noted that the foreground image only needs to move in such a way as to give the above-described stereoscopic effect or realism, and may be configured to be deformed in the same direction or in the opposite direction like the background image. In addition, the foreground image may have a shape that allows the depth to be felt, as in the case of the background image, or may be a plurality of planes that are drawn so that different depths can be felt in the vertical direction with respect to the display surface. There may be. Furthermore, a configuration in which only the foreground image is deformed may be used instead of the configuration in which both the background image and the foreground image are changed.

  In addition, the foreground image may be configured to be enlarged or reduced according to the position of the user when the user moves in the front-rear direction (that is, in the direction of approaching or moving away from the camera). It can be considered that the virtual position of the user is in front of the virtual position of the foreground image when viewed from the virtual viewpoint position. In this case, the foreground image may be handled as a background image (so as to be hidden by the user's image).

<3.3 Third Modification>
In the first embodiment, the background image is deformed according to the position of the user. In the first modification, the background image is deformed according to the orientation of the user (face). However, a combination with a configuration in which the background image is deformed according to the arrangement position or orientation (shooting direction) of the camera may be used.

  That is, the camera 10 in the first (or second) embodiment is fixed at a predetermined position of the photographing unit 3, but instead of the camera 10, a camera that can change at least one of the arrangement position and the orientation. And a sensor for detecting the arrangement position or orientation thereof. Specifically, a camera attached to the inner surface of the photographing unit 3 by a mounting tool such as a movable arm, and a camera (and its gripping part) having a size and shape that can be gripped by a user are provided, and a joint in the mounting tool is provided. A configuration in which a sensor for measuring the angle of the portion or the distance to the inner wall of the photographing unit 3 is provided may be considered.

  In addition, a plurality of cameras mounted at different positions, for example, a whole-body camera for photographing the whole body of the user and an up-shooting camera for enlarging a part of the camera are provided. A configuration in which the background image is deformed according to a change in at least one of the shooting direction and the arrangement position may be employed.

  Furthermore, when the camera position (or orientation) changes, it is assumed that the user is located in the center of the camera shooting range, and the camera position (or orientation) changes in the opposite direction to the camera position (or orientation). When the background image is scrolled in the direction, it is possible to feel a three-dimensional effect or a sense of reality in the change of the background image.

  By changing the background image as in the first embodiment or changing the background image (such as in a scroll-like form) according to one of the camera position and orientation changing in this way A person can sufficiently obtain playability such as a three-dimensional feeling and a sense of reality.

<3.4 Other Modifications>
In each of the above-described embodiments and the first and second modified examples, a rectangular parallelepiped background image has been described as an example. However, any shape, pattern, color, or the like may be used as long as the background image has a sense of depth. The depth may be felt by combining a plurality of background images. For example, a plane image depicting a distant mountain is included as a first background image, and a plane image depicting a tree or a building (virtually) placed between the depicted mountain picture and the user. In the configuration including the second background image, the first background image does not change the arrangement position (composite position) even if the position of the user (the virtual viewpoint corresponding thereto) changes. If the arrangement position of the background image is changed according to the position of the user, the second background image can be felt in front of the first background image, so that the depth of the entire background image can be felt. Become. Further, instead of the above planar image, the shape is changed according to a change in (virtual) viewpoint based on data describing a three-dimensional structure similar to a rectangular parallelepiped as in the second embodiment or other three-dimensional structures. A three-dimensional image drawn so as to change may be used as the background image.

<4. Effect>
As described above, the automatic photo creating apparatus according to the first or second embodiment or each of the modified examples thereof (hereinafter simply referred to as “the present embodiment”) is the user's position or orientation (and the modified example described above). The position or orientation of the camera as shown in FIG. 5 is detected, a predetermined background image is changed based on the detected location or orientation, and is controlled to be displayed together with the user's image, and should be included in the composite image Based on the position or orientation detected when the captured image is acquired, a composite image obtained by combining the changed background image with the captured image is output as a photo sticker. In the seal, the user can sufficiently obtain playability such as a three-dimensional effect and a sense of reality.

  According to the present embodiment, since the position or orientation of the user is detected by recognizing the user's face area included in the captured image, the position of the user can be easily and inexpensively provided without providing a sensor or the like. Can be measured.

  Furthermore, according to the present embodiment, when there are a plurality of users, one position corresponding to the plurality of users is detected by recognizing each face area of the user included in the captured image. Thus, the position can be detected more accurately than the configuration of detecting the position of each of a plurality of users. In other words, multiple users set their own positions so that their faces are not hidden so that they can be reliably captured in a photo. Since the position of the person is not erroneously calculated, it can be accurately detected.

  Further, according to the present embodiment, since the position of the user is detected based on the shift amount and the shift direction of the position (center position) of the face area in the captured image with respect to the reference position, the user sets the position. By simply moving (for example, in the left-right direction) and moving the position of the face area, the background image can be easily deformed with a deformation amount corresponding to the movement amount.

  Furthermore, according to the present embodiment, since the position of the user is detected based on the amount of deviation of the size of the face area in the captured image with respect to the reference size, the user can change the position (here, The background image can be easily deformed with the amount of deformation corresponding to the amount of movement by simply moving the position of the face region by moving in the front-rear direction.

  In addition, according to the present embodiment, the background image is changed in the direction substantially the same as or substantially opposite to the direction of change of the user's position, so that the virtual viewpoint changes even when the camera position is fixed. As a result, it is possible to generate a composite image having a three-dimensional feeling or a sense of presence as if the camera was moved.

  Furthermore, according to the present embodiment, at least a part of a background image having a shape such as a rectangular parallelepiped that is formed in advance so as to feel the depth is deformed in a direction substantially the same as or substantially opposite to the direction of change of the user's position. Therefore, it is possible to generate a composite image having a stereoscopic effect or a realistic sensation only by simple deformation processing of a figure.

  According to the third modification of the present embodiment, the camera is configured to be movable, the position or orientation is detected, and the background image changes based on the position or orientation, so that the camera is used according to the movement of the camera. A person can sufficiently obtain playability such as a three-dimensional feeling and a sense of reality.

3 ... Shooting unit 4 ... Editing unit 5 ... Output unit 6 ... Network (LAN)
DESCRIPTION OF SYMBOLS 10 ... Camera 20 ... Touch panel for imaging operation 30 ... Touch panel for output operation 35 ... Network printer 31 ... Infrared port 32 ... Speaker 70 ... First control unit 72 ... First display / operation unit (touch panel for shooting operation)
80 ... second control unit 81, 82 ... second display / operation unit (touch panel for editing operation)
90 ... third control unit 91 ... third display / operation unit (touch panel for output operation)
92 ... Print output section (network printer)
94 ... Non-contact communication part (infrared port)
400 ... Touch panel for editing operation F1, F2 ... Face area Pm ... Center coordinates

Claims (10)

An automatic photo creation device comprising a photographing means for photographing a user and an output means for outputting a composite image including a photographed image acquired by the photographing means as a photograph,
Detecting means for detecting the position or orientation of the user;
Display means;
Based on the position or orientation detected by the detection means, a background image predetermined to be a background with respect to the image of the user included in the captured image continuously acquired by the imaging means is changed. , Combined display control means for continuously combining and displaying on the display means together with the image of the user;
When the captured image to be included in the composite image is acquired, the background image that has been changed in the same manner as the change by the composite display control unit is acquired by the image capturing unit based on the position or orientation detected by the detection unit. An automatic photo creation apparatus, comprising: composite image generation means for generating a composite image combined with the captured image.   The automatic photo creation apparatus according to claim 1, wherein the detection unit detects the position or orientation of the user by recognizing the user's face area included in the captured image.   In the case where there are a plurality of users, the detecting means detects one position corresponding to the plurality of users by recognizing each face area of the user included in the captured image. The automatic photo creating apparatus according to claim 2.   The said detection means detects the said user's position based on the deviation | shift amount and deviation | shift direction of the position of the said face area in the said picked-up image with respect to a predetermined | prescribed reference position, The said user position is characterized by the above-mentioned. 3. The automatic photo creation apparatus according to 3.   The said detection means detects the position of the said user based on the deviation | shift amount of the magnitude | size of the said face area in the said picked-up image with respect to the magnitude | size used as a predetermined | prescribed reference | standard. Item 5. The automatic photo creation apparatus according to any one of items 1 to 4.   The composite display control means changes the background image in a direction substantially the same as or substantially opposite to a change direction of the position of the user detected by the detection means so that the background image can feel a depth. The automatic photo creation apparatus according to claim 2, wherein:   7. The composite display control unit according to claim 6, wherein at least a part of the background image formed in advance so as to feel a depth is deformed in a direction substantially the same as or substantially opposite to the change direction. The automatic photo creation device described. The photographing means is configured to be movable to a desired position in the vicinity by the user,
The detecting means detects the position or orientation of the photographing means;
2. The automatic photo creation apparatus according to claim 1, wherein the composite display control means changes the background image based on the position or orientation of the photographing means detected by the detection means. An automatic photo creation method,
A shooting step of shooting the user by the imaging means;
An output step of outputting a composite image including a photographed image acquired in the photographing step as a photograph;
A detecting step for detecting the position or orientation of the user;
A display step;
Based on the position or orientation detected in the detection step, a background image predetermined to be a background with respect to the user image included in the captured image continuously acquired in the imaging step is changed. A combined display control step for continuously combining and displaying in the display step together with the image of the user;
At the time when a captured image to be included in the composite image is acquired, a background image that is changed in the same manner as the change in the composite display control step is acquired in the capturing step based on the position or orientation detected in the detection step. And a synthesized image generating step of generating a synthesized image synthesized with the photographed image. On the computer,
A shooting step of shooting the user by the imaging means;
An output step of outputting a composite image including a photographed image acquired in the photographing step as a photograph;
A detecting step for detecting the position or orientation of the user;
A display step;
Based on the position or orientation detected in the detection step, a background image predetermined to be a background with respect to the user image included in the captured image continuously acquired in the imaging step is changed. A combined display control step for continuously combining and displaying in the display step together with the image of the user;
At the time when a captured image to be included in the composite image is acquired, a background image that is changed in the same manner as the change in the composite display control step is acquired in the capturing step based on the position or orientation detected in the detection step. And a synthesized image generating step of generating a synthesized image synthesized with the photographed image.

Images