JP6974573B1 - Information processing program, terminal device and information processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing program, a terminal device and an information processing method for facilitating shooting of a multi-viewpoint image. An information processing program includes an acquisition procedure for acquiring shooting target information regarding a shooting target, and a providing procedure for providing a model indicating a shooting position and a shooting direction for shooting a multi-viewpoint image of the shooting target. , Is executed by the terminal device. [Selection diagram] FIG. 14

Description

The present invention relates to an information processing program, a terminal device and an information processing method.

Conventionally, there has been provided a technique for generating a multi-viewpoint image (free-viewpoint image) which is a plurality of images of a subject (shooting target) captured from different viewpoints (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2016-119513

However, it may not be possible to say that the above-mentioned conventional technique facilitates shooting of a multi-viewpoint image. For example, in the above-mentioned conventional technology, although the user is made to utilize the multi-viewpoint image, for example, when the user himself / herself takes a picture, it becomes unclear how to take a picture, and the multi-viewpoint image occurs. It can be difficult to shoot about.

The present application has been made in view of the above, and an object thereof is to facilitate shooting of a multi-viewpoint image.

The information processing program according to the present application includes an acquisition procedure for acquiring shooting target information regarding a shooting target, a provision procedure for providing a model indicating a shooting position and a shooting direction for shooting a multi-viewpoint image of the shooting target, and a provision procedure. To the terminal device.

According to one aspect of the embodiment, it is possible to facilitate taking a picture of a multi-viewpoint image.

FIG. 1 is a diagram showing an example of processing by the information processing system according to the embodiment. FIG. 2 is a diagram showing an example of photography using the terminal device according to the embodiment. FIG. 3 is a diagram showing an example of a gazing point according to an embodiment. FIG. 4 is a diagram showing an example of shooting for generating a multi-viewpoint image according to an embodiment. FIG. 5 is a diagram showing an example of a display at the time of shooting according to the embodiment. FIG. 6 is a diagram showing an example of the arrangement of the 3D model according to the embodiment. FIG. 7 is a diagram showing an example of display of the 3D model according to the embodiment. FIG. 8 is a diagram illustrating an example of a functional configuration of the terminal device according to the embodiment. FIG. 9 is a diagram showing an example of a functional configuration realized by the first application according to the embodiment. FIG. 10 is a diagram showing an example of a functional configuration realized by the second application according to the embodiment. FIG. 11 is a diagram showing an example of a transaction target information storage unit according to an embodiment. FIG. 12 is a diagram showing a configuration example of the service providing device according to the embodiment. FIG. 13 is a flowchart showing an example of the flow of transmission processing according to the embodiment. FIG. 14 is a flowchart showing an example of the flow of information processing according to the embodiment. FIG. 15 is a diagram showing an example of processing by the information processing system according to the embodiment. FIG. 16 is a diagram showing an example of displaying a list of transaction targets according to the embodiment. FIG. 17 is a diagram showing an example of displaying an image group including a multi-viewpoint image according to an embodiment. FIG. 18 is a flowchart showing an example of the flow of display processing according to the embodiment. FIG. 19 is a diagram showing an example of a hardware configuration.

Hereinafter, embodiments for implementing the information processing program, terminal device, and information processing method according to the present application (hereinafter, referred to as “embodiments”) will be described in detail with reference to the drawings. The information processing program, terminal device, and information processing method according to the present application are not limited by this embodiment. Further, in each of the following embodiments, the same parts are designated by the same reference numerals, and duplicate description will be omitted.

[Embodiment]
[1. Information processing]
Hereinafter, an example of information processing according to the embodiment will be described with reference to FIG. 1. FIG. 1 is a diagram showing an example of processing by the information processing system according to the embodiment. In the information processing system 1, a multi-viewpoint image (free-viewpoint image), which is a plurality of images obtained by capturing a subject (hereinafter, also referred to as a “shooting target”) from different viewpoints by the terminal device 10, can be captured by the terminal device 10. Display images. In the following, a transaction target traded in the electronic commerce service provided by the information processing system 1 will be described as an example of a shooting target. The transaction target here is the use of goods or services (services). The transaction target may be any target as long as it is a target to be traded in the information processing system 1.

[1-1. Information processing system configuration]
First, the configuration of the information processing system 1 will be described. As shown in FIG. 1, the information processing system 1 includes a terminal device 10 and a service providing device 100. The terminal device 10 and the service providing device 100 are connected to each other via a predetermined communication network (network N or the like in FIG. 8) so as to be communicable by wire or wirelessly. The information processing system 1 shown in FIG. 1 may include a plurality of terminal devices 10 and a plurality of service providing devices 100.

The service providing device 100 is a server device (information processing device) that accepts submission of information based on an image taken by the terminal device 10. For example, the service providing device 100 accepts submission of a multi-viewpoint image taken by the terminal device 10. The service providing device 100 provides a service for distributing information. For example, the service providing device 100 provides a service for distributing information on a transaction target. For example, the service providing device 100 provides a service related to buying and selling of a transaction target, such as listing of a transaction target.

For example, the service providing device 100 is an information processing device that provides an electronic commerce service to a user who uses the terminal device 10. The service providing device 100 distributes (provides) various information related to the electronic commerce service to the terminal device 10. The following example shows a case where the service providing device 100 provides an auction service which is an example of an electronic commerce service. For example, the service providing device 100 provides a transaction target listing service and accepts a transaction target listing from a user (exhibitor). Further, the service providing device 100 accepts a user's bid for a transaction target that has been put up for sale. The service providing device 100 is not limited to the auction service, and may provide various services for trading a transaction target, such as a shopping service and a free market service.

Further, since the electronic commerce service such as the auction service provided by the service providing device 100 is the same as the normal electronic commerce service except that the multi-viewpoint image is registered and distributed, the points related to the electronic commerce service will be appropriately described. Is omitted. The service provided by the service providing device 100 is not limited to the electronic commerce service, for example, a navigation service, an electronic payment service, a search service, a news distribution service, a weather forecast service, a finance service, a route search service, a map providing service, and a travel service. It may be a service that distributes various information such as a service, a restaurant introduction service, a facility reservation service, an SNS (Social Networking Service) service, and a web blog service.

The terminal device 10 is a device (computer) used by the user. In the example of FIG. 1, the terminal device 10 shows a case where the terminal device 10 is a smartphone used by the user U1. The terminal device 10 accepts operations by the user. The terminal device 10 has an image sensor (hereinafter, also referred to as “camera”) for capturing an image. For example, the terminal device 10 has a camera 70, which is an out-camera that functions as an image sensor. The terminal device 10 captures an image with the camera 70. The terminal device 10 is not limited to smartphones, and can be applied to various types such as tablet terminals, mobile phones, PDAs (Personal Digital Assistants), notebook PCs (Personal Computers), etc., as long as they can be applied to the processes shown below. It may be a terminal device of. In the following description, the terminal device 10 may be identified with the user. That is, in the following, the terminal device 10 may be read as a user and the user may be read as a terminal device 10 as necessary.

[1-2. An example of information processing]
From here, the process shown in FIG. 1 will be described. For example, the terminal device 10 activates an application (for example, the first application 81 or the like) for transmitting information about a transaction target to the service providing device 100, and executes the following processing. First, the terminal device 10 is a multi-viewpoint image FV1 with nine images in the order of _{images RP1 1} , RP1 ₂ , RP1 ₃ , RP1 ₄ , RP1 ₅ , RP1 ₆ , RP1 ₇ , RP1 ₈ and RP1 _9. (Step S11). In the example of FIG. 1, the terminal device 10 acquires a multi-viewpoint image FV1 in which a sneaker (hereinafter referred to as “shoe A”) to be traded is captured from nine different viewpoints. _{In FIG. 1, among the nine images of the images RP1 1 to RP1 9} constituting the multi-viewpoint image FV1, the first image RP1 ₁ , the intermediate image RP1 ₅ , and the last image RP1 _{9 are shown.} Only the three images of are shown. The multi-viewpoint image FV1 is a multi-viewpoint image taken from the left front toe to the left back heel via the left side surface.

In the example of FIG. 1, the terminal device 10 photographs the shoe A (photographing target SB1), which is a transaction target, and generates a multi-viewpoint image FV1, which will be described later. The terminal device 10 acquires the multi-viewpoint image FV1 by photographing the shoe A and generating the multi-viewpoint image FV1.

Then, the terminal device 10 transmits the multi-viewpoint image FV1 obtained by photographing the shoe A to the service providing device 100 together with the information related to the shoe A (hereinafter, also referred to as “target-related information”) (step S12). In the example of FIG. 1, the terminal device 10 transmits the exhibition information for listing the shoe A in the auction as the target related information INF1 to the service providing device 100 together with the multi-viewpoint image FV1 in which the shoe A is photographed. For example, the target-related information INF1 includes character information explaining the contents of the shoe A. In addition, the target-related information INF1 includes information related to the auction such as the starting price of the shoe A, the lowest winning bid price, and the bidding period.

The service providing device 100 that has received the multi-viewpoint image FV1 of the shoe A and the target-related information INF1 of the shoe A from the terminal device 10 registers the multi-viewpoint image FV1 and the target-related information INF1 in association with the shoe A (step). S13). In the example of FIG. 1, the service providing device 100 adds the shoe A as an exhibit for the auction, and manages the shoe A in association with the multi-viewpoint image FV1 and the target-related information INF1.

The terminal device 10 transmits the multi-viewpoint image of the transaction target together with the target-related information related to the transaction target to the service providing device 100, so that the service providing device 100 registers the listing of the transaction target from the terminal device 10. , The multi-viewpoint image of the transaction target can be delivered. Therefore, the terminal device 10 can promote the provision of services using the multi-viewpoint image. As a result, the information processing system 1 can display a multi-viewpoint image as an image of a transaction target in an auction, flea market, or the like.

[1-3. Shooting target]
From here, a case where a user takes a picture in order to generate a multi-viewpoint image FV1 will be described. First, photography by the user will be described with reference to FIG. FIG. 2 is a diagram showing an example of photography using the terminal device according to the embodiment. In the following description, a case where the shoe A is photographed by the user U1 as the imaging target SB1 will be described as an example.

As shown in FIG. 2, the user U1 points the camera 70 provided in the terminal device 10 toward the shooting target SB1 and shoots the shooting target SB1. In FIG. 2, the terminal device 10 directs the line of sight of the camera 70 (referred to as “line of sight VL”) from the viewpoint position on the left side of the image target SB1 (referred to as “viewpoint VP”) toward the image target SB1. To shoot.

Further, the terminal device 10 determines the gazing point based on the image taken by the camera 70. The terminal device 10 determines the gazing point based on the line-of-sight VL from the viewpoint VP. The terminal device 10 determines the gazing point from two viewpoints based on the line-of-sight information. This point will be described with reference to FIG. FIG. 3 is a diagram showing an example of a gazing point according to an embodiment.

In FIG. 3, the terminal device 10 used by the user U1 is described as "terminal device 10 _a " and "terminal device 10 _b " according to the difference in the shooting position, but the description is made without particular distinction. , Terminal device 10. Similarly, the camera 70 is described as "camera 70 _a " and "camera 70 _b " according to the difference in the shooting position, but when the description is made without particular distinction, the camera 70 is described.

In Figure 3, the user U1 executes a shooting imaging target SB1 toward the first position the camera 70 _a of the terminal device 10 _a to the imaging target SB1. Thus, the terminal device 10 photographs the photographing target SB1 toward obliquely from the front and left of the viewpoint VP _a gaze VL _a camera 70 _a shooting subject SB1 to be imaged SB1.

The user U1 carries a shooting imaging target SB1 toward the second position the camera 70 _b of the terminal device 10 _b to the imaging target SB1. As a result, the terminal device 10 shoots the shooting target SB1 by directing the line-of-sight VL _b of the camera 70 _{b from the viewpoint VP b} on the left side of the shooting target SB1 toward the shooting target SB1.

The terminal device 10 determines the gazing point using the information acquired by photographing from the two viewpoints of the first position and the second position. The terminal device 10 calculates the three-dimensional position (hereinafter, simply referred to as “position”) of the gazing point using the information acquired by photographing from the two viewpoints of the first position and the second position. The terminal device 10 estimates the distance between the image shooting target SB1 and the camera 70 by using the information acquired by photographing from two viewpoints. Terminal device 10, line of sight VL _a from the viewpoint VP _a, and determining the gaze point based on the line-of-sight VL _b from the viewpoint VP _b.

For example, the terminal device 10 determines the position (three-dimensional point) at _{which the line-of-sight VL a} and the line-of-sight VL _{b intersect at the gazing point GP1.} In this way, the terminal device 10 calculates the gaze point GP1 based on the _{line-of-sight VL a} and the line-of-sight VL _b. The above is only an example, and the terminal device 10 may determine the position where the line-of-sight VL _a and the line-of-sight VL _b are close to each other in the gazing point GP1 _{, for example, when the line-of-sight VL a} and the line-of-sight VL _{b do not intersect.} .. For example, FIG. 3 shows a plan view for the sake of explanation, but in a three-dimensional space, when the line of sight VL _a and the line of sight VL _b are offset in the height direction and do not intersect, the line of sight VL _a and the line of sight VL _{b b.} The midpoint of the position closest to and may be determined as the gazing point GP1. As described above, the terminal device 10 may determine the gazing point by any method as long as the gazing point can be determined.

Further, the determination of the gazing point is not limited to the case where the gazing point is determined by shooting from two viewpoints as described above. For example, the terminal device 10 may be automatically set by appropriately using techniques such as identification of an image pickup target and identification of depth. Further, the terminal device 10 may determine the gazing point by, for example, taking a picture from one viewpoint. For example, the terminal device 10 may recognize (calculate) the gaze point (position) by recognizing the ground plane to be photographed. For example, the terminal device 10 may determine the gaze point using the gaze point determination logic provided by the external device. The line-of-sight VL may be a straight line along the optical axis of the camera 70.

The terminal device 10 determines a plurality of positions (viewpoints) for taking a picture of the multi-viewpoint image based on the gazing point GP1. This point will be described with reference to FIG. FIG. 4 is a diagram showing an example of shooting for generating a multi-viewpoint image according to an embodiment.

For example, the terminal device 10 determines (calculates) the _{viewpoints VP 1 to VP 9} based on the set shooting range and the number of shooting positions. For example, the terminal device 10 calculates _{nine viewpoints VP 1 to} VP 9 using the information on the estimated distance between the shooting target SB1 and the camera 70. In the example of FIG. 4, the terminal device 10 determines the _{viewpoints VP 1 to VP 9} based on the set shooting range (120 °) and the number of shooting positions (9). For example, the terminal device 10 is arranged at intervals of 15 ° around _{the viewpoint VP 5} , and determines _{nine viewpoints VP 1 to VP 9 in which each line of sight VL 1 to VL 9} passes through the viewpoint GP 1. The terminal device 10 appropriately uses existing three-dimensional spatial information processing technology and image processing technology to perform processing such as the above-mentioned gaze point and viewpoint determination.

The above is only an example, and the terminal device 10 can be determined by any method as long as a position (viewpoint) satisfying the set shooting range and the number of shooting positions can be determined based on the gazing point GP1. The position (viewpoint) may be determined. Further, the shooting range (120 °) and the number of shooting positions (9) shown in FIG. 4 are merely examples, and the shooting range (angle) and the number of shooting positions (cuts) are arbitrary. It can be set. For example, the shooting range may be 90 ° and the number of shooting positions may be five. Further, _{the positions of the viewpoints VP 1 to VP 9} may be changed according to the change in the positional relationship between the gazing point GP 1 and the terminal device 10 before the start of shooting. Further, the terminal device 10 may be able to switch parameters such as the parallax angle by setting.

Here, the display of the terminal device 10 regarding the position (viewpoint) for determining the gazing point GP1 and taking a picture will be described with reference to FIG. FIG. 5 is a diagram showing an example of a display at the time of shooting according to the embodiment. The display screens DP1 to DP5 shown in FIG. 5 show the display of the screen (output unit 50) of the terminal device 10. The points described with reference to FIGS. 1 to 4 will be omitted as appropriate.

First, the display screen DP1 shows a case where shooting is performed from the first viewpoint in FIG. As shown on the display screen DP1, the user U1 points the camera 70 _{a of the terminal device 10 toward the shooting target SB1 from the viewpoint VP a on} the left diagonal front side of the shooting target SB1 and taps the center of the shooting target SB1 on the display screen DP1. By doing so, the shooting target SB1 is shot. For example, the terminal device 10 calculates _a straight line passing through the position tapped by the user U1 and the camera 70 _a (viewpoint VP _a ) as the line-of-sight VL a. As shown in the display screen DP1, by displaying "Tap the center of the product", an appropriate image can be taken to generate a multi-viewpoint image.

Further, the display screen DP2 shows a case where shooting is performed from the second viewpoint in FIG. As shown on the display screen DP2, the user U1 points the camera 70 _{b of the terminal device 10 toward the shooting target SB1 from the viewpoint VP b on} the left diagonal front side of the shooting target SB1 and taps the center of the shooting target SB1 on the display screen DP2. By doing so, the shooting target SB1 is shot. For example, the terminal device 10 calculates a straight line passing through the position tapped by the user U1 and the camera 70 _b (viewpoint VP _b ) as the line-of-sight VL _b. In this way, the user U1 shoots the shooting target SB1 from the first position, and shoots the shooting target SB1 from the second position, which is a different position. As a result, the terminal device 10 determines the gazing point GP1 using the information acquired by photographing from the two viewpoints of the first position and the second position, and performs the guide display according to the distance. For example, the terminal device 10 estimates the depth from the image and determines the three-dimensional gazing point GP1 from the difference in tap positions.

Then, as shown on the display screen DP3, the terminal device 10 superimposes and displays the pin MK1 for designating the shooting angle and the shooting position guide GD1 having transparency. For example, the terminal device 10 appropriately uses existing AR (Augmented Reality) technology and image processing technology to display an object (virtual object) such as the shooting position guide GD1 superimposed on the display of the shooting area of the camera 70. .. The shooting position guide GD1 is a fan-shaped guide in which _nine circles _{corresponding to each of the viewpoints VP 1 to VP 9 shown in FIG. 4 are arranged on an arc with the gazing point GP1 as the center.} The user U1 arranges the terminal device 10 at a desired position and taps the pin MK1 to specify an angle for shooting.

The display screen DP4 shows a case where the shooting angle is specified and the display of the pin MK1 is changed to the pin MK2. The case where the image is taken from the second viewpoint in FIG. 3 is shown. The terminal device 10 fixes and displays the direction of the shooting position guide GD2 as shown on the display screen DP4 by designating the shooting angle.

After that, as shown in the display screen DP5, the terminal device 10 starts shooting for each position corresponding to each of the _{viewpoints VP 1 to VP 9.} The user U1 arranges the terminal device 10 at each viewpoint (position) and shoots at each viewpoint (position). In this way, the terminal device 10 also changes the direction of the shooting position guide GD1 according to the movement of the terminal device 10 before designating the reference point of the gazing point, and after the designation, the direction of the shooting position guide GD2 is fixed. do. Then, when the terminal device 10 completes the shooting from the corresponding viewpoint, the white circle turns yellow. The display under the camera button is also changed.

In the example of FIG. 5, the terminal device 10 displays the object OB1 indicating the central portion of the photographing area. Further, the terminal device 10 changes the display of the button BT1 to the button BT2 according to the start of shooting. In the example of FIG. 5, the terminal device 10 displays a gauge GG1 that can distinguish between a shooting completed position where shooting is completed and a shooting completed position where shooting is not completed among a plurality of positions. In the example of FIG. 5, it is shown that shooting is completed at _{three positions (corresponding to viewpoints VP 7 to VP 9} ) on the heel side of the shooting target SB1. Further, the terminal device 10 uses a color of "○" corresponding to three positions on the heel side of the shooting target SB1 in the shooting position guide GD3 ( _{corresponding to viewpoints VP 7 to VP 9) and another "○" color.} Change to a different color. The terminal device 10 may make the transmittance of the shooting position guide GD3 higher than that of the shooting position guide GD2 according to the start of shooting. The terminal device 10 appropriately uses existing three-dimensional spatial information processing technology and image processing technology to perform processing such as displaying the guide described above.

[1-4. Image editing]
Arbitrary editing is possible for the captured image. The terminal device 10 may accept the designation of the cropping range of the captured image and generate an image of only the designated range. As a result, the user U1 can extract the crop range (the range including the product, the range of the gazing point) after shooting, and can create a smooth multi-viewpoint image.

The terminal device 10 may arrange the captured multi-viewpoint images in a row and display the result by a carousel method. The terminal device 10 may switch the display of the image and display it by operating the bar displayed at the bottom of the screen, and rotate the multi-viewpoint image. Further, the terminal device 10 may accept editing for each image. The terminal device 10 displays the edited image so as to be distinguishable from other images. For example, the terminal device 10 displays the edited image so as to be distinguishable from other images by attaching an arbitrary mark such as a white circle icon to the edited image.

Further, the terminal device 10 may be able to manually or automatically correct colors and the like. The terminal device 10 may accept image corrections for each image, or may accept corrections for all images. Thereby, for example, even if the color tone or the like changes due to backlight or the like at the time of shooting, the image can be appropriately corrected and a multi-viewpoint image can be generated. Further, the terminal device 10 may automatically correct the image by using the averaging technique. Further, the terminal device 10 may provide enlargement / reduction manually or automatically.

[1-5. Multi-view image generation]
By the above-mentioned processing, the terminal device 10 has the captured images P1 ₁ , image P1 ₂ , image P1 ₃ , image P1 ₄ , image P1 ₅ , image P1 ₆ , image P1 ₇ , image P1 ₈ and image P1 _9. 9 shot images are taken. Terminal device 10, at a position corresponding to the viewpoint VP _1, captures images P1 _1. Terminal device 10, at a position corresponding to the viewpoint VP _2, captures images P1 _2. Similarly, the terminal device 10, at each corresponding position of the viewpoint _VP 3 _{~VP 9,} captures images _P1 3 _{~P1 9.}

Thus, the terminal device 10 acquires the image _P1 1 to P1 ₉ which is a group for generating a free-viewpoint images (multi-view image). Then, the terminal apparatus 10, based on the image _P1 1 to P1 ₉ obtained, the multi-view image is created. Specifically, the terminal apparatus 10 performs a predetermined calibration (camera calibration) with respect to the nine image P1 ₁ to P1 ₉ obtained, suitable for converting an image for constructing a multi-viewpoint image Parameters (for example, projective transformation matrix) are acquired. Then, the terminal device 10 projects and transforms the _{image data (images P1 1 to P1 9) using the acquired parameters.} Then, the terminal device 10, by connecting the projection transformation image (image _RP1 1 _{~RP1 9),} free viewpoint images that can transition smoothly display a plurality of images (image _RP1 1 _{~RP1 9)} (Multi-viewpoint image) is generated, but the point of generating a free-viewpoint image (multi-viewpoint image) is the same as in Patent Document 1, so detailed description thereof will be omitted. In this way, the terminal device 10 generates a free viewpoint image (multi-viewpoint image) in which the viewpoint can be controlled after the user U1 takes a picture.

[1-6. Facilitating shooting]
A free-viewpoint image (multi-viewpoint image) can be generated by shooting as described above, but shooting is time-consuming. For example, it is difficult to take a plurality of images at equal intervals, and it is desired to facilitate taking an image for generating a free-viewpoint image (multi-viewpoint image). Therefore, the terminal device 10 provides a model for guiding the following shooting. This point will be described with reference to FIGS. 6 and 7. FIG. 6 is a diagram showing an example of the arrangement of the 3D model according to the embodiment. FIG. 7 is a diagram showing an example of display of the 3D model according to the embodiment. The same points as those in FIGS. 1 to 5 will be omitted as appropriate.

As shown in FIG. 6, the terminal device 10 displays a model MD which is a 3D model (three-dimensional model) showing a shooting position and a shooting direction. For example, the terminal device 10 appropriately uses existing AR technology and image processing technology to display an object (virtual object) such as a model MD superimposed on the display of the shooting area of the camera 70. Although details will be described later, the terminal device 10 displays a model MD arranged between the shooting target SB1 and the camera 70. For example, the terminal device 10 displays a model MD arranged so that the angle and the distance to the gazing point satisfy a predetermined condition.

FIG. 7 shows a display example of the model MD in the terminal device 10. The display mode of the model MD is not limited to the mode shown in FIG. 7, and may be any display mode as long as the user can recognize the shooting position and the shooting direction at a predetermined viewpoint. In the example of FIG. 7, "1/13" is displayed on the terminal device 10, the shooting from one of the 13 viewpoints is completed, and the shooting from the remaining 12 viewpoints is not completed. Indicates that.

In FIG. 7, among the circles on the arc of the shooting position guide GD displayed on the terminal device 10 (five are shown in FIG. 7), shooting from the viewpoint corresponding to the rightmost circle (◯) is completed. And. The terminal device 10 is a model MD (black frame in FIG. 7) indicating the shooting position and shooting direction at the viewpoint corresponding to the second circle (○) from the right end among the 12 viewpoints for which shooting is not completed. Is displayed. For example, the terminal device 10 displays the model MD so as to overlap with the shooting position guide GD, which is a guide icon corresponding to the gazing point GP1. The display shown in FIG. 7 is only an example, and the terminal device 10 performs processing such as display of the model MD described above by appropriately using existing three-dimensional space information processing technology and image processing technology.

By aligning the object OB (white frame in FIG. 7) with the model MD, the user U1 shoots at the viewpoint corresponding to ◯ at the right end of the shooting position guide GD. For example, the terminal device 10 shoots according to the overlap between the object OB and the model MD. When the range in which the object OB and the model MD overlap is equal to or greater than a predetermined threshold value, the terminal device 10 determines that the condition is satisfied and executes shooting. For example, the terminal device 10 takes a picture when a 2D guide such as an object OB and a 3D model such as a model MD are superimposed at a certain threshold value or more within a predetermined time such as a designated time. As a result, the terminal device 10 can suppress camera shake during shooting. When the terminal device 10 uses a range in which the 2D guide such as the object OB and the 3D model such as the model MD do not overlap, the 2D guide such as the object OB and the 3D model such as the model MD are below a certain threshold value. Shooting may be performed when the images are superimposed. For example, the terminal device 10 may perform shooting when the range of the 3D model such as the model MD that does not overlap with the 2D guide of the object OB or the like is equal to or less than the threshold value. In this way, the threshold value may be appropriately set according to the information used for the determination. For example, when the object OB and the model MD match, the terminal device 10 may display or notify by vibration. The terminal device 10 may have some play (error) at the position where the model MD is displayed, or may use a loose threshold value. Then, the terminal device 10 (automatically) takes a picture and then corrects the taken image. For example, the terminal device 10 corrects the captured image according to the play (error) of the display of the model MD.

As described above, the terminal device 10 can suppress camera shake during shooting by shooting according to the overlap between the 2D guide such as the object OB and the 3D model such as the model MD. Further, the terminal device 10 can facilitate taking an image for generating a free viewpoint image (multi-viewpoint image) by displaying a guide like the above-mentioned model. In this way, the terminal device 10 displays the model MD as a guide in the form of AR (augmented reality) floating in space. As a result, the terminal device 10 can suppress the camera shake of the user U1 by displaying the model MD so that the user U1 recognizes the shooting position, the distance, and the like.

[1-7. About the shooting target]
In the example of FIG. 1, the case where the shoe A, which is the transaction target, is the subject of photography is shown, but the subject to be photographed is not limited to the transaction target, and may be, for example, a person or a landscape.

[1-8. About the generation target]
In the above-mentioned example, the case of generating a multi-viewpoint image is shown, but the target to be generated is not limited to the multi-viewpoint image, and the terminal device 10 may generate a three-dimensional model (3D model) to be photographed. For example, the terminal device 10 may generate a three-dimensional model (3D model) of the shoe A, which is the shooting target SB1, based on the shot image.

[2. Example of functional configuration]
Hereinafter, an example of the functional configuration of the terminal device 10 and the service providing device 100 in order to execute each of the above processes will be described.

[2-1. An example of the functional configuration of a terminal device]
First, an example of the functional configuration of the terminal device 10 will be described with reference to FIG. FIG. 8 is a diagram illustrating an example of a functional configuration of the terminal device according to the embodiment. As shown in FIG. 8, the terminal device 10 includes a communication unit 20, a storage unit 30, an input unit 40, an output unit 50, a physical sensor 60, a camera 70, and a control unit 80.

(Communication unit 20)
The communication unit 20 is realized by, for example, a NIC (Network Interface Card) or the like. Then, the communication unit 20 is connected to the network N by wire or wirelessly, and transmits / receives information to / from the service providing device 100. The communication unit 20 may be able to communicate with a distribution server that distributes various contents.

(Memory unit 30)
The storage unit 30 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory (Flash Memory), or a storage device such as a hard disk or an optical disk. The storage unit 30 has a transaction target information storage unit 31. The storage unit 30 is not limited to the transaction target information storage unit 31, and stores various information. The storage unit 30 stores information used for displaying a shooting position guide, a 3D model (model MD), and the like. The storage unit 30 stores various information regarding the transaction target to be displayed in the second application. For example, the storage unit 30 stores information on a transaction target provided by the service providing device 100.

(Transaction target information storage unit 31)
The transaction target information storage unit 31 stores information regarding the transaction target. FIG. 11 is a diagram showing an example of a transaction target information storage unit according to an embodiment. In the example shown in FIG. 11, the transaction target information storage unit 31 has items such as "transaction target ID", "transaction target", "target-related information", "multi-viewpoint image ID", and "multi-viewpoint image information". ..

The “transaction target ID” indicates identification information for identifying the transaction target. The "transaction target" indicates information for specifying a transaction target such as a name, for example, a product name or a service name (service name). The transaction target information storage unit 31 may store information indicating the transaction target category in association with the transaction target ID. For example, the transaction target information storage unit 31 may store information indicating a transaction target category such as a product or service, such as “shoes” or “haircut”.

The "target-related information" is information related to the description of the transaction target associated with the "transaction target ID". For example, the target-related information includes various information (exhibition information) necessary for listing a transaction target when listing the transaction target. The target-related information includes, for example, various information related to the transaction target, such as text information explaining the content of the transaction target, price, shipping fee, and inventory quantity. Information such as a name and target-related information may be input by the user, or may be acquired by the terminal device 10 from an external device.

The "multi-viewpoint image ID" indicates identification information of a multi-viewpoint image (free-viewpoint image) taken with the corresponding transaction target as a shooting target (subject). The "multi-viewpoint image information" has items such as "captured image data", "calibration data", and "converted image data", and information related to the multi-viewpoint image is stored.

The “captured image data” indicates the image data that is the source of the multi-viewpoint image. For example, in the "captured image data", a plurality of image data captured by the terminal device 10 by the user according to the guide by the first application are stored. Since the multi-viewpoint image is generated from a plurality of image data, a plurality of captured image data are stored for one multi-viewpoint image.

“Calibration data” indicates predetermined calibration data (camera calibration) for generating a multi-viewpoint image. In FIG. 11, the calibration data is shown by a concept such as _{“R1 1”.} The calibration data includes various data for appropriately projecting and transforming the captured image data in order for the terminal device 10 to acquire an image constituting the multi-viewpoint image. For example, the calibration data includes the camera parameters of the camera 70 (imaging device) of the terminal device 10 that captured the image. For example, the calibration data includes various camera parameters necessary for generating a multi-viewpoint image, such as the focal length of the camera 70 and the aspect ratio and skew of the image generated by the camera 70.

"Converted image data" indicates image data constituting a multi-viewpoint image. The converted image data indicates, for example, the image data after the captured image data is subjected to predetermined calibration and the projection conversion is performed.

In FIG. 11, it is shown that the transaction target (transaction target SB1) identified by the transaction target ID “SB1” is the shoe A. The target-related information of the transaction target SB1 indicates that it is the target-related information INF1. For example, the target-related information INF1 includes various information related to the transaction target SB1, such as text information explaining the contents of the transaction target SB1 which is the shoe A, a price, a shipping fee, and the number of stocks.

Further, it is shown that the multi-viewpoint image of the shoe A, which is the transaction target SB1, is a multi-viewpoint image (multi-viewpoint image FV1) identified by the multi-viewpoint image ID “FV1”.

The multi-viewpoint image FV1 indicates that the captured image data is _{nine images of "P1 1} " to "P1 9". In order to generate the multi-viewpoint image FV1, the captured images P1 ₁ , image P1 ₂ , image P1 ₃ , image P1 ₄ , image P1 ₅ , image P1 ₆ , image P1 ₇ , image P1 ₈ and image P1 _{9 are taken.} It is shown that the nine captured images of the above are used.

Further, in the multi-viewpoint image FV1, it is shown that the calibration data of each of the nine _{images of the images P1 1} to P1 _{9 are "R1 1} " to "R1 _9". For example, it is shown that the calibration data of the _{image P1 1 is the calibration data R1 1.}

The multi-viewpoint image FV1 indicates that the converted image data is _{nine images of "RP1 1} " to "RP1 9". For example, the image RP1 ₁ is converted images show that the transformed image the image P1 ₁ by the calibration data R1 _1. The multi-viewpoint image FV1 is composed of nine converted images of _{image RP1 1} , image RP1 ₂ , image RP1 ₃ , image RP1 ₄ , image RP1 ₅ , image RP1 ₆ , image RP1 ₇ , image RP1 ₈ and image RP1 _9. Indicates that it is configured.

That is, the multi-viewpoint image FV1 is composed of nine images in the order of _{images RP1 1} , RP1 ₂ , RP1 ₃ , RP1 ₄ , RP1 ₅ , RP1 ₆ , RP1 ₇ , RP1 ₈ and RP1 _9. Show that. For example, in the multi-viewpoint image FV1, the images RP1 ₁ , RP1 ₂ , RP1 ₃ , RP1 ₄ , RP1 ₅ , RP1 ₆ , RP1 ₇ , RP1 ₈ and RP1 ₉ are numbered 1 to 9 in this order. For example, the multi-viewpoint image FV1 is composed of _{nine images RP1 1 to} RP1 9 in a format such as JPEG (Joint Photographic Experts Group). Note that the multi-viewpoint image FV1, the image _RP1 1 _{~RP1 9} by free viewpoint image (multi-viewpoint number of images) provide, if possible, may be in the form of such APNG (Animated Portable Network Graphics).

The transaction target information storage unit 31 is not limited to the above, and may store various information depending on the purpose. The transaction target information storage unit 31 may store an image (initial display image) corresponding to the initial display of the multi-viewpoint image. For example, the transaction target information storage unit 31 stores the image RP1 ₁ as the initial display image of the multi-viewpoint image FV1.

Further, the transaction target information storage unit 31 may store various information used for generating each multi-viewpoint image in association with the corresponding multi-viewpoint image ID. For example, the transaction target information storage unit 31 stores information about the gazing point used when generating the multi-viewpoint image in association with the multi-viewpoint image ID. For example, the transaction target information storage unit 31 stores information indicating the gazing point GP1 in association with the multi-viewpoint image FV1.

Further, for example, the transaction target information storage unit 31 stores information indicating a plurality of positions for capturing the multi-viewpoint image in association with the multi-viewpoint image ID. For example, the transaction target information storage unit 31 stores information indicating the positions of the viewpoints VP1 to VP9 in association with the multi-viewpoint image FV1.

(Input unit 40)
The input unit 40 is an input device that receives various operations from the user. The input unit 40 receives various operations from the user via the display surface by the touch panel function. For example, the input unit 40 functions as a reception unit that accepts user operations by the function of the touch panel. Further, the input unit 40 may accept various operations from a button provided on the terminal device 10 or a keyboard or mouse connected to the terminal device 10.

For example, the input unit 40 accepts an operation such as a user's designated operation via the screen (output unit 50) of the terminal device 10. As the detection method of the user's operation by the input unit 40, the capacitance method is mainly adopted in the tablet terminal, but other detection methods such as the resistance film method, the surface acoustic wave method, the infrared method, and the electromagnetic induction method are used. Any method may be adopted as long as the user's operation can be detected and the touch panel function can be realized.

(Output unit 50)
The output unit 50 is a display device for displaying various information, that is, a screen. For example, the output unit 50 is realized by a liquid crystal display or the like. When a touch panel is adopted for the terminal device 10, the input unit 40 and the output unit 50 are integrated. The output unit 50 may be described as a screen.

(Physical sensor 60)
The physical sensor 60 is a sensor that detects the physical state of the terminal device 10. For example, the physical sensor 60 is a gyro sensor that measures the inclination of the terminal device 10 in the three axial directions. The physical sensor 60 is not limited to the gyro sensor, and any sensor such as an acceleration sensor, a temperature sensor, a volume sensor, and a brightness sensor can be applied.

(Camera 70)
The camera 70 is an image pickup device for capturing an image (moving image or still image). The camera 70 is composed of an image pickup device such as a CCD (Charged-coupled devices) sensor or a CMOS (Complementary metal-oxide-semiconductor) sensor. The camera 70 is installed on the back side of the terminal device 10 (the surface opposite to the surface on which the screen is installed) and the back side of the terminal device 10 (the surface on which the screen is installed). It may be configured by a sub camera.

(Control unit 80)
The control unit 80 is, for example, a controller, and various programs (transmission program, provision program) stored in a storage device inside the terminal device 10 by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like. , Corresponding to an example of a display program, etc.) is realized by executing the RAM as a work area. Further, the control unit 80 is a controller, and is realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

In the example shown in FIG. 8, the control unit 80 has a functional configuration for realizing the above-mentioned first display processing and second display processing by executing the first application 81 and the second application 82. .. The control unit 80 may execute other arbitrary applications such as an application that becomes an OS (Operating System) of the terminal device 10 and an application for realizing reception of various operations. Further, the first application 81 and the second application 82 are integrated and may be one application. For example, the first application 81 and the second application 82 may be applications incorporated in an application used for using electronic commerce such as an auction application.

(First application 81)
The first application 81 causes the terminal device 10 to execute the above-mentioned transmission processing and information processing. The first application 81 is an application used for generating and transmitting a multi-viewpoint image. For example, FIG. 9 is a diagram showing an example of a functional configuration realized by the first application according to the embodiment. As shown in FIG. 9, the first application 81 realizes the first acquisition unit 811, the first display unit 812, the determination unit 813, the photographing unit 814, the generation unit 815, and the transmission unit 816.

The first acquisition unit 811 acquires various information from the storage unit 30. The first acquisition unit 811 acquires various information from the transaction target information storage unit 31. The first acquisition unit 811 receives various information from an external device such as the service providing device 100. The first acquisition unit 811 acquires the shooting target information regarding the shooting target.

The first acquisition unit 811 acquires multi-viewpoint images obtained by capturing the transaction target from different viewpoints. The first acquisition unit 811 acquires a multi-viewpoint image obtained by capturing a transaction target, which is an exhibit, from different viewpoints. The first acquisition unit 811 acquires a multi-viewpoint image from the transaction target information storage unit 31. The first acquisition unit 811 acquires the image _P1 1 to P1 ₉ which is a group for generating a free-viewpoint images (multi-view image).

The first display unit 812 displays various information by the output unit 50 (liquid crystal display or the like). The first display unit 812 displays various information on the output unit 50. The first display unit 812 controls the output unit 50 and executes a process of displaying information.

The first display unit 812 functions as a providing unit that provides a 3D model indicating a shooting position and a shooting direction for shooting a multi-viewpoint image to be shot. When the provision of the 3D model is transmission to another device, the transmission unit 816 may function as the provision unit.

The first display unit 812 displays a guide for taking a picture of a multi-viewpoint image. The first display unit 812 displays a guide including a 3D model indicating a shooting position and a shooting direction. The first display unit 812 displays the 3D model on the screen of the terminal device 10. The first display unit 812 superimposes and displays the 3D model on the screen of the terminal device that displays the shooting area including the shooting target. The first display unit 812 superimposes and displays the 3D model on the screen of the terminal device that displays the shooting area including the transaction target.

The first display unit 812 displays an object indicating the central portion of the shooting area. The first display unit 812 displays a 3D model arranged between the viewpoint at which the object to be photographed is photographed and the gazing point at the object to be photographed. The first display unit 812 displays a 3D model arranged between the gazing point determined based on the line of sight from the viewpoint and the terminal device. The first display unit 812 displays a 3D model arranged between the gaze point determined based on the first line of sight from the first viewpoint and the second line of sight from the second viewpoint and the terminal device.

The first display unit 812 displays a shooting position guide indicating a plurality of positions for shooting a multi-viewpoint image based on the gazing point in the transaction target. The first display unit 812 displays a shooting position guide in the shape of a fan centered on the gazing point. The first display unit 812 displays a shooting position guide in which each of the plurality of positions is arranged on the arc of the fan. The first display unit 812 has a first radius connecting the first position on one end side of the plurality of positions and the gazing point, and a second radius connecting the second position on the other end side of the plurality of positions and the gazing point. Displays a shooting position guide, which is the shape of a fan centered on the angle between the two. For example, the first display unit 812 displays the shooting position guide GD1 with the arc directed downward with the gazing point GP1 as the center.

The first display unit 812 superimposes and displays a transparent shooting position guide on the screen of the terminal device that displays the shooting area including the transaction target. The first display unit 812 displays, among the plurality of positions, the shooting completed position where the shooting is completed and the shooting completed position where the shooting is not completed so as to be distinguishable.

The first display unit 812 performs various arithmetic processes and the like necessary for the above-mentioned display. For example, the first display unit 812 determines the gazing point. The first display unit 812 calculates the gazing point using various information. The first display unit 812 determines the gazing point based on the line of sight from the viewpoint.

The first display unit 812 determines the gazing point from a plurality of viewpoints based on the line-of-sight information. The first display unit 812 determines the gazing point from the two viewpoints based on the line-of-sight information. For example, the first display unit 812 determines the gazing point based on the first line of sight from the first viewpoint and the second line of sight from the second viewpoint.

The first display unit 812 is not limited to the above, and may calculate the gazing point using various information. For example, the first display unit 812 may determine the gazing point from one viewpoint based on the line-of-sight information and the plane information. The first display unit 812 may determine the gazing point based on the information of the line of sight from one viewpoint and the information of the surface on which the photographing target such as the transaction target is placed.

The first display unit 812 determines a plurality of positions (viewpoints) for taking a picture of the multi-viewpoint image based on the gazing point GP1. The first display unit 812 determines (calculates) the _{viewpoints VP 1 to VP 9} based on the set shooting range and the number of shooting positions. The first display unit 812 determines the _{viewpoints VP 1 to VP 9} based on the set shooting range (120 °) and the number of shooting positions (9).

For example, the first display unit 812 calculates _{nine viewpoints VP 1 to} VP 9 using the information on the estimated distance between the shooting target SB1 and the camera 70. The first display unit 812 sets the distance between the gazing _{point GP 1 and the camera 70 b} (referred to as “distance Db”) as the distance between the _{gazing point GP 1 and the viewpoints VP 1 to VP 9.} The first display unit 812 arranges the _{viewpoints VP 1 to VP 9} at a position at a distance Db from the gazing point GP1. The above is only an example, _{and the distance between the viewpoint GP 1 and the viewpoints VP 1 to VP 9} may be appropriately determined using various information. For example, the first display unit 812, using the distance between the gazing point GP1 and the camera 70 _a, may determine the distance between the gazing point GP1 and the viewpoint _VP 1 _{~VP 9.}

Further, for example, the first display unit 812 determines the position of _{the viewpoint VP b to} the position of the viewpoint VP _5. The first display unit 812, around the viewpoint VP _5, arranged at 15 ° intervals along the predetermined plane, nine viewpoints _VP 1 ～VP each visual line _VL 1 ～VL ₉ passes gaze point GP1 ₉ To determine.

For example, the first display unit 812 may arrange _{nine viewpoints VP 1 to} VP 9 at intervals of 15 ° around _{the viewpoint VP 5} on a plane corresponding to the posture of the _{camera 70 b.} For example, the first display unit 812, a viewpoint VP _b intersecting streets height direction (along the horizontal plane) on the plane, nine at 15 ° intervals around the viewpoint VP ₅ viewpoint _VP 1 ～VP ₉ It may be arranged. The above is only an example, and _{the arrangement of the nine viewpoints VP 1 to} VP 9 may be appropriately determined using various information. For example, the first display unit 812 may arrange _{nine viewpoints VP 1 to} VP 9 on a plane passing through _{the viewpoint VP a} and the viewpoint VP _b.

The first display unit 812 uses information on the positions of the gazing point GP1 and the positions of the nine viewpoints VP _{1 to VP 9} , and the fan-shaped photographing position _{centering on the gazing point GP1 and the viewpoint VP 5 at the center of the arc.} Display the guide GD1. The first display unit 812 is a fan-shaped photographing position guide having a radius of a line segment connecting the gazing point GP1 and the viewpoint VP ₁ and a line segment connecting the gazing point GP1 and the viewpoint VP _{2 with the gazing point GP1 as the center.} Display GD1.

The first display unit 812 displays a model MD which is a 3D model showing a shooting position and a shooting direction. The first display unit 812 is arranged between the gazing point GP1 and the camera 70, and displays a model MD indicating a shooting position and a shooting direction at any of the _{nine viewpoints VP 1 to VP 9.} ..

The first display unit 812 displays a model MD indicating a shooting position and a shooting direction at a viewpoint in which shooting is not completed among the _{nine viewpoints VP 1 to VP 9.} The first display unit 812 _{manages information on whether or not each of the nine viewpoints VP 1 to} VP 9 has completed shooting, and indicates a shooting position and shooting direction at viewpoints where shooting has not been completed. Is displayed.

Taking FIG. 4 as an example, the first display unit 812 displays, for example, a model MD indicating a shooting position and a shooting direction _{at the viewpoint VP 9 in which shooting is incomplete.} The first display unit 812 arranges the model MD at a position between the gazing point GP1 and the viewpoint VP _9.

When the shooting at the _{viewpoint VP 9} is completed, the first display unit 812 changes the display of the model MD so as to indicate the shooting position and the shooting direction at the viewpoint where the shooting other than the _{viewpoint VP 9 is not completed.} For example, when the shooting at the _{viewpoint VP 9} is completed, the first display unit 812 indicates the shooting position and the shooting direction at the _{viewpoint VP 8} which is a viewpoint adjacent to the viewpoint _{VP 9 and whose shooting is not completed.} To change the display of the model MD. In this case, the first display unit 812 displays the model MD indicating the shooting position and the shooting direction at the _{viewpoint VP 8.} The first display unit 812 arranges the model MD at a position between the gazing point GP1 and the viewpoint VP _8.

Further, when the shooting at the _{viewpoint VP 8} is completed, the first display unit 812 changes the display of the model MD so as to indicate the shooting position and the shooting direction at the viewpoint where the shooting other than the _{viewpoint VP 8 is not completed.} do. For example, when the shooting at the _{viewpoint VP 8} is completed, the first display unit 812 indicates the shooting position and the shooting direction at the _{viewpoint VP 7} which is a viewpoint adjacent to the viewpoint _{VP 8 and whose shooting is not completed.} To change the display of the model MD. In this case, the first display unit 812 displays the model MD indicating the shooting position and the shooting direction at the _{viewpoint VP 7.} The first display unit 812 arranges the model MD at a position between the gazing point GP1 and the viewpoint VP _7. The first display unit 812 displays the model MD so that the shadow indicates the shooting position and the shooting direction at the viewpoint where the shadow is not completed until all of the _{nine viewpoints VP 1 to VP 9 are completed.}

The above is only an example, and the first display unit 812 may display a model MD indicating a shooting position and a shooting direction according to a desired embodiment.

The determination unit 813 performs various determination processes. The determination unit 813 determines whether or not to perform imaging. The determination unit 813 determines whether or not to perform imaging by using the information indicating the position of each object. The determination unit 813 determines whether or not to perform imaging by using the threshold value for determining whether or not to perform imaging.

The determination unit 813 determines the overlap between the central portion of the photographing region of the terminal device 10 and the 3D model. The determination unit 813 determines the overlap between the central portion indicated by the object and the 3D model. The determination unit 813 determines that a predetermined condition is satisfied when the range in which the central portion and the 3D model overlap is equal to or greater than a predetermined threshold value. For example, the determination unit 813 makes a determination using an image processing technique such as back projection. For example, the determination unit 813 identifies where the 3D model is by using the back projection technique, and determines the overlap between the central portion indicated by the object and the 3D model.

The shooting unit 814 controls the camera 70 and executes a process of shooting an image. When the determination unit 813 determines that the overlap between the central portion and the 3D model satisfies a predetermined condition, the photographing unit 814 performs imaging. For example, the photographing unit 814 photographs when a 2D guide such as an object OB and a 3D model such as a model MD are superimposed at a certain threshold value or more within a predetermined time such as a designated time. Further, when the photographing unit 814 uses the range in which the 2D guide such as the object OB and the 3D model such as the model MD do not overlap, the 2D guide such as the object OB and the 3D model such as the model MD are below a certain threshold value. Shooting may be performed only when the images are superimposed.

The generation unit 815 generates a free viewpoint image (multi-viewpoint image). The generation unit 815 generates a free viewpoint image (multi-viewpoint image) by performing a predetermined conversion process on the captured image (original image data). The generation unit 815 is a free-viewpoint image (many) as if a plurality of images captured from multiple viewpoints were taken with the optical axes of the respective image pickup devices (cameras) overlapped at one point at a three-dimensional position. A projection transformation matrix is derived for conversion to the viewpoint image). In this case, the generation unit 815 refers to the camera parameters including the position and orientation of the camera 70 as the parameters of the image pickup device that captures each image.

Generating unit 815, based on the image _P1 1 to P1 ₉ acquired by the first acquisition unit 811, to generate a free viewpoint image (the multi-view image). The generation unit 815 performs a _{predetermined calibration (camera calibration) on the nine images P1 1 to} P19 to perform appropriate parameters (for example, projective transformation) for converting the images into images for constituting a multi-viewpoint image. Matrix) is generated. For example, generator 815 projective transformation image _P1 1 to P1 ₉ using the parameters of the camera 70 (the camera parameter). For example, the generating unit 815, by using the calibration data _R1 1 _{~R1 9,} to the projective transformation of the image _P1 1 to P1 _9. The generation unit 815, by connecting the image _RP1 1 _{~RP1 9} which is projective transformation, free-viewpoint images that can transition smoothly display a plurality of images _RP1 1 _{~RP1 9} (the multi-view image) Generate.

The terminal device 10 may acquire a free viewpoint image (multi-viewpoint image) generated by an external device. For example, the terminal device 10, the external device to generate a free viewpoint image (the multi-view image) (generator), and transmitted by the transmitting unit 816 an image _P1 1 to P1 ₉ and calibration data _R1 1 _{~R1 9,} generator it may receive the image _RP1 1 _{~RP1 9} the first acquisition unit 811 generated by. In this case, the first application 81 may not include the generation unit 815.

The transmission unit 816 transmits various information to an external device such as the service providing device 100. The transmission unit 816 transmits a multi-viewpoint image together with the target-related information related to the transaction target to the service providing device 100 that provides the service for distributing the information of the transaction target. The transmission unit 816 transmits a multi-viewpoint image together with the target-related information of the transaction target to the service providing device that provides the service related to the sale and purchase of the transaction target.

The transmission unit 816 transmits a multi-viewpoint image together with the target-related information of the transaction target to the service providing device that provides the service related to the listing of the transaction target. The transmission unit 816 transmits a multi-viewpoint image together with the target-related information of the transaction target to the service providing device that provides the service related to the listing of the transaction target. The transmission unit 816 transmits a multi-viewpoint image together with the target-related information which is the listing information of the transaction target, which is the exhibit, to the service providing device that provides the listing service of the transaction target. For example, when the transmitting unit 816 functions as a providing unit, the transmitting unit 816 transmits the 3D model to an external device such as the service providing device 100.

(Second application 82)
The second application 82 causes the terminal device 10 to execute the above-mentioned display process. The second application 82 is an application used for displaying contents such as multi-viewpoint images. For example, FIG. 10 is a diagram showing an example of a functional configuration realized by the second application according to the embodiment. As shown in FIG. 10, the second application 82 realizes the second acquisition unit 821, the second display unit 822, and the reception unit 823.

The second acquisition unit 821 acquires various information from the storage unit 30. The second acquisition unit 821 acquires various information from the transaction target information storage unit 31. The second acquisition unit 821 receives various information from an external device such as the service providing device 100. The second acquisition unit 821 receives the information provided (delivered) from the service providing device 100. For example, the second acquisition unit 821 receives an image group including a multi-viewpoint image as information to be traded.

The second acquisition unit 821 receives the information of the transaction target provided by the service providing device 100 and stores it in the storage unit 30. When the transaction target information is not stored in the storage unit 30, the second application 82 may manage the transaction target information so that it can be referred to in the second application 82.

The second acquisition unit 821 acquires an image group including a first image which is a multi-viewpoint image captured by a different viewpoint and a second image different from the first image. The second acquisition unit 821 acquires an image group from the storage unit 30.

The second display unit 822 displays various information by the output unit 50 (liquid crystal display or the like). The second display unit 822 displays various information on the output unit 50. The second display unit 822 controls the output unit 50 and executes a process of displaying information.

The second display unit 822 performs various arithmetic processes and the like necessary for the display described later. The second display unit 822 switches the image included in the first image and displays it on the screen of the terminal device 10 in response to the operation of switching the display of the image group on the screen of the terminal device 10 by the user who uses the terminal device 10. .. The second display unit 822 switches the image included in the first image and displays it on the screen of the terminal device 10 according to the operation at the time of display in the first display mode displayed for each image in the image group.

The second display unit 822 displays in the order of each image included in the first image according to the operation. The second display unit 822 displays each image in chronological order in which the first image was taken, according to the operation. The second display unit 822 displays the image group in the second display mode in which the first image and the second image are displayed side by side in a list, and when one image displayed in the second display mode is designated, the second display unit 822 is displayed. The display mode is switched to the first display mode in which one image is displayed.

In the second display mode, the second display unit 822 displays a predetermined image of the first image as an image showing the first image. In the second display mode, the second display unit 822 displays the image having the first order among the first images.

The second display unit 822 switches and displays the image included in the first image according to the operation of the user received by the reception unit 823. The second display unit 822 switches the image included in the first image and displays it on the screen of the terminal device 10 according to the movement of the contact position. The second display unit 822 displays each image along the time series in which the first image was taken according to the movement of the contact position in a predetermined direction.

The second display unit 822 displays the second image in the order next to the first image according to the movement of the contact position in a predetermined direction when the image having the last order among the first images is displayed. The second display unit 822 displays each image in the reverse order of the time series in which the first image was taken, in response to the movement of the contact position in the direction opposite to the predetermined direction. The second display unit 822 displays the second image in the order before the first image in response to the movement of the contact position in the opposite direction when the first image in the order is displayed.

The second display unit 822 displays the multi-viewpoint image FV1 of the shoe A and the image group IGP including the images IM1 to IM2, IM4 to IM6 of the other shoes. In the second display unit 822, the images IM1 to IM2 of the other shoes and the multi-viewpoint image FV1 of the shoe A are arranged at the uppermost stage, and the images IM4 to IM6 of the other shoes are arranged at the second stage, and the images of the shoes are displayed. Display the content CT1 to be displayed in a list.

The second display unit 822 changes the display from the image list display mode (second display mode) to the display mode for each image (first display mode) according to the operation of the user U1. In the state where only the image IM1 of the other shoe is displayed, the second display unit 822 displays only the image IM1 of the other shoe in response to the forward swipe operation by the user U1. The display is switched to the state where only the image IM2 is displayed. In the state where only the image IM2 of the other shoe is displayed, the second display unit 822 displays only the image IM2 of the other shoe in response to the swipe operation in the reverse direction by the user U1 to display the image IM2 of the other shoe. The display is switched to the state where only the image IM1 is displayed.

In the state where only the image IM2 of the other shoes is displayed, the second display unit 822 displays only the image IM2 of the other shoes in response to the forward swipe operation by the user U1. The display is switched to the state where only the viewpoint image FV1 is displayed. _{The second display unit 822 displays the first image RP1 1} of the multi-viewpoint image FV1 of the shoe A from the state where only the image IM2 of the other shoes is displayed in response to the forward swipe operation by the user U1. Switch the display to the displayed state.

The second display unit 822 displays the multi-viewpoint image FV1 from the image RP1 _{1 to the image RP1 2} , the image RP1 ₃ , the image RP1 ₄ , and the image RP1 ₅ in response to the movement of the swipe operation in the forward direction of the user U1. Image RP1 ₆ , image RP1 ₇ , image RP1 ₈ and image RP1 ₉ are displayed in this order in chronological order.

The second display unit 822, in accordance with the movement of the swipe in the forward direction of the user U1 in a state of displaying the last image RP1 _9, displays the next image IM4 of the multi-view image FV1. The second display unit 822 switches the display from the state in which only the multi-viewpoint image FV1 of the shoe A is displayed to the state in which only the image IM4 of the other shoes is displayed in response to the forward swipe operation by the user U1.

The reception unit 823 accepts the user's operation. The reception unit 823 receives the user's operation via the input unit 40. For example, the reception unit 823 accepts the user's contact with the screen (output unit 50) of the terminal device 10 as a user operation. The reception unit 823 accepts a swipe operation by the user. In addition, the reception unit 823 accepts various operations such as flick, tap, double tap, long tap, pinch in, and pinch out.

The reception unit 823 receives the operation by the user in the first display mode. The reception unit 823 accepts an operation of moving the contact position in a predetermined direction while the user is in contact with the screen of the terminal device 10. The reception unit 823 accepts an operation of moving the contact position in a direction opposite to a predetermined direction while the user is in contact with the screen of the terminal device 10.

The reception unit 823 receives a request for information by the user according to the operation of the user. The reception unit 823 transmits the received information request to the service providing device 100 via the communication unit 20. The reception unit 823 receives the request for the transaction information of the category "shoes" by the user, and transmits the request for the transaction information of the category "shoes" to the service providing device 100.

Note that each process by the control unit 80 described above is not limited to the case of using a dedicated application such as the first application 81 or the second application 82, and for example, using Javascript (registered trademark), HTML (HyperText Markup Language), or the like. It may be realized.

[2-2. An example of the functional configuration of a service providing device]
Subsequently, an example of the functional configuration of the service providing device 100 will be described with reference to FIG. 12. FIG. 12 is a diagram showing a configuration example of the service providing device according to the embodiment. As shown in FIG. 12, the service providing device 100 includes a communication unit 110, a storage unit 120, and a control unit 130. The service providing device 100 includes an input unit (for example, a keyboard, a mouse, etc.) that receives various operations from the administrator of the service providing device 100, and an output unit (for example, a liquid crystal display, etc.) for displaying various information. You may have.

(Communication unit 110)
The communication unit 110 is realized by, for example, a NIC or the like. Then, the communication unit 110 is connected to the network N by wire or wirelessly, and transmits / receives information to / from the terminal device 10.

(Memory unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 120 stores various information necessary for providing the service. For example, the storage unit 120 stores various information necessary for providing the auction service.

The storage unit 120 stores various information regarding the transaction target. The storage unit 120 stores various information regarding the transaction target to be provided in the service provision. For example, the storage unit 120 stores a multi-viewpoint image of a transaction target and target-related information in association with identification information that identifies a provider that provides the transaction target.

(Control unit 130)
The control unit 130 is, for example, a controller, and various programs (corresponding to an example of a notification program) stored in a storage device inside the service providing device 100 are executed by a CPU, MPU, or the like using the RAM as a work area. Is realized by. Further, the control unit 130 is a controller, and is realized by, for example, an integrated circuit such as an ASIC or FPGA.

As shown in FIG. 12, the control unit 130 has a reception unit 131, a processing unit 132, and a distribution unit 133, and realizes or executes the functions and operations of information processing described below.

(Receiver 131)
The receiving unit 131 receives various information. The receiving unit 131 receives various information from an external device such as the terminal device 10. The receiving unit 131 receives a multi-viewpoint image from the terminal device 10. The receiving unit 131 receives a multi-viewpoint image from the terminal device 10.

The receiving unit 131 accepts the listing of the transaction target from the terminal device 10. The receiving unit 131 receives the multi-viewpoint image of the transaction target together with the target-related information related to the transaction target. The receiving unit 131 receives the multi-viewpoint image together with the target-related information which is the listing information of the transaction target which is the exhibit. For example, the receiving unit 131 receives a 3D model indicating a shooting position and a shooting direction for shooting a multi-viewpoint image to be shot.

The receiving unit 131 receives a request for information from the terminal device 10. The receiving unit 131 receives information indicating the category of the transaction target as an information request. The receiving unit 131 accepts a bid for a transaction target exhibited by the terminal device 10.

(Processing unit 132)
The processing unit 132 executes various processes related to service provision. The processing unit 132 registers the information received by the receiving unit 131 in the storage unit 120. The processing unit 132 registers various information regarding the transaction target in the storage unit 120.

The processing unit 132 selects various types of information. The processing unit 132 selects various information based on the various information stored in the storage unit 120. Further, the processing unit 132 selects information from the storage unit 120. The processing unit 132 selects the information of the transaction target corresponding to the user's request from the transaction targets registered in the storage unit 120. The processing unit 132 selects the information of the transaction target corresponding to the category requested by the user from the transaction targets registered in the storage unit 120.

(Distribution unit 133)
The distribution unit 133 distributes various information. The distribution unit 133 transmits various information to the terminal device 10. The distribution unit 133 distributes the information of the transaction target requested to the terminal device 10. The distribution unit 133 transmits the information of the transaction target selected by the processing unit 132 to the terminal device 10. The distribution unit 133 transmits the information of the transaction target including the multi-viewpoint image to the terminal device 10. The distribution unit 133 transmits an image group including a multi-viewpoint image to the terminal device 10.

[3. Information processing flow]
Next, an example of the transmission processing procedure according to the embodiment will be described with reference to FIG. FIG. 13 is a flowchart showing an example of the flow of transmission processing according to the embodiment. For example, the terminal device 10 acquires a multi-viewpoint image obtained by capturing a transaction target from different viewpoints (step S101). Then, the terminal device 10 transmits a multi-viewpoint image together with the target-related information related to the transaction target to the service providing device 100 that provides the service for distributing the information of the transaction target (step S102).

Next, an example of the information processing procedure according to the embodiment will be described with reference to FIG. FIG. 14 is a flowchart showing an example of the flow of information processing according to the embodiment. For example, the terminal device 10 acquires shooting target information regarding the shooting target (step S201). Then, the terminal device 10 provides a model showing a shooting position and a shooting direction for shooting a multi-viewpoint image to be shot (step S202). For example, the terminal device 10 provides a 3D model showing a shooting position and a shooting direction for shooting a multi-viewpoint image to be shot.

[4. An example of display processing]
The free-viewpoint image (multi-viewpoint image) generated by the above-mentioned processing may be used for information distribution, and the distributed multi-viewpoint image is displayed on the distribution destination terminal device 10. This point will be described with reference to FIGS. 15 to 17.

First, the flow of the distribution process will be described with reference to FIG. FIG. 15 is a diagram showing an example of processing by the information processing system according to the embodiment. The same points as in FIG. 1 and the like will be omitted as appropriate. For example, the terminal device 10 activates an application (for example, a second application 82 or the like) for displaying information about a transaction target, and executes the following processing. First, the terminal device 10 requests the service providing device 100 for information regarding the transaction target (step S21). In the example of FIG. 15, the terminal device 10 transmits a request for information about the transaction target corresponding to the category "shoes".

Then, the service providing device 100 that has received the request from the terminal device 10 selects information about the transaction target corresponding to the category "shoes" from the transaction targets whose information is stored in the storage unit 120 (see FIG. 12) (see FIG. 12). Step S22).

Then, the service providing device 100 delivers information regarding the transaction target corresponding to the selected category “shoes” to the terminal device 10 (step S23). In the example of FIG. 15, the service providing device 100 delivers an image group (hereinafter referred to as “image group IGP”) relating to the shoe including the multi-viewpoint image FV1 of the shoe A to the terminal device 10. For example, the service providing device 100 distributes a multi-viewpoint image FV1 of shoe A and an image group IGP including images IM1 to IM2, IM4 to IM6 of other shoes. For example, in the image group IGP, it is assumed that the images IM1 to IM2 of other shoes, the multi-viewpoint image FV1, and the images IM4 to IM6 of other shoes are in order.

Then, the terminal device 10 that has received the image group IGP displays information about the image group IGP (step S24). For example, the terminal device 10 displays a list of each image included in the image group IGP as shown in FIG. FIG. 16 is a diagram showing an example of displaying a list of transaction targets according to the embodiment. In the terminal device 10, the images IM1 to IM2 of the other shoes and the multi-viewpoint image FV1 of the shoe A are arranged at the uppermost stage, and the images IM4 to IM6 of the other shoes are arranged at the second stage, and the images of the shoes are displayed in a list. The content CT1 to be displayed is displayed. The content CT1 may be generated by the terminal device 10 that has received the image group IGP based on the image group IGP, or may be generated and distributed by the service providing device 100. In the list display, the multi-viewpoint image may be displayed so as to be distinguishable from a normal image. For example, the terminal device 10 attaches an icon or the like indicating that the multi-viewpoint image FV1 is a multi-viewpoint image, and the multi-viewpoint image FV1 is a multi-viewpoint image unlike the other images IM1 to IM2 and IM4 to IM6. You may show that.

Then, the terminal device 10 changes the display from the image list display mode (second display mode) to the display mode for each image (first display mode) according to the operation of the user U1. For example, when the user U1 specifies the image IM1 of another shoe, the display is changed to the first display mode in which only the image IM1 of the other shoe is displayed. Hereinafter, the display switching in the first display mode will be described with reference to FIG. FIG. 17 is a diagram showing an example of displaying an image group including a multi-viewpoint image according to an embodiment. In FIG. 17, the terminal device 10 is distinguished as a terminal device 10-1 to 10-6 according to a change in the display content of the terminal device 10, but when the description is made without particular distinction, it is described as the terminal device 10. do.

In FIG. 17, the terminal device 10-1 shows a state in which only the image IM1 of another shoe is displayed. The terminal device 10 displays a group of images in order according to an operation (swipe operation) in which the user moves a finger in contact with the screen of the terminal device 10 from right to left (forward direction). For example, the terminal device 10 displays only the image IM1 of another shoe, and the terminal device 10 displays only the image IM1 of the other shoe in response to a forward swipe operation by the user U1. The display is switched to the terminal device 10-2 displaying only the image IM2 of another shoe. That is, the terminal device 10 displays the next image in response to the forward swipe operation by the user U1.

The terminal device 10 displays the image group in the reverse order according to the operation (swipe operation) in which the user moves the finger in contact with the screen of the terminal device 10 from left to right (reverse direction). For example, in a state where only the image IM2 of another shoe is displayed, the terminal device 10 displays only the image IM2 of the other shoe in response to a swipe operation in the reverse direction by the user U1. The display is switched to the terminal device 10-1 displaying only the image IM1 of the other shoes. That is, the terminal device 10 displays the previous image in response to the swipe operation in the reverse direction by the user U1.

In FIG. 17, the terminal device 10-2 shows a state in which only the image IM2 of another shoe is displayed. For example, the terminal device 10 displays only the image IM2 of another shoe, and the terminal device 10 displays only the image IM2 of the other shoe in response to a forward swipe operation by the user U1. The display is switched to the terminal device 10-3 displaying only the multi-viewpoint image FV1 of the shoe A. Specifically, the terminal device 10 displays only the image IM2 of the other shoes, and the terminal device 10 displays only the image IM2 of the other shoes in response to the forward swipe operation by the user U1. 2, among the multi-view image FV1 shoes a, switches the display on the terminal apparatus 103 that displays the first image RP1 _1.

Then, the terminal device 10 displays the multi-viewpoint image FV1 from the image RP1 _{1 to the image RP1 2} , the image RP1 ₃ , the image RP1 ₄ , and the image RP1 ₅ in response to the movement of the swipe operation in the forward direction of the user U1. Image RP1 ₆ , image RP1 ₇ , image RP1 ₈ and image RP1 ₉ are displayed in this order in chronological order. 17, the terminal device 10-4, among the multi-view image FV1, shows a display state of the image RP1 _5, the terminal device 10-5, among the multi-view image FV1, displaying the last image RP1 ₉ Indicates the state. Note that, in FIG. 17, among the multi-viewpoint images FV1 for explanation, only three images, the first image RP1 ₁ , the intermediate image RP1 ₅ , and the last image RP1 ₉ are displayed. Although the state is shown in the figure, the other image RP1 ₂ , image RP1 ₃ , image RP1 ₄ , image RP1 ₆ , image RP1 ₇ , and image RP1 ₈ are also in order according to the movement of the swipe operation in the forward direction of the user U1. It is displayed along with.

The terminal device 10 in response to movement of the swipe in the forward direction of the user U1 in the state of the terminal device 10-6 that displays the last image RP1 _9, displays the next image IM4 of the multi-view image FV1 .. Specifically, the terminal device 10 responds to a forward swipe operation by the user U1 in the state of the terminal device 10-5 displaying _{the first image RP1 9} in the multi-viewpoint image FV1 of the shoe A. The display is switched from the terminal device 10-5 displaying only the multi-viewpoint image FV1 of the shoe A to the terminal device 10-6 displaying only the image IM4 of the other shoes. The terminal device 10 appropriately uses existing image processing technology to perform processing such as switching display of an image group including the above-mentioned multi-viewpoint image.

As described above, the terminal device 10 switches and displays the images included in the multi-viewpoint image in response to the operation of switching the display of the image group (swipe operation), thereby displaying the image group including the multi-viewpoint image. The switching operation can be facilitated.

In this way, the terminal device 10 displays the image of the image group in the carousel in the first display mode. The terminal device 10 displays an image from a different viewpoint with respect to the multi-viewpoint image in response to a swipe operation, a slide operation, or the like by the user. If the image being displayed is not a multi-viewpoint image, the terminal device 10 performs normal image switching.

For example, in the second display mode, the terminal device 10 displays only the thumbnail of the first image of the multi-viewpoint image or the thumbnail of the last image of the multi-viewpoint image in a list. Then, when the thumbnail is displayed on the entire screen (first display mode or the like), the terminal device 10 switches and displays the multi-viewpoint image by the operation of switching the display of the image group. Further, the terminal device 10 seamlessly displays another image by the same operation. For example, the terminal device 10 displays the last image and the last image of the multi-viewpoint image in a carousel, and sequentially displays other images of the multi-viewpoint image during the switching. As a result, the terminal device 10 can seamlessly display the carousel without performing the operation of switching the carousel many times. Therefore, the terminal device 10 can seamlessly display a multi-viewpoint image by the same operation as other images in various services for displaying an image in a carousel format.

[4-1. Other display examples]
It should be noted that the above-mentioned display is only an example, and if the terminal device 10 can switch and display the images included in the multi-viewpoint image according to the operation of switching the display of the image group, what are the other display modes? It may be a display mode.

The terminal device 10 may overlay and display the multi-viewpoint image. For example, the terminal device 10 may overlay and display images other than the first and last images among the multi-viewpoint images. Further, when displaying the multi-viewpoint image, the terminal device 10 may cancel the switching display of the image included in the multi-viewpoint image and display a button or the like for transitioning to the display of the previous and next images. When displaying a multi-viewpoint image, the terminal device 10 arranges arrow marks at both left and right ends, and when the arrow mark on the left side is selected, the terminal device 10 displays the next image of the multi-viewpoint image and the arrow mark on the right side. If is selected, the image before the multi-viewpoint image may be displayed. The operation may be reversed, and the terminal device 10 displays the image before the multi-viewpoint image when the arrow mark on the left side is selected, and displays the image before the multi-viewpoint image when the arrow mark on the right side is selected. , The next image of the multi-viewpoint image may be displayed.

Further, the terminal device 10 displays the first image (thumbnail) and the last image (thumbnail) of the multi-viewpoint images, and switches the display from the image selected by the user to the other image in order. A viewpoint image may be displayed. In this case, the terminal device 10 puts the first image (thumbnail) and the last image (thumbnail) of the multi-viewpoint images in the list display area arranged in a part (for example, the lower side) of the screen. indicate. Then, when the user selects one of the first image (thumbnail) and the last image (thumbnail) displayed in the list display area, the terminal device 10 is a part of the screen (thumbnail). For example, in the selection target display area arranged on the upper side), the display may be switched from the image selected by the user to the other image in order to display the multi-viewpoint image. When the user selects the first image, the terminal device 10 may display the multi-viewpoint images in order from the first image to the last image in the selection target display area in chronological order. When the user selects the last image, the terminal device 10 may display the multi-viewpoint images from the last image to the first image in the reverse order of the time series in the selection target display area.

The terminal device 10 displays the first image and the last image in the list display area, and displays the multi-viewpoint image in the selection target display area according to the display state of the selection target display area and the user's designation for the list display area. You may switch. For example, when the terminal device 10 displays the first image in the selection target display area and the user selects the last image in the list display area, the terminal device 10 has multiple viewpoints from the first image to the last image in the selection target display area. The images may be displayed in order in chronological order. Further, when the terminal device 10 displays the last image in the selection target display area and the user selects the first image in the list display area, the terminal device 10 has multiple viewpoints from the last image to the first image in the selection target display area. The images may be displayed in reverse order in chronological order.

[4-2. Information processing flow]
Next, an example of the display processing procedure according to the embodiment will be described with reference to FIG. FIG. 18 is a flowchart showing an example of the flow of display processing according to the embodiment. For example, the terminal device 10 acquires an image group including a first image which is a multi-viewpoint image captured from different viewpoints and a second image different from the first image (step S301). Then, the terminal device 10 switches the image included in the first image and displays it on the screen of the terminal device in response to the operation of switching the display of the image group on the screen of the terminal device by the user who uses the terminal device 10 (step). S302).

[5. others〕
Further, among the processes described in the above-described embodiment, all or a part of the processes described as being automatically performed can be manually performed, and conversely, the processes described as being manually performed. It is also possible to automatically perform all or part of the above by a known method. In addition, information including processing procedures, specific names, various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown in the figure.

Further, each component of each of the illustrated devices is a functional concept, and does not necessarily have to be physically configured as shown in the figure. That is, the specific form of distribution / integration of each device is not limited to the one shown in the figure, and all or part of them may be functionally or physically distributed / physically in arbitrary units according to various loads and usage conditions. Can be integrated and configured.

In addition, the above-described embodiments can be appropriately combined as long as the processing contents do not contradict each other.

[6. program〕
Further, the terminal device 10 and the service providing device 100 according to the above-described embodiment are realized by, for example, a computer 1000 having a configuration as shown in FIG. FIG. 19 is a diagram showing an example of a hardware configuration. The computer 1000 is connected to the output device 1010 and the input device 1020, and the arithmetic unit 1030, the primary storage device 1040, the secondary storage device 1050, the output IF (Interface) 1060, the input IF 1070, and the network IF 1080 are connected by the bus 1090. Have.

The arithmetic unit 1030 operates based on a program stored in the primary storage device 1040 or the secondary storage device 1050, a program read from the input device 1020, or the like, and executes various processes. The primary storage device 1040 is a memory device that temporarily stores data used by the arithmetic unit 1030 for various operations such as RAM. Further, the secondary storage device 1050 is a storage device in which data used by the calculation device 1030 for various calculations and various databases are registered, such as a ROM (Read Only Memory), an HDD (Hard Disk Drive), and a flash memory. Is realized by.

The output IF 1060 is an interface for transmitting information to be output to an output device 1010 that outputs various information such as a monitor and a printer. For example, USB (Universal Serial Bus), DVI (Digital Visual Interface), and the like. It is realized by a connector of a standard such as HDMI (registered trademark) (High Definition Multimedia Interface). Further, the input IF 1070 is an interface for receiving information from various input devices 1020 such as a mouse, a keyboard, a scanner, and the like, and is realized by, for example, USB.

The input device 1020 is, for example, an optical recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), or a tape. It may be a device that reads information from a medium, a magnetic recording medium, a semiconductor memory, or the like. Further, the input device 1020 may be an external storage medium such as a USB memory.

The network IF 1080 receives data from another device via the network N and sends it to the arithmetic unit 1030, and also transmits the data generated by the arithmetic unit 1030 to the other device via the network N.

The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output IF 1060 and the input IF 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040, and executes the loaded program.

For example, when the computer 1000 functions as the terminal device 10, the arithmetic unit 1030 of the computer 1000 realizes the function of the control unit 80 by executing the program loaded on the primary storage device 1040.

[7. effect〕
As described above, the terminal device (terminal device 10 in the embodiment) according to the embodiment has an acquisition unit (first acquisition unit 811 in the embodiment) and a provision unit (first display unit 812 in the embodiment). .. The first acquisition unit 811 acquires the shooting target information regarding the shooting target. The first display unit 812 provides a model (3D model MD in the embodiment) showing a shooting position and a shooting direction for shooting a multi-viewpoint image to be shot.

As described above, the terminal device 10 according to the embodiment provides a model showing a shooting position and a shooting direction for shooting a multi-viewpoint image to be shot. As a result, the terminal device 10 can facilitate shooting for generating a multi-viewpoint image, so that shooting for a multi-viewpoint image can be facilitated.

Further, in the terminal device 10 according to the embodiment, the first display unit 812 displays the model on the screen of the terminal device 10 (output unit 50 in the embodiment). As a result, the terminal device 10 can recognize the shooting position and the shooting direction by the user who uses the terminal device 10, and can facilitate the shooting of the multi-viewpoint image by the user, so that the shooting of the multi-viewpoint image can be easily performed. can do.

Further, in the terminal device 10 according to the embodiment, the first display unit 812 superimposes and displays the model on the screen of the terminal device 10 that displays the shooting area including the shooting target. As a result, the terminal device 10 can easily shoot a multi-viewpoint image by superimposing and displaying the guide on the shooting area.

Further, the terminal device 10 according to the embodiment has a determination unit (determination unit 813 in the embodiment). The determination unit 813 determines the overlap between the central portion of the photographing region of the terminal device 10 and the model. As a result, the terminal device 10 can determine whether the shooting position and the shooting direction are in a state suitable for shooting, so that shooting of a multi-viewpoint image can be facilitated.

Further, in the terminal device 10 according to the embodiment, the first display unit 812 displays an object indicating the central portion of the photographing area. As a result, the terminal device 10 can make the user recognize what kind of state the overlap between the object indicating the central portion of the shooting area and the model is, and can facilitate shooting of a multi-viewpoint image. ..

Further, in the terminal device 10 according to the embodiment, the determination unit 813 determines the overlap between the central portion indicated by the object and the model. As a result, the terminal device 10 can determine whether the shooting position and the shooting direction are in a state suitable for shooting, so that shooting of a multi-viewpoint image can be facilitated.

Further, the terminal device 10 according to the embodiment has a photographing unit (in the embodiment, an photographing unit 814). When the determination unit 813 determines that the overlap between the central portion and the model satisfies a predetermined condition, the photographing unit 814 performs imaging. As a result, the terminal device 10 can take a picture of the multi-viewpoint image only by the user changing the position and orientation of the terminal device 10, so that the picture of the multi-viewpoint image can be facilitated.

Further, in the terminal device 10 according to the embodiment, the determination unit 813 determines that a predetermined condition is satisfied when the range in which the central portion and the model overlap is equal to or greater than a predetermined threshold value. As a result, the terminal device 10 can take a picture of the multi-viewpoint image only by the user changing the position and orientation of the terminal device 10, so that the picture of the multi-viewpoint image can be facilitated.

Further, in the terminal device 10 according to the embodiment, the first display unit 812 displays a model arranged between the viewpoint at which the photographing target is photographed and the gazing point at the photographing object. As a result, the terminal device 10 can facilitate shooting of a multi-viewpoint image by displaying a model between the viewpoint and the gazing point.

Further, in the terminal device 10 according to the embodiment, the first display unit 812 displays a model arranged between the gazing point determined based on the line of sight from the viewpoint and the terminal device 10. As a result, the terminal device 10 can facilitate shooting of a multi-viewpoint image by displaying a model between the gazing point determined based on the viewpoint and the terminal device 10.

Further, in the terminal device 10 according to the embodiment, the first display unit 812 has a gaze point determined based on the first line of sight from the first viewpoint and the second line of sight from the second viewpoint, and the terminal device 10. Display the model placed between. As a result, the terminal device 10 can facilitate shooting of a multi-viewpoint image by displaying a model between the gazing point determined based on the line of sight from the two viewpoints and the terminal device 10.

Further, in the terminal device 10 according to the embodiment, the first display unit 812 displays a shooting position guide indicating a plurality of positions for shooting a multi-viewpoint image based on the gazing point. Thereby, the terminal device 10 can facilitate the shooting of the multi-viewpoint image by the user by displaying the shooting position guides indicating a plurality of positions for shooting the multi-viewpoint image.

Further, in the terminal device 10 according to the embodiment, the first display unit 812 displays a shooting position guide in the shape of a fan centered on the gazing point. Thereby, the terminal device 10 can facilitate the shooting of the multi-viewpoint image by the user by displaying the shooting position guide in the shape of a fan.

Further, in the terminal device 10 according to the embodiment, the first display unit 812 displays a shooting position guide in which each of the plurality of positions is arranged on the arc of the fan. As a result, the terminal device 10 can facilitate the user to take a picture of the multi-viewpoint image by arranging and displaying each of the plurality of positions on the arc of the fan.

Further, in the terminal device 10 according to the embodiment, the first display unit 812 superimposes and displays a transparent shooting position guide on the screen of the terminal device 10 that displays the shooting area including the transaction target. As a result, the terminal device 10 can facilitate the user to take a picture of the multi-viewpoint image.

Further, in the terminal device 10 according to the embodiment, the first display unit 812 can distinguish between the shooting completed position where the shooting is completed and the shooting completed position where the shooting is not completed among the plurality of positions. .. As a result, the terminal device 10 can recognize whether or not the user has completed the shooting at each position, and can facilitate the shooting of the multi-viewpoint image by the user.

The embodiments of the present application have been described in detail with reference to the drawings, but this is an example, and various modifications and improvements are made based on the knowledge of those skilled in the art, including the embodiments described in the disclosure column of the invention. It is possible to carry out the present invention in other forms described above.

Further, the above-mentioned "section, module, unit" can be read as "means" or "circuit". For example, the reception unit can be read as a reception means or a reception circuit.

1 Information processing system 10 Terminal equipment 20 Communication unit 30 Storage unit 31 Transaction target information storage unit 40 Input unit 50 Output unit 60 Physical sensor 70 Camera 80 Control unit 81 1st application 811 1st acquisition unit 812 1st display unit (providing unit) )
813 Judgment unit 814 Imaging unit 815 Generation unit 816 Transmission unit 82 Second application 821 Second acquisition unit 822 Second display unit 823 Reception unit 100 Service provider 110 Communication unit 120 Storage unit 130 Control unit 131 Reception unit 132 Processing unit 133 Distribution Department

Claims (13)

The acquisition procedure for acquiring shooting target information related to the shooting target, and
A procedure for superimposing a model indicating a shooting position and a shooting direction for shooting a multi-viewpoint image of a shooting target on a screen of a terminal device displaying a shooting area including the shooting target, and a procedure for providing the model.
A determination procedure for determining the overlap between the central portion of the photographing region of the terminal device and the model, and
When it is determined by the determination procedure that the overlap between the central portion and the model satisfies a predetermined condition, the imaging procedure for performing imaging and the imaging procedure are performed.
It was performed to the terminal device,
The determination procedure is
If the range between the center and the model overlap is above a predetermined threshold, the information processing program characterized that you determined that the predetermined condition is satisfied. The above-mentioned provision procedure
The information processing program according to claim 1 , wherein an object indicating the central portion of the photographing area is displayed. The determination procedure is
The information processing program according to claim 2 , wherein the overlap between the central portion indicated by the object and the model is determined. The above-mentioned provision procedure
The information processing program according to any one of claims 1 to 3 , wherein the model is displayed between the viewpoint for photographing the image target and the gazing point in the image target. The above-mentioned provision procedure
The information processing program according to claim 4 , wherein the model is displayed between the gaze point determined based on the line of sight from the viewpoint and the terminal device. The above-mentioned provision procedure
A claim characterized by displaying the model arranged between the gaze point determined based on the first line of sight from the first viewpoint and the second line of sight from the second viewpoint and the terminal device. Item 5. The information processing program according to item 5. The above-mentioned provision procedure
The information processing program according to any one of claims 4 to 6 , wherein a shooting position guide indicating a plurality of positions for shooting the multi-viewpoint image based on the gazing point is displayed. The above-mentioned provision procedure
The information processing program according to claim 7 , wherein the shooting position guide, which is in the shape of a fan centered on the gazing point, is displayed. The above-mentioned provision procedure
The information processing program according to claim 8 , wherein each of the plurality of positions displays the shooting position guide arranged on the arc of the fan. The above-mentioned provision procedure
The information processing according to any one of claims 7 to 9 , wherein the shooting position guide having transparency is superimposed and displayed on the screen of the terminal device that displays the shooting area including the shooting target. program. The above-mentioned provision procedure
Among the plurality of positions, any one of claims 7 to 10 , wherein the shooting completed position where the shooting is completed and the shooting completed position where the shooting is not completed are displayed in an distinguishable manner. The information processing program described. An acquisition unit that acquires shooting target information related to the shooting target,
A providing unit that superimposes and displays a model indicating a shooting position and a shooting direction for shooting a multi-viewpoint image of a shooting target on a screen of a terminal device that displays a shooting area including the shooting target.
A determination unit for determining the overlap between the central portion of the photographing region of the terminal device and the model.
When the determination unit determines that the overlap between the central portion and the model satisfies a predetermined condition, the photographing unit for performing imaging and the photographing unit
Have a,
The determination unit
A terminal device characterized in that when the range in which the central portion and the model overlap is equal to or greater than a predetermined threshold value, it is determined that the predetermined condition is satisfied. The acquisition process to acquire the shooting target information related to the shooting target,
A step of providing a model showing a shooting position and a shooting direction for shooting a multi-viewpoint image of a shooting target by superimposing the model on a screen of a terminal device displaying a shooting area including the shooting target.
A determination step for determining the overlap between the central portion of the photographing region of the terminal device and the model, and
When it is determined by the determination step that the overlap between the central portion and the model satisfies a predetermined condition, the imaging step of performing imaging and the imaging step
Seen including,
The determination step is
An information processing method characterized in that when the range in which the central portion and the model overlap is equal to or greater than a predetermined threshold value, it is determined that the predetermined condition is satisfied.

Images