JP2019185757A - Image processing device, imaging system, image processing method, and program - Google Patents

Abstract

To solve the problem that when an attempt is made to superimpose a plane image P obtained by imaging separately from an entire celestial sphere image CE on some region of the entire celestial sphere image CE, an image misalignment occurs because an equidistant cylindrical projection image EC (entire celestial sphere image CE) and a plane image P are different in a projection scheme.SOLUTION: An image processing device calculates a region CA1 that corresponds to an image P in an image EC, specifies a peripheral region image PI that includes the region CA1, and converts a superimposing image S to the projection scheme of the image P. The image processing device calculates a region CA02 that corresponds to the image P in an image S' after conversion, finds a region CA2 on the basis of a plurality of corresponding blocks in a third image obtained by dividing a second image into blocks and matching to a third image, and inversely converts the projection scheme of the region CA2 to the projection scheme of the image EC. The image processing device calculates a third corresponding region CA3 that corresponds to the region CA2 in the image EC, and superimposes the image P on the region CA3.SELECTED DRAWING: Figure 21

Description

  The present invention relates to an image processing apparatus, a photographing system, an image processing method, and a program.

  Conventionally, an enlarged planar image obtained by photographing separately from a wide-angle planar image is inserted into a partial area of the wide-angle planar image, so that the partial area is enlarged even when the partial area is enlarged. A technique capable of displaying a simple image is disclosed (see Patent Document 1).

  By the way, in recent years, there has been provided a special digital camera that obtains two hemispherical image data based on a 360-degree omnidirectional image by one shooting (see Patent Document 2). This digital camera creates one equirectangular projection image data based on two hemispherical image data, and transmits the equirectangular projection image data to a communication terminal such as a smartphone. The communication terminal that has obtained the equirectangular projection image data creates an omnidirectional image based on the equirectangular projection image data. However, since the image is curved as it is, it is difficult for the user to see. Therefore, by displaying a predetermined area image indicating a predetermined area of the omnidirectional image on the communication terminal, the user is photographed with a general digital camera. Can be viewed with the same feeling as a flat image.

  However, for example, in the case where a planar image obtained by shooting separately from the omnidirectional image is superimposed on a partial area of the omnidirectional image, one image having a different projection method is combined with the other image. When the images are superimposed or the like, there arises a problem that the image shifts.

  The invention according to claim 1 is an acquisition means for acquiring a first image of the first projection method and a second image of a second projection method different from the first projection method; First projection method conversion means for converting an image into the second projection method and generating a third image, and extraction means for extracting a plurality of feature points from the second image and the third image, respectively. And a second corresponding to the second image in the third image based on the plurality of feature points of the second image and the plurality of feature points of the third image respectively extracted by the extraction means Based on a plurality of blocks in the third image obtained by dividing the second image into blocks and matching the third image with corresponding region calculation means for obtaining a corresponding region, the second correspondence Corresponding area correction means for correcting the area, and the correction A plurality of predetermined points included in the second corresponding area of the second projection method conversion unit that converts the plurality of predetermined points into the first projection method, and the plurality of the predetermined plurality of points converted by the second projection method conversion unit. Position calculating means for obtaining position information of a point in the first image, and storage means for storing the position information obtained by the position calculating means in association with a plurality of points of the second image. An image processing apparatus is characterized.

  As described above, according to the present invention, even if the other image is combined with one image having a different projection method, an effect of suppressing image shift can be achieved.

(A) is a left side view of the special photographing apparatus, (b) is a rear view of the special photographing apparatus, (c) is a plan view of the special photographing apparatus, and (d) is a bottom view of the special photographing apparatus. It is. It is a usage image figure of a special imaging device. (A) is a hemispheric image (front) photographed by a special photographing device, (b) is a hemispheric image (rear) photographed by a special photographing device, and (c) is an image represented by equirectangular projection. FIG. (A) is the conceptual diagram which showed the state which covered the sphere with an equirectangular projection image, (b) is the figure which showed the omnidirectional image. It is the figure which showed the position of a virtual camera and a predetermined area | region at the time of making a spherical image into a three-dimensional solid sphere. (A) is a three-dimensional perspective view of FIG. 5, (b) is a figure which shows the state in which the image of the predetermined area is displayed on the display of a communication terminal. FIG. 6 is a diagram illustrating a relationship between predetermined area information and an image of a predetermined area T. 1 is a schematic diagram of an imaging system according to a first embodiment of the present invention. It is a perspective view of an adapter. It is a usage image figure of an imaging system. It is a hardware block diagram of a special imaging device. It is a hardware block diagram of a general imaging device. It is a hardware block diagram of a smart phone. It is a functional block diagram of the imaging system concerning a 1st embodiment. (A) is a conceptual diagram of a cooperative photographing device management table, (b) is a conceptual diagram showing a cooperative photographing device setting screen. It is a detailed functional block diagram of an image / sound processing unit according to the first embodiment. It is detail drawing of a corresponding | compatible area correction | amendment part. It is a block diagram of superimposition display metadata. (A) is a conceptual diagram showing each lattice region in the second corresponding region, (b) is a conceptual diagram showing each lattice region in the third corresponding region. It is the sequence diagram which showed the imaging | photography method which concerns on 1st Embodiment. It is a conceptual diagram of the image in the process of the production | generation process of a superimposed display parameter. It is a conceptual diagram at the time of specifying a peripheral region image. It is a conceptual diagram conceptual diagram showing processing of a corresponding area correction unit. It is a figure explaining the concept of a motion vector. FIG. 7 is a diagram illustrating correction processing based on similarity and luminance variance values, where (a) illustrates the effectiveness X based on similarity, and (b) illustrates the effectiveness Y based on luminance variance. It is a figure explaining the concept of the process which correct | amends a representative point with the motion vector after correction | amendment. It is the conceptual diagram which showed the other process of the corresponding area correction | amendment part. FIG. 10 is a diagram showing all representative points when the second corresponding area CA02 is divided by the same number as the planar image P. It is a figure explaining the concept regarding correction | amendment of a motion vector. It is a figure explaining the concept regarding correction | amendment of a motion vector, (a) is the correction position in a non-shared point, (b) is the correction position of the shared point in two blocks, (c) is correction | amendment of the shared point in four blocks. It is a conceptual diagram which shows each position. It is the figure which showed the superimposition synthetic | combination position before a correction | amendment at the time of using the valid / invalid determination of a block matching and correction | amendment. It is a conceptual diagram at the time of dividing | segmenting a 2nd corresponding area | region into a some grid | lattice area | region. It is a conceptual diagram which shows the 3rd corresponding | compatible area | region in the equirectangular projection image EC. It is a conceptual diagram of the image in the process of the preparation process of a correction parameter. It is a conceptual diagram of the image in the process of the superimposition process. It is a two-dimensional conceptual diagram when a planar image is superimposed on an omnidirectional image. It is a three-dimensional conceptual diagram when a planar image is superimposed on an omnidirectional image. It is a two-dimensional conceptual diagram when a planar image is superimposed on an omnidirectional image without using the position parameter of the present embodiment. It is a two-dimensional conceptual diagram when a planar image is superimposed on an omnidirectional image using the position parameters of the present embodiment. (A) Display example of wide image without superimposing display, (b) Display example of tele image without superimposing display, (c) Display example of wide image with superimposing display, (d) In case of superimposing display It is the conceptual diagram which showed the example of a display of a tele image. It is the schematic of the imaging | photography system which concerns on the 2nd Embodiment of this invention. It is a hardware block diagram of an image processing server. It is a functional block diagram of the imaging system which concerns on 2nd Embodiment. It is a detailed functional block diagram of an image / sound processing unit according to the second embodiment. It is the sequence diagram which showed the imaging | photography method which concerns on 2nd Embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the omnidirectional image described later is an example of the first image, and the superimposed image is an example of the second image. The peripheral area image is an example of a third image.

[Outline of Embodiment]
Hereinafter, an outline of the present embodiment will be described.

  First, a method for generating an omnidirectional image will be described with reference to FIGS.

  First, the external appearance of the special photographing apparatus 1 will be described with reference to FIG. The special photographing apparatus 1 is a digital camera for obtaining a photographed image that is the basis of a panoramic image of a celestial sphere (360 °). 1A is a left side view of the special photographing apparatus, FIG. 1B is a rear view of the special photographing apparatus, and FIG. 1C is a plan view of the special photographing apparatus. (D) is a bottom view of the special photographing apparatus.

  As shown in FIGS. 1 (a), 1 (b), 1 (c), and (d), a fish-eye lens is placed on the front side (front side) of the special photographing apparatus 1 at the top. A fisheye lens 102b is provided on the rear side (rear side) 102a. Inside the special imaging device 1, imaging elements (image sensors) 103a and 103b, which will be described later, are provided, and a hemispherical image (angle of view 180 °) is obtained by photographing a subject or a landscape through the lenses 102a and 102b, respectively. Above). A shutter button 115 a is provided on the surface opposite to the front side of the special imaging device 1. In addition, a power button 115b, a Wi-Fi (Wireless Fidelity) button 115c, and a shooting mode switching button 115d are provided on the side surface of the special imaging device 1. Each time the power button 115b and the Wi-Fi button 115c are pressed, they are switched on and off. The shooting mode switching button 115d switches between a still image shooting mode and a moving image shooting mode each time the button is pressed. Note that the shutter button 115a, the power button 115b, the Wi-Fi button 115c, and the shooting mode switching button 115d are part of the operation unit 115, and the operation unit 115 is not limited to these buttons.

  A tripod screw hole 151 for attaching the special photographing device 1 or the general photographing device 3 to a camera tripod is provided at the center of the bottom 150 of the special photographing device 1. A micro USB (Universal Serial Bus) terminal 152 is provided on the left end side of the bottom 150. An HDMI (High-Definition Multimedia Interface) terminal 153 is provided on the right end side of the bottom 150. HDMI is a registered trademark.

  Next, the use situation of the special photographing apparatus 1 will be described with reference to FIG. In addition, FIG. 2 is a use image figure of a special imaging device. As shown in FIG. 2, the special imaging device 1 is used, for example, for a user to take a picture of a subject around the user with his / her hand. In this case, two hemispherical images can be obtained by imaging the subject around the user by the imaging device 103a and the imaging device 103b shown in FIG.

  Next, the outline of processing from the image captured by the special imaging device 1 to the creation of the equirectangular projection image EC and the omnidirectional image CE will be described with reference to FIGS. 3 and 4. 3A is a hemispheric image (front side) photographed by the special photographing apparatus 1, FIG. 3B is a hemispheric image photographed by the special photographing apparatus (rear side), and FIG. It is the figure which showed the image (henceforth "an equirectangular projection image") represented by the cylindrical projection. FIG. 4A is a conceptual diagram showing a state where a sphere is covered with an equirectangular projection image, and FIG. 4B is a diagram showing an omnidirectional image.

  As shown in FIG. 3A, the image obtained by the image sensor 103a is a hemispherical image (front side) curved by a fish-eye lens 102a described later. Also, as shown in FIG. 3B, the image obtained by the image sensor 103b is a hemispherical image (rear side) curved by a fish-eye lens 102b described later. Then, the hemispherical image (front side) and the hemispherical image reversed 180 degrees (rear side) are synthesized by the special imaging device 1, and as shown in FIG. 3C, the equirectangular projection image EC. Is created.

  Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), as shown in FIG. 4A, the equirectangular projection image is pasted so as to cover the spherical surface, and FIG. An omnidirectional image CE as shown in b) is created. In this way, the omnidirectional image CE is represented as an image in which the equirectangular projection image EC faces the center of the sphere. OpenGL ES is a graphics library used for visualizing 2D (2-Dimensions) and 3D (3-Dimensions) data. Note that the omnidirectional image CE may be a still image or a moving image.

  As described above, since the omnidirectional image CE is an image that is pasted so as to cover the spherical surface, it is uncomfortable when viewed by a human. Therefore, by displaying a predetermined area (hereinafter referred to as “predetermined area image”) of a part of the omnidirectional image CE as a flat image with little curvature, a display that does not give a sense of incongruity to humans can be achieved. This will be described with reference to FIGS.

  FIG. 5 is a diagram showing the positions of the virtual camera and the predetermined area when the omnidirectional image is a three-dimensional solid sphere. The virtual camera IC corresponds to the position of the viewpoint of the user who views the omnidirectional image CE displayed as a three-dimensional solid sphere. FIG. 6A is a three-dimensional perspective view of FIG. 5, and FIG. 6B is a diagram showing a predetermined area image when displayed on the display. In FIG. 6A, the omnidirectional image CE shown in FIG. 4 is represented by a three-dimensional solid sphere CS. Assuming that the omnidirectional image CE generated in this way is a three-dimensional sphere CS, the virtual camera IC is positioned inside the omnidirectional image CE as shown in FIG. The predetermined area T in the omnidirectional image CE is a shooting area of the virtual camera IC, and is specified by predetermined area information indicating the shooting direction and angle of view of the virtual camera IC in the three-dimensional virtual space including the omnidirectional image CE. The

  Then, the predetermined area image Q shown in FIG. 6A is displayed on the predetermined display as an image of the photographing area of the virtual camera IC as shown in FIG. 6B. The image shown in FIG. 6B is a predetermined area image represented by the predetermined (default) predetermined area information. The following description will be made using the shooting direction (ea, aa) and the angle of view (α) of the virtual camera IC.

  The relationship between the predetermined area information and the image of the predetermined area T will be described with reference to FIG. FIG. 7 is a diagram showing the relationship between the predetermined region information and the relationship between the images of the predetermined region T. As shown in FIG. 7, “ea” represents an elevation angle, “aa” represents an azimuth angle, and “α” represents an angle of view. That is, the attitude of the virtual camera IC is changed so that the gazing point of the virtual camera IC indicated by the shooting direction (ea, aa) becomes the center point CP of the predetermined area T that is the shooting area of the virtual camera IC. Become. The predetermined area image Q is an image of the predetermined area T in the omnidirectional image CE. f is the distance from the virtual camera IC to the center point CP. L is a distance between an arbitrary vertex of the predetermined region T and the center point CP (2L is a diagonal line). In FIG. 7, a trigonometric function represented by the following (formula 1) is generally established.

L / f = tan (α / 2) (Formula 1)
[First Embodiment]
Subsequently, a first embodiment of the present invention will be described with reference to FIGS.

<Outline of shooting system>
First, an outline of the configuration of the photographing system of the present embodiment will be described with reference to FIG. FIG. 8 is a schematic diagram of the configuration of the photographing system of the present embodiment.

  As shown in FIG. 8, the photographing system of this embodiment includes a special photographing device 1, a general photographing device 3, a smartphone 5, and an adapter 9. The special photographing device 1 is connected to the general photographing device 3 through an adapter 9.

  Among these, the special photographing device 1 is a special digital camera for photographing a subject, a landscape, and the like to obtain two hemispherical images that are the basis of the omnidirectional (panoramic) image as described above.

  The general photographing device 3 is a digital single-lens reflex camera, but may be a compact digital camera. The general photographing apparatus 3 is provided with a shutter button 315a which is a part of an operation unit 315 described later.

  The smartphone 5 can perform wireless communication with the special imaging device 1 and the general imaging device 3 using a short-range wireless communication technology such as Wi-Fi, Bluetooth (registered trademark), or NFC (Near Field Communication). Further, the smartphone 5 can display images respectively acquired from the special imaging device 1 and the general imaging device 3 on a display 517 described later provided in the device itself.

  Note that the smartphone 5 may communicate with the special imaging device 1 and the general imaging device 3 through a wired cable without using the short-range wireless communication technology. The smartphone 5 is an example of an image processing apparatus, and the image processing apparatus includes a tablet PC (Personal Computer), a notebook PC, and a desktop PC. Note that the smartphone is also an example of a communication terminal described later.

  FIG. 9 is a perspective view of the adapter. As shown in FIG. 9, the adapter 9 includes a shoe adapter 901, a bolt 902, an upper adjuster 903, and a lower adjuster 904. Among these, the shoe adapter 901 is attached to the accessory shoe of the general photographing apparatus 3 by sliding. At the center of the shoe adapter 901, a bolt 902 that is rotatably attached to the tripod screw hole 151 is provided. The bolt 902 is provided with an upper adjuster 903 and a lower adjuster 904 so as to be rotatable. The upper adjuster 903 plays a role of fixing an object (for example, the special photographing apparatus 1) attached to the bolt 902. The lower adjuster 904 plays a role of fixing an object (for example, the general photographing apparatus 3) to which the shoe adapter 901 is attached.

  FIG. 10 is a usage image diagram of the imaging system. As shown in FIG. 10, the user takes a picture of a subject or the like with the general photographing device 3 to which the special photographing device 1 is attached using the adapter 9 by putting the smartphone 5 in a pocket of clothes. Note that the smartphone 5 may be placed in a range where wireless communication with the special imaging device 1 or the general imaging device 3 is possible without being put in a pocket of clothes.

<< Hardware Configuration of Embodiment >>
Next, the hardware configuration of the special imaging device 1, the general imaging device 3, and the smartphone 5 according to the present embodiment will be described in detail with reference to FIGS.

<Hardware configuration of special imaging device>
First, the hardware configuration of the special photographing apparatus 1 will be described with reference to FIG. FIG. 11 is a hardware configuration diagram of the special imaging apparatus 1. In the following, the special imaging device 1 is an omnidirectional (omnidirectional) special imaging device using two image sensors, but any number of two or more image sensors may be used. In addition, it is not always necessary to use a dedicated device for omnidirectional photography. By attaching a retrofit omnidirectional imaging unit to a normal digital camera or smartphone, it has substantially the same function as the special photography device 1. Also good.

  As shown in FIG. 11, the special imaging apparatus 1 includes an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, and a ROM (Read Only). Memory) 112, SRAM (Static Random Access Memory) 113, DRAM (Dynamic Random Access Memory) 114, operation unit 115, network I / F 116, communication unit 117, antenna 117a, electronic compass 118, gyro sensor 119, acceleration sensor 120, And a terminal 121.

  Among these, the imaging unit 101 includes wide-angle lenses (so-called fish-eye lenses) 102a and 102b each having an angle of view of 180 ° or more for forming a hemispherical image, and two imaging units provided corresponding to the wide-angle lenses. Elements 103a and 103b are provided. The image sensors 103a and 103b are image sensors such as a CMOS (Complementary Metal Oxide Semiconductor) sensor and a CCD (Charge Coupled Device) sensor that convert an optical image obtained by the fisheye lenses 102a and 102b into image data of an electric signal and output the image data. A timing generation circuit for generating a horizontal or vertical synchronization signal, a pixel clock, and the like, and a register group in which various commands and parameters necessary for the operation of the image sensor are set.

  The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. On the other hand, the imaging elements 103a and 103b of the imaging unit 101 are connected to the imaging control unit 105 through a serial I / F bus (I2C bus or the like). The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to the CPU 111 via the bus 110. Further, ROM 112, SRAM 113, DRAM 114, operation unit 115, network I / F 116, communication unit 117, and electronic compass 118 are connected to the bus 110.

  The image processing unit 104 takes in the image data output from the image sensors 103a and 103b through the parallel I / F bus, performs predetermined processing on the respective image data, and then combines these image data. The data of the equirectangular projection image as shown in FIG. 3C is created.

  In general, the imaging control unit 105 sets a command or the like in a register group of the imaging elements 103a and 103b using the I2C bus with the imaging control unit 105 as a master device and the imaging elements 103a and 103b as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to capture status data and the like of the register groups of the imaging elements 103a and 103b and send them to the CPU 111.

  The imaging control unit 105 instructs the imaging elements 103a and 103b to output image data at the timing when the shutter button of the operation unit 115 is pressed. Some special imaging devices 1 may have a preview display function or a function corresponding to a moving image display by a display (for example, the display 517 of the smartphone 5). In this case, output of image data from the image sensors 103a and 103b is continuously performed at a predetermined frame rate (frame / min).

  Further, as will be described later, the imaging control unit 105 also functions as a synchronization control unit that synchronizes the output timing of image data of the imaging elements 103a and 103b in cooperation with the CPU 111. In the present embodiment, the special imaging apparatus 1 is not provided with a display, but a display unit may be provided.

  The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 through the I / F bus and performs predetermined processing on the sound data.

  The CPU 111 controls the overall operation of the special photographing apparatus 1 and executes necessary processes. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are work memories, and store programs executed by the CPU 111, data being processed, and the like. In particular, the DRAM 114 stores image data being processed by the image processing unit 104 and processed equirectangular projection image data.

  The operation unit 115 is a general term for operation buttons such as the shutter button 115a. The user operates the operation unit 115 to input various shooting modes and shooting conditions.

  The network I / F 116 is a general term for an interface circuit (USB I / F or the like) with an external medium such as an SD card or a personal computer. The network I / F 116 may be wireless or wired. Data of the equirectangular projection image stored in the DRAM 114 is recorded on an external medium via the network I / F 116, or an external terminal (device) such as the smartphone 5 via the network I / F 116 as necessary. ).

  The communication unit 117 communicates with an external terminal (device) such as the smartphone 5 through the antenna 117a provided in the special imaging device 1 by a short-range wireless communication technology such as Wi-Fi, NFC, or Bluetooth. The communication unit 117 can also transmit the equirectangular projection image data to an external terminal (device) such as the smartphone 5.

  The electronic compass 118 calculates the azimuth of the special imaging device 1 from the earth's magnetism and outputs azimuth information. This orientation information is an example of related information (metadata) along Exif, and is used for image processing such as image correction of a captured image. Note that the related information includes each data of the image capturing date and time and the data capacity of the image data.

  The gyro sensor 119 is a sensor that detects a change in angle (Roll angle, Pitch angle, Yaw angle) accompanying the movement of the omnidirectional camera 20. The change in angle is an example of related information (metadata) along Exif, and is used for image processing such as image correction of a captured image.

  The acceleration sensor 120 is a sensor that detects acceleration in three axial directions. The special imaging device 3a calculates the attitude (angle with respect to the direction of gravity) of the own device (special imaging device 3a) based on the acceleration detected by the acceleration sensor 120. By providing both the gyro sensor 119 and the acceleration sensor 120 in the special imaging device 3a, the accuracy of image correction is improved.

  The terminal 121 is a concave terminal for Micro USB.

<Hardware configuration of general imaging device>
Next, the hardware of the general photographing apparatus will be described with reference to FIG. FIG. 12 is a hardware configuration diagram of the general photographing apparatus 3. As shown in FIG. 12, the general imaging apparatus 3 includes an imaging unit 301, an image processing unit 304, an imaging control unit 305, a microphone 308, a sound processing unit 309, a bus 310, a CPU 311, a ROM 312, an SRAM 313, a DRAM 314, an operation. The unit 315 includes a network I / F 316, a communication unit 317, an antenna 317 a, an electronic compass 318, and a display 319. The image processing unit 304 and the imaging control unit 305 are connected to the CPU 311 via the bus 310.

  The configurations 304, 310, 311, 312, 313, 314, 315, 316, 317, 317a, and 318 are the configurations 104, 110, 111, 112, 113, 114, and 115 in the special imaging apparatus 1 of FIG. , 116, 117, 117a, and 118, the description thereof is omitted.

  Furthermore, as shown in FIG. 12, the photographing unit 301 of the general photographing apparatus 3 is provided with a lens unit 306 and a mechanical shutter 307 in order from the outside in the direction of the image sensor 303 on the front surface of the image sensor 303. .

  The imaging control unit 305 basically performs the same configuration and processing as the imaging control unit 105, but further drives the lens unit 306 and the mechanical shutter 307 based on a user operation received by the operation unit 315. To control.

  The display 319 is an example of a display unit that displays an operation menu and an image during or after shooting.

<Smartphone hardware configuration>
Next, the hardware of the smartphone will be described with reference to FIG. FIG. 13 is a hardware configuration diagram of the smartphone. As illustrated in FIG. 13, the smartphone 5 includes a CPU 501, a ROM 502, a RAM 503, an EEPROM 504, a CMOS sensor 505, an image sensor I / F 513 a, an acceleration / direction sensor 506, a media I / F 508, and a GPS receiving unit 509. Yes.

  Among these, the CPU 501 controls the operation of the entire smartphone 5. The ROM 502 stores programs used for driving the CPU 501 such as the CPU 501 and IPL (Initial Program Loader). The RAM 503 is used as a work area for the CPU 501. The EEPROM 504 reads or writes various data such as a smartphone program under the control of the CPU 501. The CMOS sensor 505 captures a subject (mainly a self-portrait) under the control of the CPU 501 and obtains image data. The image sensor I / F 513 a is a circuit that controls driving of the CMOS sensor 512. The acceleration / direction sensor 506 is various sensors such as an electronic magnetic compass, a gyrocompass, and an acceleration sensor that detect geomagnetism. A media I / F 508 controls reading or writing (storage) of data with respect to a recording medium 507 such as a flash memory. The GPS receiver 509 receives GPS signals from GPS satellites.

  The smartphone 5 includes a long-distance communication circuit 511, an antenna 511a, a CMOS sensor 512, an image sensor I / F 513b, a microphone 514, a speaker 515, a sound input / output I / F 516, a display 517, an external device connection I / F 518, and a short distance. A communication circuit 519, an antenna 519a of the short-range communication circuit 519, and a touch panel 521 are provided.

  Among these, the long-distance communication circuit 511 is a circuit that communicates with other devices via a communication network 100 described later. The CMOS sensor 512 is a kind of built-in imaging unit that captures an image of a subject under the control of the CPU 501 and obtains image data. The image sensor I / F 513b is a circuit that controls driving of the CMOS sensor 512. The microphone 514 is a kind of built-in sound collecting means for inputting sound. The sound input / output I / F 516 is a circuit that processes input / output of a sound signal between the microphone 514 and the speaker 515 under the control of the CPU 501. The display 517 is a kind of display means such as a liquid crystal display or an organic EL display that displays a subject image, various icons, and the like. The external device connection I / F 518 is an interface for connecting various external devices. The short-range communication circuit 519 is a communication circuit such as Wi-Fi, NFC, or Bluetooth. The touch panel 521 is a kind of input means for operating the smartphone 5 when the user presses the display 517.

  In addition, the smartphone 5 includes a bus line 510. The bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501.

  Note that a recording medium such as an HD (Hard Disk) or a CD-ROM in which each of the above programs is stored can be provided domestically or abroad as a program product.

<< Functional Configuration of Embodiment >>
Next, the functional configuration of the present embodiment will be described with reference to FIGS. FIG. 14 is a functional block diagram of the special imaging device 1, the general imaging device 3, and the smartphone 5 that constitute a part of the imaging system of the present embodiment.

<Functional configuration of special imaging device>
First, the functional configuration of the special imaging device 1 will be described in detail with reference to FIGS. 11 and 14. As illustrated in FIG. 14, the special imaging device 1 includes a reception unit 12, an imaging unit 13, a sound collection unit 14, an image / sound processing unit 15, a determination unit 17, a short-range communication unit 18, and a storage / readout. It has a part 19. Each of these units is a function realized by any of the constituent elements shown in FIG. 11 being operated by a command from the CPU 111 in accordance with a program for the special photographing apparatus developed from the SRAM 113 onto the DRAM 114, or Means.

  In addition, the special photographing apparatus 1 includes a storage unit 1000 constructed by the ROM 112, the SRAM 113, and the DRAM 114 shown in FIG.

(Functional configuration of special imaging equipment)
Next, using FIG. 11 and FIG. 14, each functional configuration of the special imaging device 1 will be described in more detail.

  The reception unit 12 of the special imaging device 1 is mainly realized by the processing of the operation unit 115 and the CPU 111 illustrated in FIG. 11 and receives operation input from the user.

  The imaging unit 13 is realized mainly by the processing of the imaging unit 101, the image processing unit 104, the imaging control unit 105, and the CPU 111 shown in FIG. obtain. As shown in FIGS. 3A and 3B, the captured image data is two hemispherical image data that is the basis of the omnidirectional image data.

  The sound collecting unit 14 is realized by the processing of the CPU 108, the sound processing unit 109, and the CPU 111 shown in FIG. 11, and collects sounds around the special photographing apparatus 1.

  The image / sound processing unit 15 is realized mainly by a command from the CPU 111 and performs various processes on the captured image data obtained by the imaging unit 13 or the sound data obtained by the sound collecting unit 14. For example, the image / sound processing unit 15 performs equirectangular projection image data based on two hemispherical image data (see FIGS. 3A and 3B) obtained by the two image sensors 103a and 103b, respectively. (See FIG. 3C).

  The determination unit 17 is realized by the processing of the CPU 111 and makes various determinations.

  The short-range communication unit 18 is mainly realized by the processing of the CPU 111, the communication unit 117, and the antenna 117a, and communicates with the short-range communication unit 58 of the smartphone 5 by a short-range wireless communication technique such as Wi-Fi. Can do.

  The storage / reading unit 19 is realized mainly by the processing of the CPU 111 shown in FIG. 11, and stores various data (or information) in the storage unit 1000 or stores various data (or information) from the storage unit 1000. Read out.

<Functional configuration of general imaging device>
Next, the functional configuration of the general imaging device 3 will be described in detail with reference to FIGS. 12 and 14. As shown in FIG. 14, the general imaging device 3 includes a reception unit 32, an imaging unit 33, a sound collection unit 34, an image / sound processing unit 35, a display control unit 36, a determination unit 37, and a short-range communication unit 38. And a storage / reading unit 39. Each of these units is a function realized by any one of the constituent elements shown in FIG. 12 being operated by a command from the CPU 311 according to a program for the special photographing device developed from the SRAM 313 to the DRAM 314, or Means.

  Further, the general photographing apparatus 3 has a storage unit 3000 constructed by the ROM 312, the SRAM 313, and the DRAM 314 shown in FIG.

(Functional configuration of general imaging equipment)
The reception unit 32 of the general imaging device 3 is mainly realized by the processing of the operation unit 315 and the CPU 311 illustrated in FIG. 12, and receives operation input from the user.

  The imaging unit 33 is mainly realized by the processing of the imaging unit 301, the image processing unit 304, the imaging control unit 305, and the CPU 311 shown in FIG. obtain. This photographed image data is planar image data photographed by the perspective projection method.

  The sound collecting unit 34 is realized by the processing of the microphone 308 and the sound processing unit 309 and the CPU 311 shown in FIG. 12 and collects the sound around the general photographing apparatus 3.

  The image / sound processing unit 35 is realized mainly by a command from the CPU 311, and performs various processes on the captured image data obtained by the imaging unit 33 or the sound data obtained by the sound collection unit 34.

  The display control unit 36 is realized by the processing of the CPU 311 illustrated in FIG. 12, and causes the display 319 to display the planar image P related to the captured image data during or after capturing.

  The determination unit 37 is realized by the processing of the CPU 311 and performs various determinations. For example, the determination unit 37 determines whether the user has pressed the shutter button 315a.

  The short-range communication unit 38 is mainly realized by the CPU 311, the communication unit 317, and the antenna 317 a, and can communicate with the short-range communication unit 58 of the smartphone 5 by a short-range wireless communication technique such as Wi-Fi. .

  The storage / reading unit 39 is realized mainly by the processing of the CPU 311 shown in FIG. 12, and stores various data (or information) in the storage unit 3000 or stores various data (or information) from the storage unit 3000. Read out.

<Functional configuration of smartphone>
Next, the functional configuration of the smartphone 5 will be described in detail with reference to FIGS. 13 to 16. As illustrated in FIG. 14, the smartphone 5 includes a long-distance communication unit 51, a reception unit 52, an imaging unit 53, a sound collection unit 54, an image / sound processing unit 55, a display control unit 56, a determination unit 57, A distance communication unit 58 and a storage / reading unit 59 are provided. Each of these units is a function or means realized by any of the constituent elements shown in FIG. 13 being operated by a command from the CPU 501 according to the smartphone 5 program expanded from the EEPROM 504 to the RAM 503. is there.

  The smartphone 5 includes a storage unit 5000 configured by the ROM 502, the RAM 503, and the EEPROM 504 illustrated in FIG. In this storage unit 5000, a cooperative photographing apparatus management DB 5001 is constructed. This cooperative photographing apparatus management DB 5001 is configured by a cooperative photographing apparatus management table in FIG. FIG. 15A is a conceptual diagram of the cooperative photographing apparatus management table.

(Cooperative shooting device management table)
Next, the cooperative photographing apparatus management table will be described with reference to FIG. As shown in FIG. 15A, for each image capturing device, related relationship information indicating the cooperation relationship of each image capturing device, the IP address of the image capturing device, and the device name of the image capturing device are managed in association with each other. Of these, the related relationship information indicates that one imaging device that starts imaging when the shutter of its own device is pressed is “main”, and shooting is performed in response to the shutter being pressed by the “main” imaging device. Another imaging device to start is indicated as “sub”. Note that the IP address is for Wi-Fi communication, and in the case of wireless communication using Bluetooth instead of the manufacturer ID (Identification) and product ID in the case of communication using a USB wired cable. Replaces BD (Bluetooth Device Address).

(Each smartphone functional configuration)
The long-distance communication unit 51 of the smartphone 5 is mainly realized by the processing of the long-distance communication circuit 511 and the CPU 501 illustrated in FIG. 13, and is connected to another device (for example, other network) via a communication network such as the Internet. Various data (or information) is transmitted to and received from a smartphone or server.

  The accepting unit 52 is realized mainly by the processes of the touch panel 521 and the CPU 501 and accepts various selections or inputs from the user. The touch panel 521 may be shared with the display 517. Further, input means (buttons) other than the touch panel may be used.

  The imaging unit 53 is realized mainly by the processing of the CMOS sensors 505 and 512 and the CPU 501 shown in FIG. 13, and images a subject, a landscape, and the like to obtain captured image data. This photographed image data is planar image data photographed by the perspective projection method.

  The sound collection unit 54 is realized by the processing of the microphone 514 and the CPU 501 illustrated in FIG. 13 and collects sounds around the smartphone 5.

  The image / sound processing unit 55 is realized mainly by a command from the CPU 501, and performs various processes on the captured image data obtained by the imaging unit 53 or the sound data obtained by the sound collection unit 54.

  The display control unit 56 is realized by the processing of the CPU 501 illustrated in FIG. 13, and causes the display 517 to display the planar image P related to the captured image data during or after the imaging by the imaging unit 53. In addition, the display control unit 56 uses the superimposed display metadata created by the image / sound processing unit 55 to set each lattice area LA0 of the planar image P, which will be described later, to the position indicated by the position parameter and the correction parameter. By matching with the brightness value and the color value indicated by (2), the planar image P is superimposed and displayed on the omnidirectional image CE. The position parameter is an example of “position information”. The correction parameter is an example of “correction information”.

  The determination unit 57 is realized by the processing of the CPU 501 shown in FIG. 13 and makes various determinations.

  The near field communication unit 58 is mainly realized by the processing of the CPU 501, the near field communication circuit 519 and the antenna 519 a, and the near field communication unit 18 of the special imaging device 1, the near field communication unit 38 of the general imaging device 3, and the like. It is possible to communicate by a short-range wireless communication technology such as Wi-Fi.

  The storage / reading unit 59 is mainly realized by the processing of the CPU 501 shown in FIG. 13, and stores various data (or information) such as superimposed display metadata in the storage unit 5000 or from the storage unit 5000. Various data (or information) such as superimposed display metadata is read out. The storage / reading unit 59 serves as an acquisition unit that acquires various data from the storage unit 5000.

(Detailed functional configuration of image / sound processor)
Here, each functional configuration of the image / sound processor 55 will be described in detail with reference to FIG. FIG. 16 is a detailed functional block diagram of the image / sound processor.

  The image / sound processing unit 55 includes a metadata creating unit 55a that performs encoding and a superimposing unit 55b that performs decoding. The metadata creation unit 55a executes a process of step S22 described later shown in FIG. In addition, the superimposing unit 55b performs a process of step S23 described later illustrated in FIG.

{Each functional configuration of the metadata creation unit}
First, each functional configuration of the metadata creation unit 55a will be described. The metadata creation unit 55a includes an extraction unit 550, a first corresponding region calculation unit 552, a gaze point specification unit 554, a projection method conversion unit 556, a second corresponding region calculation unit 558, a corresponding region correction unit 559, and a region division unit. 560, a projection method inverse conversion unit 562, a shape conversion unit 564, a correction parameter creation unit 566, and a superimposed display metadata creation unit 570. Note that the shape conversion unit 564 and the correction parameter creation unit 566 are not necessary when it is not necessary to correct brightness and color. Also, reference numerals indicating images and regions described below are shown in FIG. FIG. 21 is a conceptual diagram of an image in the process of creating a superimposed display parameter.

  The extraction unit 550 extracts feature points based on local features of each image. A local feature is a pattern or structure that can be seen in an image, such as an edge or a blob, and is a feature value obtained by quantifying a local feature. In the present embodiment, the extraction unit 550 extracts each feature point from different images. The two images used by the extraction unit 550 may have different projection methods as in this embodiment as long as the distortion is not significantly large. For example, the extraction unit 550 includes a rectangular equirectangular projection image EC obtained by the equirectangular projection method, a rectangular planar image P obtained by the perspective projection method, and the planar image P. It is used between the peripheral region image PI after being converted by the method conversion unit 556. The equirectangular projection method is an example of the first projection method, and the perspective projection method is an example of the second projection method. The equirectangular projection image is an example of a first projection image, and the planar image P is an example of a second projection image.

  The first corresponding region calculation unit 552 first obtains each feature amount fv1 based on the plurality of feature points fp1 in the equirectangular projection image EC, and each feature amount based on the plurality of feature points fp2 in the planar image P. Find fv2. Several methods have been proposed as a description method of the feature amount. In the present embodiment, it is desirable that the feature amount is invariable or robust to scale and rotation. The first corresponding area calculation unit 552 is based on the similarity between the feature quantity fv1 of the plurality of feature points fp1 of the equirectangular projection image EC and the feature quantity fv2 of the plurality of feature points fp2 in the planar image P. By calculating the corresponding points between the images, and calculating the homography corresponding to the planar image P in the equirectangular projection image EC from the calculated relationship between the corresponding points, and using this homography for conversion, A first homography transformation is performed. Here, the corresponding points are a plurality of feature points selected based on the similarity from each image. As a result, the first corresponding area calculation unit 552 calculates the first corresponding area CA1. In this case, a quadrangle (rectangular) center point CP1 composed of the four vertices of the planar image P is converted into a gazing point GP1 in the equirectangular projection image EC by the first homography conversion.

  If the vertex coordinates of the four vertices of the planar image P are p1 = (x1, y1), p2 = (x2, y2), p3 = (x3, y3), and p4 = (x4, y4), the first The corresponding region calculation unit 552 can determine the center point CP1 (x, y) based on (Expression 2) shown below.

  In FIG. 21, the image shape of the planar image P is a rectangle, but the center coordinates can be calculated even for partial images of various rectangles such as a square, a trapezoid, and a rhombus by using the intersection of diagonal lines. When the image shape of the planar image P is limited to a rectangle or a square, the midpoint of the diagonal line may be used as the center coordinate PC of the partial image in order to omit the calculation. (Equation 3) in the case of calculating the midpoint of the diagonal line P1P3 is shown below.

  The gazing point specifying unit 554 specifies a point (referred to as “gazing point” in the present embodiment) on the equirectangular projection image EC where the center point CP1 of the planar image P is located after the first homography transformation.

  By the way, since the coordinates of the gazing point GP1 are coordinates on the equirectangular projection image EC, it is convenient to convert them into expressions of latitude and longitude. Specifically, the vertical direction of the equirectangular projection image EC is expressed as latitude coordinates from −90 degrees (−0.5π) to +90 degrees (+ 0.5π), and the horizontal direction from −180 degrees (−π). Expressed as longitude coordinates of +180 degrees (+ π). In this way, pixel position coordinates corresponding to the image size of the equirectangular projection image EC can be calculated from the latitude / longitude coordinates.

  The projection method conversion unit 556 generates the peripheral region image PI by converting the peripheral region PA around the gazing point GP1 in the equirectangular projection image EC into the same perspective projection method as that of the planar image P. In this case, it is possible to specify when the point of view after conversion of the gazing point GP1 is the center point CP2 and the same angle of view as the diagonal angle of view α of the planar image P is the vertical angle of view (or horizontal angle of view). The peripheral area PA of the projective transformation source is specified so that a possible square peripheral area image PI can be created as a result. This will be described in more detail below.

(Projection method conversion)
First, the conversion of the projection method will be described. As described with reference to FIGS. 3 to 5, the omnidirectional image CE is created by covering the solid sphere CS with the equirectangular projection image EC. Therefore, each pixel data of the equirectangular projection image EC can correspond to each pixel data on the surface of the solid sphere CS of the three-dimensional omnidirectional image. Therefore, the conversion formula by the projection method conversion unit 556 expresses the coordinates in the equirectangular projection image EC as (latitude, longitude) = (ea, aa), and the coordinates on the three-dimensional solid sphere CS are orthogonal coordinates (x , Y, z), it can be expressed by the following (formula 4).
(x, y, z) = (cos (ea) × cos (aa), cos (ea) × sin (aa), sin (ea)) (Equation 4)

However, the radius of the solid sphere CS at this time is 1.

On the other hand, the planar image P, which is a transmission projection image, is a two-dimensional image. When this is expressed by two-dimensional polar coordinates (radial radius, declination) = (r, a), the radial radius r is a diagonal field angle. Corresponding to α, the possible range is 0 ≦ r ≦ tan (diagonal angle of view / 2). Further, when the planar image P is represented by a two-dimensional orthogonal coordinate system (u, v), the conversion relationship with polar coordinates (radial radius, declination) = (r, a) is represented by the following (formula 5). Can do.
u = r × cos (a), v = r × sin (a) (Equation 5)
Next, let us consider that this (Formula 5) corresponds to three-dimensional coordinates (radial radius, polar angle, azimuth angle). Since only the surface of the solid sphere CS is considered now, the moving radius in the three-dimensional polar coordinate is “1”. Further, the projection for perspective projection transformation of the equirectangular projection image EC attached to the surface of the solid sphere CS is considered to be the above-described two-dimensional polar coordinate (radial radius, declination) when the virtual camera IC is located at the center of the solid sphere CS. ) = (R, a) can be expressed by the following (Expression 6) and (Expression 7).
r = tan (polar angle) (Formula 6)
a = Azimuth angle (Formula 7)
If the polar angle is t, then t = arctan (r), so the three-dimensional polar coordinates (radial radius, polar angle, azimuth) are (radial radius, polar angle, azimuth) = (1, arctan (r ), a).

A conversion formula for converting from the three-dimensional polar coordinate to the orthogonal coordinate system (x, y, z) can be expressed by the following (Formula 8).
(x, y, z) = (sin (t) x cos (a), sin (t) x sin (a), cos (t)) (Equation 8)
According to the above (Equation 8), the equirectangular projection image EC by the equirectangular projection method and the planar image P by the perspective projection method can be mutually converted. That is, by using the radius r corresponding to the diagonal angle of view α of the planar image P to be created, the conversion map indicating which coordinate of the equirectangular projection image EC corresponds to each pixel of the planar image P. The coordinates can be calculated, and the peripheral area image PI, which is a perspective projection image, can be created from the equirectangular projection image EC based on the conversion map coordinates.

  By the way, in the projection method conversion, the position where the (latitude, longitude) of the equirectangular projection image EC is (90 °, 0 °) becomes the center point CP2 of the peripheral region image PI which is a perspective projection image. Shows the conversion. Therefore, when performing perspective projection conversion using an arbitrary point of the equirectangular projection image EC as a gazing point, the coordinates of the gazing point (latitude, latitude, Coordinate rotation may be performed such that (longitude) is arranged at a position of (90 °, 0 °).

  Since the conversion formula relating to the rotation of the solid sphere CS is a general coordinate rotation formula, description thereof is omitted.

(Identification of surrounding area image)
Next, a method for specifying the region of the peripheral region image PI will be described with reference to FIG. FIG. 22 is a conceptual diagram when specifying a peripheral area image.

  When the corresponding area calculation unit 552 determines the correspondence between the planar image P and the peripheral area image PI, it is desirable that the second corresponding area CA02 is included as widely as possible in the peripheral area image PI. Therefore, if the peripheral area image PI is set to a wide area, a situation in which the second corresponding area CA02 is not included does not occur. However, if the peripheral area image PI is set to an excessively large area, the number of pixels for which the similarity is calculated increases, and thus processing takes time. Therefore, it is preferable that the area of the peripheral area image PI is as small as possible within the range including the second corresponding area CA02. Therefore, in the present embodiment, the peripheral area image PI is specified by the following method.

In the present embodiment, the 35 mm equivalent focal length of the planar image is used for specifying the peripheral area image PI, which is obtained from Exif data recorded at the time of shooting. Since the 35 mm equivalent focal length is a focal length based on a so-called 24 mm × 36 mm film size, the following calculation formulas (Equation 9) and (Equation 10) correspond to the diagonal of the film and the focal length. The angle of view can be calculated.
Film diagonal = sqrt (24 * 24 + 36 * 36) (Equation 9)
Composite image angle of view / 2 = arctan ((film diagonal / 2) / 35mm equivalent focal length of composite image) ... (Equation 10)
By the way, the image covering this angle of view is circular, but since the actual image sensor (film) is rectangular, it is a rectangular image inscribed in the circle. In the present embodiment, the vertical field angle α of the peripheral area image PI is set to be the same as the diagonal field angle α of the planar image P. As a result, the peripheral area image PI shown in FIG. 22B becomes a square circumscribing a circle covering the diagonal angle of view α of the planar image P shown in FIG. The angle α can be calculated from the diagonal of the square and the focal length of the planar image P as shown in the following (Expression 11) and (Expression 12).
Square diagonal = sqrt (film diagonal * film diagonal + film diagonal * film diagonal) ... (Formula 11)
Vertical angle of view α / 2 = arctan ((square square / 2) / 35mm equivalent focal length of planar image)) ... (Formula 12)
By performing projective transformation at such a vertical angle of view α, the peripheral region image PI that can cover the image at the diagonal angle of view α of the planar image P as wide as possible with the gazing point as the center, and the vertical angle of view α is not too large. (Perspective projection image) can be created.

(Calculation of location information)
Subsequently, returning to FIG. 16, the second corresponding region calculation unit 558 calculates the feature amounts fv2 of the plurality of feature points fp2 in the planar image P and the feature amounts fv3 of the plurality of feature points fp3 in the peripheral region image PI. Corresponding points between images are calculated based on the calculated similarities of the feature quantities fv2 and fv3, and a homography corresponding to the planar image P is calculated in the peripheral region image PI from the calculated relationship between the corresponding points between the images. The second homography conversion is performed by using this homography for the conversion. As a result, the second corresponding area calculation unit 558 calculates the second corresponding area CA02.

  Note that, before the first homography conversion, the image size of at least one of the planar image P and the equirectangular projection image EC may be resized in order to advance the calculation time of the first homography. For example, when the planar image P is 40 million pixels and the equirectangular projection image EC is 30 million pixels, the planar image P is resized to 30 million pixels, or both images are adjusted to 10 million pixels. Or resize it. Similarly, the image size of at least one of the planar image P and the peripheral region image PI may be resized before the second homography calculation.

  In addition, the homography in the present embodiment is a transformation matrix that represents the projection relationship between the equirectangular projection image EC and the plane image P. The homography transformation matrix calculated by the homography calculation process is applied to the coordinates in the plane image P. Can be used to calculate corresponding coordinates on the equirectangular projection image EC (the omnidirectional image CE).

  The corresponding area correction unit 559 corrects the second corresponding area CA02 to create a second corresponding area CA2 that is close to the area corresponding to the planar image P in the equirectangular projection image EC. Here, the corresponding area correction unit 559 will be described in detail with reference to FIGS. 17 and 23 to 31. FIG. 17 is a detailed diagram of the corresponding area correction unit.

  The corresponding area correction unit 559 includes a block division unit 21, a matching unit 22, a motion vector calculation unit 23, a motion vector correction unit 24, and a representative point correction unit 25. The corresponding area correction unit 559 receives the data of the second corresponding area CA02 calculated by the second corresponding area calculation unit 558, and corrects the second corresponding area CA02. The corrected corresponding area calculation result is output to the area dividing unit 560.

  FIG. 23 is a conceptual diagram showing processing of the corresponding area correction unit. The block dividing unit 21 divides the planar image P into a plurality of blocks. Here, although it is divided into nine, it may be divided into any number. Here, the number of blocks may be two or more, although it depends on the size of the image and the characteristics of the image. For example, if the number is 4 to 16, it may be possible to search appropriately and block matching may be easy.

  The matching 22 calculates a corresponding area with the peripheral area image PI using each of the divided blocks as a template. That is, a corresponding area with the peripheral area image PI is found for each block. At this time, the search area of each block may be limited to a part of the peripheral area image PI, not the entire peripheral area image PI. For example, when block matching is performed on the result detected by the second corresponding area calculation unit 558, the plane image PI is divided into the same number of blocks as the number of blocks of the plane image P, and each pixel is limited to the surrounding pixels of the corresponding block. Block matching may be performed. Here, it is desirable that the number of blocks in the peripheral image region PI is the same as the number of blocks in the planar image P or set as follows. As a result, it is possible to prevent the block matching result from deviating greatly, and at the same time to reduce the calculation time. In addition, by specifying the range of peripheral pixels, it is possible to balance matching accuracy, calculation time, and the like. As shown in FIG. 23, the position of the second corresponding area CA02 in the peripheral area image PI needs to be corrected because the actual position of each block calculated by the matching unit 22 is different.

  Further, the motion vector calculation unit 23 calculates a motion vector for correcting the position of each block. FIG. 24 is a diagram for explaining the concept of motion vectors. As shown in FIG. 24, first, a plurality of representative points RP01 to RP04 for obtaining a motion vector are set in advance. The representative points in this embodiment will be described later.

  The motion vector can be obtained by comparing the initial position of each representative point serving as a reference with the position of the representative point after movement by block matching, and calculating the vector. In the present embodiment, the position of the representative point after movement after block matching is compared with the positions (initial positions) of the representative points RP01 to RP04 in the second corresponding area CA02. FIG. 24 shows the case where the motion vectors of the respective blocks are different from each other. Further, each block may be the same motion vector. Furthermore, although four points are selected as representative points of the initial position, four or more points may be selected as will be described later.

  The respective motion vectors MV01 to MV04 are shown from the representative points RP01 to PR04 of the second corresponding area CA02 to the four corner points of each block shown in FIG. However, each block is not necessarily matched accurately. In FIG. 24, when there is no boundary with an object such as the sky, there is a high possibility that matching is not accurately performed. Therefore, the representative points RP01 cannot be moved according to the motion vectors MV01 to MV04.

  Therefore, the motion vector correction unit 24 corrects the motion vector. FIG. 25 illustrates correction processing based on similarity and luminance variance values, where (a) illustrates the effectiveness X based on similarity, and (b) illustrates the effectiveness Y based on luminance variance.

  This similarity indicates, for example, an index such as SSD or ZNCC used as an index when each block in the planar image P is template-matched with the peripheral image PI, and is represented by a real number that takes a value from 0 to 1. Is done. For example, SSD uses the sum of squares of the difference between pixel values between two images as an index, and indicates that the smaller the value, the more similar. ZNCC is an index that uses a normalized cross-correlation obtained after subtracting the average of pixel values from two images. An index other than SSD or ZNCC may be used. In addition, when using the degree of difference taking a real value from 0 to 1 as a matching index, a value obtained by subtracting the degree of difference from 1 may be used as the degree of similarity.

  Further, the luminance dispersion value is a value V expressed by the following Expression 20.

n is the number of pixels in the block, the first x is the luminance value of each pixel, and the next x is the average value of the luminance values of each pixel.

  Here, the effectiveness X based on the similarity will be described. A block with low similarity is likely to have unreliable matching results. Therefore, if a matching result of a block with low similarity is applied, the position may be corrected based on an incorrect matching result. Therefore, for the block having a low similarity, the effectiveness X is set to 0 so that the result is not used for the correction process. For blocks with a medium similarity, correction is performed while minimizing adverse effects. When the similarity is sufficiently large, the effectiveness X is set to 1.

  Next, the effectiveness Y based on the luminance dispersion value will be described. A block with a small luminance variance value is often an image with few feature points, and even if the degree of similarity is high, there is a high possibility that the matching result is not reliable. Therefore, if a matching result of a block having a low luminance variance value is applied, the position may be corrected based on an incorrect matching result as in the case where the similarity is low. Therefore, for a block having a small luminance variance, the effectiveness Y is set to 0 so that the result is not used in the correction process. For a block with a moderate luminance dispersion, correction is performed while minimizing adverse effects. When the luminance variance is sufficiently large, the effectiveness Y is set to 1. It should be noted that the reference values LX (for example, 0.4) and HX (for example, 0.7) for similarity and the reference values LY and HY for luminance dispersion may be freely specified. The final correction effectiveness is calculated by the following (formula 21). Here, the range of the parameter setting of “low, medium, high” of the similarity between the effectiveness X and the effectiveness Y is, for example, low is 0 or more and less than 0.3, medium is 0.3 or more and less than 0.7, The height can be 0.7 or more and 1.0. The degree of similarity may be two or three or more, and the parameter range can be arbitrarily set.

  In the motion vector correction unit 24, the motion vector value is corrected by multiplying the motion vector by the calculated correction effectiveness, and the corrected motion vector is used as a final motion vector. By increasing the correction effectiveness only when the similarity is high, the corresponding region is corrected to be large when the matching result is correct, and the correction is minimized when the matching result is incorrect. Thereby, the moving vector is corrected.

  The representative point correction unit 25 corrects the representative point according to the changed motion vector. FIG. 26 is a diagram for explaining the concept of processing for correcting a representative point using a corrected motion vector. For the upper left and upper left blocks, since the similarity is large but the variance value of luminance is small, it is determined that the image has few feature points, and the reliability of the matching result is low. That is, when the effectiveness Y = 0, the correction effectiveness = 0, and no correction is performed. Therefore, the motion vector has a value close to 0, and the position in the second corresponding area calculation unit 559 is used almost as it is.

  On the other hand, for the representative point of the lower right block, since the similarity is large and the variance value of the luminance is large, it is determined that the matching result in the divided block is reliable. That is, the effectiveness X = 1, the effectiveness Y = 1, and the correction effectiveness = 1. Therefore, correction processing is performed based on the detection position of the divided block. The representative point of the lower left block has a sufficiently high similarity and a medium luminance dispersion value, so that the correction effectiveness = Y. Therefore, the correction by the divided block is performed for the size of Y.

  FIG. 26 shows an example of switching the validity or invalidity of a block when using only the four corner blocks. However, the present invention may be applied when all the blocks are used.

  The representative points RP1 to RP4 are finally obtained by the corresponding region correction unit 559 as described above.

  Next, another processing method by the corresponding area correction unit 559 will be described with reference to FIGS. FIG. 27 is a conceptual diagram showing another process of the corresponding area correction unit. 23, four representative points RP01 to RP04 related to four blocks in the peripheral area image PI have been described, but in FIG. 27, 16 representative points related to all blocks (here, nine) divided into blocks. RP011 to RP044 will be described.

  FIG. 28 is a diagram showing all representative points when the second corresponding area CA02 is divided by the same number as the planar image P. FIG. FIG. 29 and FIG. 30 are diagrams for explaining the concept related to motion vector correction in the case shown in FIG. FIG. 30 is a diagram for explaining a concept related to motion vector correction, where (a) is a correction position at a non-shared point, (b) is a correction position of a shared point in two blocks, and (c) is in four blocks. It is a conceptual diagram which shows the correction position of a shared point, respectively.

  As shown in FIGS. 29 and 30A, in the case of the representative points at the four corners of the four corner blocks divided by the second corresponding area CA02, the points indicated by the motion vector MV011 from the representative point RP011 A corrected motion vector MV11 is obtained for BP11. In this case, the motion vector MV011 and the corrected motion vector MV11 are the same.

  Further, as shown in FIGS. 29 and 30B, in the case of the representative points on the four sides of the second corresponding area CA02 (excluding the representative points at the four corners), for example, the movement from the representative point RP021 A corrected vector MV21 is obtained for the barycentric point G1 (BP21) of the points BP12 and BP21 indicated by the vectors MV012 and MV021.

  Further, as shown in FIGS. 29 and 30B, in the case of representative points other than the representative points on the four sides of the second corresponding area CA02, for example, motion vectors MV014, MV023 from the representative point RP02. A corrected vector MV22 is obtained for the barycentric point G2 (BP22) of the points BP14, BP23, and BP51 indicated by MV051.

  As described above, the corresponding area correction unit 559 corrects the second corresponding area CA02 to create a new second corresponding area CA2, thereby finally performing a superimposition process described later (see step S23). Thus, the display as shown in FIG. 31 is performed in the display process (step S24) described later. FIG. 31 is a diagram illustrating superposition and synthesis positions before and after correction when block matching and correction validity / invalidity determination are used.

  FIG. 31A is a diagram illustrating the superimposition synthesis position L1 before correction by block matching and the post-correction superposition synthesis position L2. The left side of FIG. 31A is a blue sky, and there are very few feature points. For this reason, the matching result without correction is affected by the blue sky region, and an erroneous region is detected as the superimposed composition position L1. In this case, when the synthesis position L1 of the peripheral area image PI is converted into the same shape as the planar image P using the motion vector, the target object (in this case, the electric light) is in the middle of the screen as shown in FIG. The image is greatly distorted as compared with the planar image P shown in FIG. 31 (d) without being positioned. Here, FIG. 31D is a planar image P itself captured by the general imaging device 3.

  On the other hand, when block matching and correction are performed, if the peripheral area image PI is converted into the same shape as the planar image P using a motion vector, the target object is located in the middle of the screen as shown in FIG. . This is almost uncomfortable as compared with FIG. 31 (d), and the influence of the region with few feature points can be corrected by block matching. As described above, correction by block matching and validity determination may not only improve the matching result between images with large parallax but also improve the matching result between images with few feature points.

  Subsequently, returning to FIG. 16, the region dividing unit 560 divides a partial region in the image into a plurality of lattice regions. Here, a method of dividing the second corresponding region into a plurality of lattice regions will be described in detail with reference to FIG. FIG. 32 is a conceptual diagram when the second corresponding region is divided into a plurality of lattice regions.

  As shown in FIG. 32A, the region dividing unit 560 is a quadrangle formed of four vertices of the vertex coordinates of the second corresponding region CA2 calculated by the corresponding region calculating unit 552 by the second homography transformation. Is divided into a plurality of lattice areas LA2, as shown in FIG. For example, the image is equally divided by 30 in the horizontal direction and 20 in the vertical direction.

  Next, a specific method for dividing the plurality of lattice areas LA2 will be described.

First, a calculation formula for equally dividing the second corresponding area CA2 is shown. When the two points are A (X1, Y1) and B (X2, Y2) and the line segment between the two points is divided into n pieces at equal intervals, the coordinates of the point Pm corresponding to the mth point from A are as follows: Calculated by (Equation 13).
Pm = (X1 + (X2−X1) × m / n, Y1 + (Y2−Y1) × m / n) (Equation 13)
Since the coordinates obtained by equally dividing the line segment can be calculated by the above (Formula 13), the coordinates obtained by dividing the upper side and the lower side of the quadrangle may be obtained, and the line segment composed of the divided coordinates may be further divided. When the coordinates of the upper left, upper right, lower right, and lower left corners of the quadrangle are TL, TR, BR, and BL, respectively, coordinates obtained by equally dividing the line segment TL-TR and the line segment BR-BL into 30 parts are obtained. Next, in each of the coordinates divided from the 0th to the 30th, coordinates obtained by equally dividing 20 into the line segments composed of the coordinates corresponding to the same order are obtained. Thereby, the coordinates for dividing the quadrangular area into 30 × 20 small areas can be obtained. FIG. 32B shows TL coordinates (LO _00,00 , LA _00,00 ) as an example of coordinates.

  Subsequently, returning to FIG. 16 and FIG. 21, the projection method inverse conversion unit 562 reversely converts the projection method of the second corresponding area CA2 into the same equirectangular projection method as the equirectangular projection image EC, In the equirectangular projection image EC, a third corresponding area CA3 corresponding to the second corresponding area CA2 is calculated. Specifically, the projection method inverse transform unit 562 calculates a third corresponding area CA3 including the respective lattice areas LA3 corresponding to the plurality of lattice areas LA2 in the second corresponding area CA2 in the equirectangular projection image EC. To do. The third corresponding area CA is shown in FIG. 21, but is also shown in FIG. 33 as an enlarged view. FIG. 33 is a conceptual diagram showing a third corresponding region in the equirectangular projection image EC. Thereby, the planar image P is superimposed and displayed on the omnidirectional image CE created by the equirectangular projection image EC so as to be finally matched (mapped) with the third corresponding area CA3. By the process of the projection method inverse transform unit 562, a position parameter indicating the coordinates of each grid point in each grid area LA3 is created. The positional parameters are shown in FIGS. 18 and 19 (b). Note that the grid point is an example of a plurality of points.

  As described above, the positional relationship between the equirectangular projection image EC and the planar image P can be calculated by creating the positional parameters.

  By the way, although the position parameter has been calculated, when superimposing display is performed as it is, when the equirectangular projection image EC and the flat image P are greatly different in brightness and color, an unnatural superimposition display may occur. Therefore, the shape conversion unit 564 and the correction parameter creation unit 566 described below play a role of preventing unnatural superimposition display even when the brightness and color tone are greatly different.

  Prior to color matching described later, the shape conversion unit 564 projects the four vertices of the second corresponding area CA2 onto the four vertices of the planar image P, so that the second corresponding area CA2 has the same shape as the planar image P. Convert. Specifically, in order to match each lattice area LA2 of the second corresponding area CA2 shown in FIG. 34 (a) with each lattice area LA0 of the planar image P shown in FIG. 34 (c). The shape of the second corresponding area CA2 is converted into the same shape as the planar image P. As a result, the shape of the second corresponding area CA2 shown in FIG. 34 (a) is converted into the second corresponding area CA2 'shown in FIG. 34 (b). Accordingly, since the lattice area LA2 is converted into the lattice area LA2 ', the shape is the same as the lattice area LA0 of the planar image P.

The correction parameter creation unit 566 has a planar image P having the same shape as each grid area LA2 ′ with respect to the brightness value and color value of each grid area LA2 ′ in the second corresponding area CA2 ′ converted into the same shape. A correction parameter for matching the brightness value and the color value of each lattice area LA0 is created. Specifically, the correction parameter creation unit 566 has the brightness values and color values (R, G, B) of all the pixels constituting the four grid areas having one grid point LA0 as a grid point in common. The average value a = (R _ave , G _ave , B _ave ) is calculated, and the brightness values and colors of all the pixels constituting the four grid areas having one grid point LA2 ′ as a grid point in common. The average value a ′ = (R ′ _ave , G ′ _ave , B ′ _ave ) of the values (R ′, G ′, B ′) is calculated. When one point of each of the lattice points LA0 and one point of each of the lattice points LA2 ′ is the four corners of the second corresponding region CA2 and the third corresponding region CA3, the correction parameter creating unit 566 has one lattice region. Then, the average value a and the average value a ′ of the brightness value and the color value are calculated. When one point of each of the lattice points LA0 and one point of each of the lattice points LA2 ′ is a point on the outer periphery of the second corresponding area CA2 and the third corresponding area CA3, the correction parameter creating unit 566 includes two inner lattice points. An average value a and an average value a ′ of the brightness value and the color value are calculated from the area. In this embodiment, the correction parameter is gain data for correcting the brightness value and the color value of the planar image P. Therefore, as shown in the following (Expression 14), the average value a ′ Is divided by the average value a to obtain the correction parameter Pa.
Pa = a '/ a (Formula 14)
As a result, in the superimposed display described later, the gain value indicated by the correction parameter is multiplied for each lattice area LA0, so that the color and brightness of the planar image P become the equirectangular projection image EC (the omnidirectional image CE). ) And the pixel value are close to each other, and the display can be superimposed and displayed without a sense of incongruity. The correction parameter is not only calculated from the average value, but may be calculated using a median value, a mode value, or the like instead of or in addition to the average value.

  In this embodiment, the pixel values (R, G, B) are used for calculating the correction values for the brightness value and the color value. However, the YUV format for luminance and color difference signals, the JPEG sYCC (YCbCr) format, etc. Using the luminance value and color difference value in, the average value of the luminance value and color difference value of all the pixels constituting the grid area is obtained in the same way, and the average value is divided to correct the correction parameter in the superimposed display described later It is good. The method of converting RGB values to YUV and sYCC (YCbCr) is well known and will not be described in detail. However, using (Equation 15) as a reference, the JPEG compressed image format JFIF (JPEG file interchange format) format is used. Here is an example of converting from RGB to YCbCr.

  The superimposed display metadata creation unit 570 uses the position parameter, the correction parameter, and the like to superimpose the position when the planar image P is superimposed on the omnidirectional image CE, and the correction value of the brightness value and the color value. Create display metadata.

(Superimposed display metadata)
Next, the data structure of the superimposed display metadata will be described with reference to FIG. FIG. 18 shows the data structure of the superimposed display metadata.

  As shown in FIG. 18, the superimposed display metadata includes equirectangular projection image information, planar image information, superimposed display information, and metadata creation information.

  Among these, the equirectangular projection image information is information sent together with the photographed image data from the special photographing apparatus 1. The equirectangular projection image information includes image identification information and attached information. The image identification information in the equirectangular projection image information is image identification information for identifying the equirectangular projection image. In FIG. 18, the file name of the image is shown as an example of the image identification information in the equirectangular projection image information, but an image ID for identifying the image may be used.

  The attached information in the equirectangular projection image information is related information attached to the equirectangular projection image information. In FIG. 18, posture correction information (Pitch, Yaw, Roll) of equirectangular projection image data obtained when captured by the special imaging device 1 is shown as an example of attached information. This posture correction information may be stored in the Exif (Exchangeable image file format) standardized as the image recording format of the special imaging device 1, and various formats provided by GPano (Google Photo Sphere schema) It may be stored in. An omnidirectional image has a feature that if it is taken at the same position, an image of 360 ° omnidirectional image can be taken even if the posture is different. The display position is not determined unless the center of the image is specified (gaze point). Therefore, generally, the omnidirectional image CE is corrected and displayed so that the zenith is directly above the photographer. As a result, a natural display in which the horizon is straightly corrected is possible.

  Next, the planar image information is information sent from the general imaging device 3 together with the captured image data. The planar image information includes image identification information and attached information. The image identification information in the planar image information is image identification information for identifying the planar image P. In FIG. 18, the file name of the image is shown as an example of the image identification information, but an image ID for identifying the image may be used.

  The attached information in the planar image information is related information attached to the planar image information. In FIG. 18, the value of 35 mm equivalent focal length is shown as an example of the attached information in the planar image information. The value of the 35 mm equivalent focal length is not essential for displaying the celestial sphere image CE by superimposing the planar image P, but serves as reference information for determining the angle of view to be displayed when superimposing display is performed. Take as an example.

  Next, the superimposed display information is information created by the smartphone 5, and includes information on the number of area divisions, positions of lattice points in each lattice area (position parameters), and correction values (correction parameters) of brightness values and color values. Is included. Among these, the area division number information indicates the number of divisions in the horizontal (longitude) direction and the number of divisions in the vertical (latitude) direction when dividing the first corresponding area CA1 into a plurality of lattice areas.

  The position parameter is a vertex indicating at which position in the equirectangular projection image EC (global celestial image CE) each lattice point when the planar image P is divided into a plurality of lattice-like regions is arranged. Mapping information. In this embodiment, the correction parameter is gain data for correcting the color value of the planar image P. Since the correction target may be a monochrome image, the correction parameter is a parameter for adjusting at least the brightness value among the brightness value and the color value.

  By the way, when the omnidirectional image CE is obtained, if the perspective projection method that is the same projection method as the planar image P is used, 360 ° omnidirectional imaging cannot be performed. Therefore, even if an attempt is made to superimpose a planar image P obtained by photographing separately from the omnidirectional image CE on a partial area of the omnidirectional image CE, the equirectangular projection image EC and the planar image P are projected. Since the methods are different, the equirectangular projection image EC (global celestial sphere image CE) and the planar image P do not match, and the planar image P does not merge well with the omnidirectional image CE. Therefore, an image with a wide angle of view such as an omnidirectional image is often created by an equirectangular projection method as one of the projection methods. However, if the equirectangular projection method is used, the length in the horizontal direction increases as the distance from the standard parallel as in the so-called Mercator projection, so it is greatly different from the image of the perspective projection method used in general cameras. The image will be different. Therefore, even if only the scales of the images are changed and superimposed, the images do not match, and the planar image P does not merge well with the omnidirectional image CE. Therefore, in the present embodiment, the position parameter is created by the process shown in FIG.

  Here, the position parameter and the correction parameter will be described in detail with reference to FIG. FIG. 19A is a conceptual diagram showing each lattice region in the second corresponding region. FIG. 19B is a conceptual diagram showing each lattice region in the third corresponding region.

As shown in FIG. 19A, the first corresponding area CA1 which is a partial area of the equirectangular projection image EC is converted into the same perspective projection method as that of the planar image P. In the present embodiment, the second corresponding area CA2 is divided into a plurality of lattice areas having a horizontal division number of 30 and a vertical division number of 20. FIG. 19A shows the coordinates (LO _00,00 , LA _00,00 ), (LO _01,00 , LA _01,00 ),..., (LO _30,20 , LA ₃₀ ) of each lattice area. _{, 20} ), and correction values (R _00,00 , G _00,00 , B _00,00 ), (R _01,00 , G _01,00 , B _01,00 ), ..., (R _30,20 , G _30,20 , B _30,20 ) are shown. In order to simplify the drawing, the coordinates of the grid points at the four vertices and the correction values of the brightness value and the color value are shown, but in reality, the coordinates of all the grid points, the brightness values, and There is a correction value for the color value. Further, the correction values R, G, and B of the brightness value and the color value respectively indicate a red correction gain, a green correction gain, and a blue correction gain. The correction values R, G, and B of each brightness value and color value are actually the brightness of an image within a predetermined range centered on the coordinates of each grid point (a range that does not overlap with the predetermined range of adjacent grid points). The correction values for the color value and the color value are shown.

On the other hand, as shown in FIG. 19B, the third corresponding region obtained by inversely transforming the second corresponding region CA2 into the same equirectangular projection method as the equirectangular projection image EC. In the present embodiment, CA3 is similarly divided into a plurality of lattice regions having 30 horizontal divisions and 20 vertical divisions. FIG. 19B shows the coordinates (LO ′ _00,00 , LA ′ _00,00 ), (LO ′ _01,00 , LA ′ _01,00 ),..., (LO ′ _{30 , 20} , LA ′ _30,20 ), and brightness value and color value correction values having the same values as the correction values of the second corresponding area CA2. In this case, too, the coordinates of the grid points, the brightness values, and the correction values of the color values at the four vertices are shown in order to simplify the drawing. There is a correction value for the color value.

  Next, the metadata creation information indicates version information indicating the version of the superimposed display metadata.

  As described above, the position parameter indicates the positional correspondence between the planar image P and the equirectangular projection image EC (the omnidirectional image CE). However, if the position parameter is intended to represent information on which coordinates of all the pixel positions of the planar image P are associated with the equirectangular projection image EC (the omnidirectional image CE), the general imaging device 3 In the case of a digital camera having a high number of pixels, information corresponding to about 40 million pixels must be represented. For this reason, the amount of position parameter data increases, and handling of data storage and the like is difficult. Therefore, the position parameter of the present embodiment is that the plane image P is divided into 600 (30 × 20) regions, so that the coordinates of the lattice points of the plane image are the equirectangular projection image EC (the omnidirectional image CE). ) Shows only data indicating where to correspond. In addition, when the smartphone 5 performs superimposed display, the superimposed display can be realized by interpolating the position of the image in each region from the coordinates of the grid points.

{Functional structure of superimposition unit}
Subsequently, the functional configuration of the superimposing unit 55b will be described with reference to FIG. The superimposing unit 55b includes a pasting area creating unit 582, a correcting unit 584, an image creating unit 586, an image superimposing unit 588, and a projective transformation unit 590.

  Among these, the pasting area creation unit 582 creates an area part (hereinafter referred to as “partial solid sphere”) PS corresponding to the third corresponding area CA3 in the virtual solid sphere CS.

  The correction unit 584 performs correction to match the brightness value and color value of the planar image P with the brightness value and color value of the equirectangular projection image EC based on the correction parameter in the superimposed display metadata. Note that the correction unit 584 does not necessarily have to correct brightness and color. Further, when the correction unit 584 performs correction using the correction parameter, the brightness may be corrected or the color may not be corrected.

  The image creating unit 586 creates the superimposed image S by pasting the planar image P (or the corrected image C after correcting the planar image P) to the partial solid sphere PS. The image creation unit 586 creates mask data M based on the region of the partial solid sphere PS. Furthermore, the image creation unit 586 creates the omnidirectional image CE by pasting the equirectangular projection image EC to the solid sphere CS.

  Here, the mask data M is transmission ratio data that can be used when the superimposed image S is superimposed on the omnidirectional image CE. The mask data M is used to gradually bring the brightness and color around the boundary when the superimposed image S is superimposed on the omnidirectional image CE from the inner superimposed image S side to the outer omnidirectional image CE side. The transmittance around the mask gradually increases from the superimposed image S to the omnidirectional image CE from the inside to the outside. Thereby, even if the superimposition image S is superimposed on the omnidirectional image CE, it is possible to prevent the superimposition from being recognized as much as possible. Note that the creation of the mask data M is not essential.

  The image superimposing unit 588 superimposes the superimposed image S and the mask data M on the omnidirectional image CE. Thereby, the low-definition omnidirectional image CE on which the high-definition superimposed image S is superimposed so that the boundary is not conspicuous is completed.

  As shown in FIG. 7, the projective transformation unit 590 generates a superimposed image S based on a predetermined line-of-sight direction of the virtual camera IC (a center point CP of the predetermined area T) and the angle of view α of the predetermined area T. Projective transformation is performed so that the predetermined area T in the omnidirectional image CE in a state where is superimposed can be seen on the display 517. Projection conversion unit 590 also performs processing for matching predetermined area T with the resolution of the display area on display 517 when performing projective conversion. Specifically, when the resolution of the predetermined area T is smaller than the resolution of the display area of the display 517, the projection conversion unit 590 enlarges the predetermined area T so as to match the display area of the display 517. On the other hand, when the resolution of the predetermined area T is larger than the resolution of the display area of the display 517, the projective transformation unit 590 reduces the predetermined area T so as to match the display area of the display 517. Thereby, the display control unit 56 can display the predetermined area image Q indicating the predetermined area T over the entire display area of the display 517.

<< Processing or Operation of Embodiment >>
Subsequently, processing or operation of the present embodiment will be described with reference to FIGS. 20 to 40. First, a photographing method executed by the photographing system will be described with reference to FIG. FIG. 20 is a sequence diagram illustrating a photographing method according to the present embodiment. In the following description, a case where a subject, landscape, or the like is photographed will be described. However, ambient sounds may be recorded by the sound collection unit 14 simultaneously with photographing.

  As illustrated in FIG. 20, the reception unit 52 of the smartphone 5 receives the start of cooperative shooting from the user (step S11). In this case, as illustrated in FIG. 15B, the display control unit 56 causes the display 517 to display the cooperative photographing apparatus setting screen illustrated in FIG. On this screen, a radio button for designating a main imaging device for linked shooting and a check box for designating (selecting) a sub imaging device for linked shooting are displayed for each imaging device. Yes. Further, for each photographing apparatus, the name of the photographing apparatus and the reception intensity of the radio wave are displayed. Then, when the user designates (selects) a desired photographing apparatus as a main and a sub, and presses a “confirm” button, the accepting unit accepts the start of cooperative photographing. Since there may be a plurality of sub photographing apparatuses, a plurality of photographing apparatuses can be designated (selected) by using check boxes.

  And the short distance communication part 58 of the smart phone 5 transmits the imaging | photography start inquiry information which shows the inquiry of a photography start by polling with respect to the short distance communication part 38 of the general imaging device 3 (step S12). Thereby, the near field communication part 38 of the general imaging device 3 receives imaging start inquiry information.

  Next, the determination unit 37 of the general photographing apparatus 3 determines whether or not the photographing is started based on whether or not the reception unit 32 receives the pressing of the shutter button 315a from the user (step S13).

  Next, the short-range communication unit 38 of the general imaging device 3 transmits response information indicating the response content according to the determination result in step S13 to the smartphone 5 (step S14). If it is determined in step S13 that shooting has started, the response information includes shooting start information indicating the start of shooting. In this case, the response information also includes image identification information of the general imaging device 3. On the other hand, when it is not determined in step S13 that shooting has started, the response information includes shooting standby information indicating shooting standby. Thereby, the near field communication part 58 of the smart phone 5 receives response information.

  Next, a case where it is determined in step S13 that shooting has started and a case where the response information received in step S14 includes shooting start information will be described.

  First, the general photographing apparatus 3 starts photographing (step S15). In this photographing process, after the shutter button 315a is pressed, the image capturing unit 33 captures an image of a subject, a landscape, and the like, so that captured image data (in this case, planar image data) is obtained, and the storage / reading unit 39 stores the storage unit 3000. This is the process until the captured image data is stored.

  Next, in the smartphone 5, the short-range communication unit 58 transmits shooting start request information indicating a shooting start request to the special shooting device 1 (step S16). Thereby, the short-range communication unit 18 of the special imaging device 1 receives the imaging start request information.

  On the other hand, the special photographing apparatus 1 starts photographing (step S17). In this photographing process, the imaging unit 13 photographs a subject, a landscape, and the like to generate photographed image data (two hemispherical image data as shown in FIGS. 3A and 3B). The sound processing unit 15 creates single equirectangular projection image data as shown in FIG. 3C based on the two hemispherical image data, and the storage / reading unit 19 stores the storage unit 1000. This is the process until the equirectangular projection image data is stored.

  Next, in the smartphone 5, the short-range communication unit 58 transmits captured image request information indicating that a captured image is requested to the general imaging device 3 (step S <b> 18). This photographed image request information includes the image identification information received in step S14. Thereby, the short-range communication unit 38 of the general imaging device 3 receives the captured image request information.

  Next, the short-range communication unit 38 of the general imaging device 3 transmits the planar image data obtained in step S15 to the smartphone 5 (step S19). At this time, image identification information for identifying the plane image data to be transmitted, and attached information are also transmitted. The image identification information and the attached information are shown as planar image information in FIG. Thereby, the near field communication part 58 of the smart phone 5 receives plane image data, image identification information, and attached information.

  On the other hand, the short-range communication unit 18 of the special imaging device 1 transmits the equirectangular projection image data obtained in step S18 to the smartphone 5 (step S20). At this time, image identification information for identifying the equirectangular projection image data to be transmitted, and attached information are also transmitted. The image identification information and attached information are shown as equidistant cylindrical projection image information in FIG. Thereby, the near field communication part 58 of the smart phone 5 receives equirectangular projection image data, image identification information, and attached information.

  Next, the storage / reading unit 59 of the smartphone 5 stores the electronic file of planar image data received in step S19 and the electronic file of equirectangular projection image data received in step S20 in the same electronic folder. And it memorize | stores in the memory | storage part 5000 (step S21).

  Next, the image / sound processing unit 55 of the smartphone 5 uses the superimposition that is used when the planar image P that is a high-definition image is superimposed and displayed on a partial region of the omnidirectional image CE that is a low-definition image. Display metadata is created (step S22). At this time, the storage / reading unit 59 stores the superimposed display metadata in the storage unit 5000.

  Here, mainly with reference to FIG. 21 to FIG. 34, the process of creating the superimposed display metadata will be described in detail. Even if the resolutions of the imaging devices of the general imaging device 3 and the special imaging device 1 are the same, the imaging device of the special imaging device 1 generates an equirectangular projection image that is the basis of the 360 ° omnidirectional image CE. Since all must be covered, the definition per fixed area in the captured image is low.

{Processing for creating metadata for overlay display}
First, a process for creating superimposed display metadata for superimposing a high-definition plane image P on the omnidirectional image CE created by the low-definition equirectangular projection image EC and displaying the same on the display 517. explain. As shown in FIG. 18, the superimposed display metadata includes a position parameter and a correction parameter. Therefore, a method for creating the position parameter and the correction parameter will be mainly described.

  The extraction unit 550 extracts a plurality of feature points fp1 in the rectangular equirectangular projection image EC obtained by the equirectangular projection method and a plurality of feature points fp2 in the rectangular planar image P obtained by the perspective projection method. (Step S110).

  Next, the first corresponding region calculation unit 552 performs the first homography conversion to feature amounts fv1 of the plurality of feature points fp1 in the equirectangular projection image EC and features of the plurality of feature points fp2 in the planar image P. Based on the similarity to the quantity fv2, as shown in FIG. 21, in the equirectangular projection image EC, a quadrangular first corresponding area CA1 corresponding to the planar image P is calculated (step S120). More specifically, the first corresponding area calculation unit 552 calculates the feature amounts fv1 of the plurality of feature points fp1 of the equirectangular projection image EC and the feature amounts fv2 of the plurality of feature points fp2 in the planar image P. FIG. 21 shows a first homography transformation obtained by calculating corresponding points between images based on the similarity and obtaining homography corresponding to the planar image P in the equirectangular projection image EC. As described above, in the equirectangular projection image EC, a quadrangular first corresponding area CA1 corresponding to the planar image P is calculated. This process is a process (provisional decision process) for roughly estimating the corresponding position for the time being, although it is impossible to accurately associate the flat image P having a different projection method with the equirectangular projection image EC.

  Next, the gazing point identification unit 554 identifies a point (gaze point GP1) on the equirectangular projection image EC where the center point CP1 of the planar image P is located after the first homography transformation (step S130).

  Next, the projection method conversion unit 556 sets the vertical field angle α of the peripheral area image PI to be the same as the diagonal field angle α of the planar image P as shown in FIG. Thus, the peripheral area PA centered on the gazing point GP1 in the equirectangular projection image EC is converted into the same perspective projection method as that of the planar image P so that the peripheral area image PI can be created (step S140).

  Next, the extraction unit 550 extracts a plurality of feature points fp3 in the peripheral area image PI obtained by the projection method conversion unit 556 (step S150).

  Next, the second corresponding region calculation unit 558 performs the feature amount fv2 of the plurality of feature points fp2 in the planar image P and the feature amount fv3 of the plurality of feature points fp3 in the peripheral region image PI by the second homography transformation. The second corresponding area CA02 having a square shape corresponding to the planar image P is calculated in the peripheral area image PI based on the similarity to (step S160). Note that the planar image P is a high-definition image of 40 million pixels, for example, and is resized to an appropriate size in advance.

  Next, the corresponding area correction unit 559 corrects the second corresponding area CA02 calculated by the second corresponding area calculation unit 558 to create a second corresponding area CA2. Specifically, as described above, in FIG. 23 (FIG. 27), the block dividing unit 21 shown in FIG. 17 divides the planar image P into a plurality of blocks (steps S161 and S261). Then, the matching unit 22 matches each block divided by the block dividing unit 21 on the correction area image PI (steps S162 and S262).

  Next, the motion vector calculation unit 23 calculates a motion vector from the representative point RP01 and the like in the second corresponding region CA02 with respect to predetermined points at the four corners of the corresponding block on the peripheral region image matched by the matching unit 22. Calculate (steps S163 and S263). Further, the motion vector correction unit 24 corrects the motion vector by the method shown in FIG. 25 (FIGS. 29 and 30) (steps S164 and S264). Thereafter, the representative point correction unit 25 corrects the representative points RP01 and the like to obtain representative points RP1, RP2, RP3, RP4 (RP11, RP14, RP41, RP44), and finally, these four points are each rectangular. A second corresponding area CA2 as a vertex is created (steps S165 and S265).

  Next, the area dividing unit 560 divides the second corresponding area CA2 into a plurality of lattice areas LA2 as shown in FIG. 32B (step S170).

Next, as shown in FIG. 21, the projection method inverse conversion unit 562 converts the projection method of the second corresponding area CA2 into the equirectangular projection method same as the equirectangular projection image EC (inverse conversion). (Step S180). Thereby, as shown in FIG. 33, the projection method inverse transform unit 562 starts from each lattice area LA3 corresponding to the plurality of lattice areas LA2 in the second corresponding area CA2 in the equirectangular projection image EC. A third corresponding area CA3 is calculated. FIG. 33 is a conceptual diagram showing a third corresponding region in the equirectangular projection image EC. By the process of the projection method inverse transform unit 562, a position parameter indicating the coordinates of each lattice point of each lattice region LA3 is created.
The position parameter is shown in FIGS. 18 and 19B as described above.

  Next, correction parameter creation processing will be described with reference to FIGS. 21 and 34. FIG. 34 is a conceptual diagram of an image in the process of the correction parameter creation process.

  After the process of step S180, the shape conversion unit 564 projects the second correspondence by projecting the four vertices of the second corresponding area CA2 as illustrated in FIG. The area CA2 is converted into the same shape as the planar image P, and a second corresponding area CA2 ′ as shown in FIG. 34B is obtained (step S190).

  Next, as shown in FIG. 34C, the region dividing unit 560 converts the planar image P into the same number and the same shape as each lattice region LA2 ′ in the converted second corresponding region CA2 ′. Dividing into a plurality of lattice areas LA0 (step S200).

  Next, the correction parameter creation unit 566 performs the brightness value of each grid area LA0 of the planar image P corresponding to each grid area LA2 ′ with respect to the color value of each grid area LA2 ′ in the second corresponding area CA2 ′. Then, a correction parameter for matching the color value is created (step S210).

  Finally, as shown in FIG. 18, the superimposed display metadata creation unit 570 includes predetermined distance cylindrical projection image information acquired from the special imaging device 1 and planar image information acquired from the general imaging device 3. The superimposed display metadata is created based on the area division number information, the position parameter created by the projection method inverse transform unit 562, the correction parameter created by the correction parameter creation unit 566, and the metadata creation information (step) S220). The superimposed display metadata is stored in the storage unit 5000 by the storage / readout unit 59.

  Thus, the process of step S22 shown in FIG. Then, the storage / reading unit 59 and the display control unit 56 perform superimposition processing using the superimposition display metadata (step S23).

{Superposition processing}
Subsequently, the superimposition process will be described in detail with reference to FIGS. 35 to 40. FIG. 35 is a conceptual diagram of an image in the process of superimposition.

  First, the storage / reading unit 59 (acquisition unit) shown in FIG. 14 obtains from the storage unit 5000 the data of the equirectangular projection image EC obtained by the equirectangular projection method, the perspective projection method. The obtained data of the plane image P and the superimposed display metadata are read out and acquired.

  Next, as shown in FIG. 35, the pasting area creation unit 582 creates a partial solid sphere PS corresponding to the third corresponding area CA3 in the virtual solid sphere CS based on the position parameters (step). S310). At this time, pixels other than the lattice points not indicated by the position parameter are interpolated using, for example, linear interpolation.

  Next, the correction unit 584 performs correction to match the brightness value and color value of the planar image P with the brightness value and color value of the equirectangular projection image EC based on the correction parameter in the superimposed display metadata. (Step S320). Hereinafter, the corrected planar image P is referred to as “corrected image C”.

  Next, the image creation unit 586 creates the superimposed image S by pasting the corrected image C on the partial solid sphere PS (step S330). At this time, pixels other than the lattice points not indicated by the position parameter are interpolated using, for example, linear interpolation. The image creation unit 586 creates mask data M based on the partial solid sphere PS (step S340). Furthermore, the image creation unit 586 creates the omnidirectional image CE by pasting the equirectangular projection image EC on the solid sphere CS (step S350). Then, the image superimposing unit 588 superimposes the superimposed image S on the omnidirectional image CE using the mask data M (step S360). Thereby, the low-definition omnidirectional image CE on which the high-definition superimposed image S is superimposed so that the boundary is not conspicuous is completed.

  Next, as shown in FIG. 7, the projective transformation unit 590 is based on the predetermined visual line direction of the virtual camera IC (the center point CP of the predetermined area T) and the angle of view α of the predetermined area T. Projective transformation is performed so that the predetermined area T in the omnidirectional image CE with the superimposed image S superimposed can be seen on the display 517 (step S370). At this time, the projective transformation unit 590 also performs processing for matching the predetermined area T with the resolution of the display area on the display 517. Thereby, the display control part 56 can perform the process of the display of the predetermined area image Q which shows the predetermined area T over the whole display area of the display 517 (step S24). Here, a superimposed image S that is a planar image P ′ in which the planar image P is superimposed is included in the predetermined region image Q.

  Next, the state of the superimposed display will be described in detail with reference to FIGS. FIG. 36 is a two-dimensional conceptual diagram when a planar image is superimposed on an omnidirectional image. Here, the case where the planar image P is superimposed on FIG. 5 is shown. As shown in FIG. 36, the high-definition superimposed image S is superimposed on the inside of the solid sphere CS according to the position parameter with respect to the low-definition omnidirectional image CE attached to the solid sphere CS. .

  FIG. 37 is a three-dimensional conceptual diagram when a planar image is superimposed on an omnidirectional image. FIG. 37 shows a state in which the omnidirectional image CE and the superimposed image S are pasted on the solid sphere CS and the image including the superimposed image S is the predetermined region image Q.

  FIG. 38 is a two-dimensional conceptual diagram when a planar image is superimposed on an omnidirectional image without using the positional parameters of the present embodiment. FIG. 39 is a two-dimensional conceptual diagram when a planar image is superimposed on an omnidirectional image using the position parameters of this embodiment.

  As shown in FIG. 38A, when the virtual camera IC is positioned at the center point of the solid sphere CS, the subject P1 is represented as an image P2 on the omnidirectional image CE. , Represented as an image P3 on the superimposed image S. As shown in FIG. 38A, the image P2 and the image P3 are located on a straight line connecting the virtual camera IC and the subject P1, and thus the superimposed image S is superimposed on the omnidirectional image CE. Even if displayed in the state, the omnidirectional image CE and the superimposed image S are not displaced. However, as shown in FIG. 38B, when the virtual camera IC is separated from the center point of the solid sphere CS, the image P2 is located on a straight line connecting the virtual camera IC and the subject P1. The image P3 is located slightly inside. For this reason, when an image on the superimposed image S on the straight line connecting the virtual camera IC and the subject P1 is an image P3 ′, the omnidirectional image CE and the superimposed image S are shifted from each other between the image P3 and the image P3 ′. Deviation of the amount g occurs. Thereby, the superimposed image S is shifted from the omnidirectional image CE and displayed.

  On the other hand, in the present embodiment, since the position parameter indicated by the plurality of lattice regions is used, the superimposed image S is represented as an omnidirectional ball as shown in FIGS. 39 (a) and 39 (b). It can be superimposed along the image CE. Thereby, as shown in FIG. 39A, not only when the virtual camera IC is positioned at the center point of the solid sphere CS, but also as shown in FIG. Even when the three-dimensional sphere CS is away from the center point, the image P2 and the image P3 are located on a straight line connecting the virtual camera IC and the subject P1. Therefore, even if the superimposed image S is displayed in a state of being superimposed on the omnidirectional image CE, there is no deviation between the omnidirectional image CE and the superimposed image S.

  40A shows a display example of a wide image without superimposition display, FIG. 40B shows a display example of a tele image without superimposition display, and FIG. 40C shows a display example of a wide image with superimposition display. FIG. 40D is a conceptual diagram showing a display example of a tele image when superimposed display is performed. Note that the wavy lines in the figure are merely shown for convenience of explanation, and may or may not be actually displayed on the display 517.

  As shown in FIG. 40 (a), when the planar image P is not superimposed and displayed on the omnidirectional image CE, when the enlarged display is performed up to the area indicated by the wavy line in FIG. 40 (a), FIG. As shown in (b), the low-definition image remains as it is, and the user will see an unclear image. On the other hand, as shown in FIG. 40 (c), when the planar image P is displayed superimposed on the omnidirectional image CE, it is enlarged to the area indicated by the wavy line in FIG. 40 (c). When displayed, a high-definition image is displayed as shown in FIG. 40D, and the user can see a clear image. In particular, when a signboard or the like on which characters are drawn is displayed in the area indicated by the wavy line, if the high-definition plane image P is not superimposed and displayed, the characters will be blurred even if they are enlarged and displayed. I don't know if is written. However, if the high-definition planar image P is displayed in a superimposed manner, the characters can be clearly seen even when enlarged and displayed, so that the user can grasp what is written.

<< Main effects of this embodiment >>
As described above, according to the present embodiment, even if the other image (for example, the planar image P) is superimposed on one image (for example, the equirectangular projection image EC) having a different projection method, the image shifts. There exists an effect that it can control. More specifically, since the projection methods used when creating the images of the special imaging device 1 and the general imaging device 3 are different, the planar image P acquired from the general imaging device 3 is converted into an equirectangular projection acquired from the special imaging device 1. Even if it is simply superimposed on the omnidirectional image CE created by the image EC, the omnidirectional image CE and the planar image P do not match, and the planar image does not merge well with the omnidirectional image. Therefore, as shown in FIG. 21, the smartphone 5 according to the present embodiment calculates a first corresponding area CA1 corresponding to the planar image P in the equirectangular projection image EC, so that rough superimposition is first performed. The display position is determined (see S120). Then, the smartphone 5 creates the peripheral region image PI by converting the projection method of the peripheral region PA of the gazing point GP1 of the first corresponding region CA1 into the same projection method as that of the planar image P (see S140). Next, the smartphone 5 calculates a second corresponding area CA2 corresponding to the planar image P in the peripheral area image PI (see S160), and sets the projection method of the second corresponding area CA2 to the equirectangular projection image EC. By performing inverse conversion to the projection method, a third corresponding area CA3 corresponding to the second corresponding area CA2 is calculated in the equirectangular projection image EC (see S180). Thereby, as shown in FIG. 40 (c), the high-definition plane image P is superimposed and displayed on a partial area of the predetermined area image that is a part of the low-definition omnidirectional image CE. Therefore, there is an effect that the image of the omnidirectional image and the image of the plane image are combined, and the image of the plane is well merged with the image of the omnidirectional image.

  In addition, when block matching and correction are performed, if the peripheral area image PI is converted into the same shape as the planar image P using the motion vector, the target object is located in the middle of the screen as shown in FIG. . This is almost uncomfortable as compared with FIG. 31 (d), and the influence of the region with few feature points can be corrected by block matching. As described above, correction by block matching and validity determination has an effect of improving not only a matching result between images having a large parallax but also a matching result between images having few feature points.

  Furthermore, according to the present embodiment, the position parameter indicating the position at which the superimposed image S is superimposed on the omnidirectional image CE is indicated by the third corresponding region constituted by a plurality of lattice regions, and thus is shown in FIG. As shown, the superimposed image S can be superimposed along the omnidirectional image CE. Thereby, even if the superimposed image S is superimposed on the omnidirectional image CE and the position of the virtual camera IC is moved, an image shift can be suppressed.

[Second Embodiment]
Subsequently, a second embodiment of the present invention will be described with reference to FIGS. 41 to 45.

<Outline of shooting system>
First, an outline of the configuration of the photographing system of the present embodiment will be described with reference to FIG. FIG. 41 is a schematic diagram of the configuration of the imaging system of the present embodiment.

  As shown in FIG. 41, the imaging system according to the present embodiment further includes an image processing server 7 in addition to the components according to the first embodiment. The same configurations as those of the first embodiment are denoted by the same reference numerals, and description thereof is omitted. The smartphone 5 and the image processing server 7 can mutually communicate with the image processing server 7 via a communication network 100 such as the Internet or an intranet.

In the first embodiment, the smartphone 5 performs the process of creating superimposed display metadata and the process of superimposing images, whereas in the present embodiment, the image processing server 7 performs these processes.
Note that the smartphone 5 of the present embodiment is an example of a communication terminal, and the image processing server is an example of an image processing apparatus.

  The image processing server 7 is a server computer, and includes a case where image processing is performed in a distributed manner by a plurality of server computers.

<< Hardware Configuration of Embodiment >>
Next, the hardware configuration of the image processing server 7 of this embodiment will be described in detail with reference to FIG. FIG. 42 is a hardware configuration diagram of the image processing server. Note that the hardware configurations of the special imaging device 1, the general imaging device 3, and the smartphone 5 according to the present embodiment are the same as those in the first embodiment, and thus the description thereof is omitted.

<Hardware configuration of image processing server>
FIG. 42 is a hardware configuration diagram of the image processing server. The image processing server 7 is constructed by a computer, and as shown in FIG. 6, a CPU 701, ROM 702, RAM 703, HD 704, HDD (Hard Disk Drive) 705, recording medium 706, media I / F 707, display 708. A network I / F 709, a keyboard 711, a mouse 712, a CD-RW drive 714, and a bus line 710. Since the image processing server 7 functions as a server, an input device such as a keyboard 711 and a mouse 712 and an output device such as a display 708 may not be provided.

  Among these, the CPU 701 controls the overall operation of the image processing server 7. The ROM 702 stores a program used for driving the CPU 701. The RAM 703 is used as a work area for the CPU 701. The HD 704 stores various data such as programs. The HDD 705 controls reading or writing of various data with respect to the HD 704 according to the control of the CPU 701. The media I / F 707 controls reading or writing (storage) of data with respect to a recording medium 706 such as a flash memory. The display 708 displays various information such as a cursor, menu, window, character, or image. A network I / F 709 is an interface for performing data communication using the communication network 100. The keyboard 711 is a kind of input means having a plurality of keys for inputting characters, numerical values, various instructions, and the like. The mouse 712 is a kind of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like. The CD-RW drive 714 controls reading of various data with respect to a CD-RW (Compact Disc-ReWritable) 713 as an example of a removable recording medium.

  Further, the image processing server 7 includes a bus line 710. The bus line 710 is an address bus, a data bus, or the like for electrically connecting components such as the CPU 701 shown in FIG.

<< Functional Configuration of Embodiment >>
Next, the functional configuration of this embodiment will be described with reference to FIGS. 43 and 44. FIG. 43 is a functional block diagram of the photographing system according to the present embodiment. In addition, since the functional structure of the special imaging device 1, the general imaging device 3, and the smart phone 5 of this embodiment is the same as that of 1st Embodiment, those description is abbreviate | omitted. In the case of this embodiment, the image / sound processing unit 55 of the smartphone 5 may or may not have the functional configurations shown in FIG.

<Functional configuration of image processing server>
As shown in FIG. 43, the image processing server 7 includes a long-distance communication unit 71, a reception unit 72, an image / sound processing unit 75, a display control unit 76, a determination unit 77, and a storage / reading unit 79. is doing. Each of these units is a function realized by any of the constituent elements shown in FIG. 42 being operated by an instruction from the CPU 701 according to the image processing server 7 program developed from the HD 704 onto the RAM 703, or Means.

  Further, the image processing server 7 has a storage unit 7000 constructed by the ROM 702, the RAM 703, and the HD 704 shown in FIG.

(Functional configuration of image processing server)
The long-distance communication unit 71 of the image processing server 7 is mainly realized by the processing of the network I / F 707 and the CPU 701 illustrated in FIG. 42, and other devices (for example, other servers, Send and receive various data (or information) to and from your smartphone.

  The accepting unit 72 is realized mainly by the processing of the keyboard 711, the mouse 712, and the CPU 701, and accepts various selections or inputs from the user.

  The image / sound processing unit 75 is realized mainly by instructions from the CPU 701 and performs various processes on various data sent from the smartphone 5.

  The display control unit 76 is realized mainly by the processing of the CPU 701, and unlike the display control unit 56 of the first embodiment, creates data of a predetermined area image Q for displaying the planar image P on the display 517 of the smartphone 5. To do. Further, the display control unit 76 uses the superimposed display metadata created by the image / sound processing unit 75 to indicate each lattice area LA0 of the planar image P by the position indicated by the position parameter and the correction parameter. By matching with the brightness value and the color value, data for displaying the planar image P superimposed on the omnidirectional image CE is created.

  The determination unit 77 is realized by the processing of the CPU 701 shown in FIG. 42 and makes various determinations.

  The storage / reading unit 79 is mainly realized by the processing of the CPU 701 shown in FIG. 42, and stores various data (or information) such as superimposed display metadata in the storage unit 7000 or from the storage unit 7000. Various data (or information) such as superimposed display metadata is read out. The storage / reading unit 79 serves as an acquisition unit that acquires various data from the storage unit 7000.

(Detailed functional configuration of image / sound processor)
Here, the functional configuration of the image / sound processor 75 will be described in detail with reference to FIG. FIG. 44 is a detailed functional block diagram of the image / sound processor.

  The image / sound processing unit 75 is roughly divided into a metadata creating unit 75a that performs encoding and a superimposing unit 75b that performs decoding. The metadata creation unit 75a executes a process of step S44 described later shown in FIG. In addition, the superimposing unit 75b executes a process of step S45 described later illustrated in FIG.

{Each functional configuration of the metadata creation unit}
First, each functional configuration of the metadata creation unit 75a will be described. The metadata creation unit 75a includes an extraction unit 750, a first corresponding region calculation unit 752, a gazing point specification unit 754, a projection method conversion unit 756, a second corresponding region calculation unit 758, a corresponding region correction unit 759, and a region division unit. 760, a projection method inverse conversion unit 762, a shape conversion unit 764, a correction parameter creation unit 766, and a superimposed display metadata creation unit 770. These are the extraction unit 550, the first corresponding region calculation unit 552, the gaze point specification unit 554, the projection method conversion unit 556, the second corresponding region calculation unit 558, and the corresponding region correction unit 559, respectively, in the first embodiment. Since it has the same functions as the area dividing unit 560, the projection method inverse conversion unit 562, the shape conversion unit 564, the correction parameter creation unit 566, and the superimposed display metadata creation unit 570, description thereof will be omitted.

{Functional structure of superimposition unit}
Next, the functional configuration of the superimposing unit 75b will be described. The superimposing unit 75b includes a pasting area creating unit 782, a correcting unit 784, an image creating unit 786, a superimposing unit 788, and a projective transformation unit 790. Each of these includes the pasting area creation unit 582, the correction unit 584, the image creation unit 586, the image superimposing unit 588, and the projective transformation unit 590 in the first embodiment, and a description thereof will be omitted.

<< Processing or Operation of Embodiment >>
Subsequently, the processing or operation of the present embodiment will be described with reference to FIG. A photographing method executed by the photographing system will be described with reference to FIG. FIG. 45 is a sequence diagram illustrating a shooting method according to the present embodiment. In addition, since the process of step S31-S41 is a process similar to step S11-S31 of 1st Embodiment, these description is abbreviate | omitted.

  In the smartphone 5, the transmission / reception unit 51 transmits superimposition request information indicating a superimposition request for superimposing the other image on one image having a different projection method to the image processing server 7 via the communication network 100 (step S42). . Data in the electronic folder (planar image data, equirectangular projection image data) stored in the storage unit 5000 is transmitted to the superimposition request information. Thereby, the transmission / reception unit 71 of the image processing server 7 receives the data in the electronic folder.

  Next, in the image processing server 7, the storage / readout unit 79 stores the data in the electronic folder received in step S42 in the storage unit 7000 (step S43). Then, the metadata creating unit 75a shown in FIG. 44 creates superimposed display metadata (step S44). Further, the superimposing unit 75b performs a superimposing process (step S45). Since the processes of steps S44 and S45 have the same contents as the processes of steps S22 and S23, respectively, their descriptions are omitted.

  Next, the display control unit 76 creates data of the predetermined area image Q that displays the predetermined area image Q indicating the predetermined area T over the entire display area of the display 517 of the smartphone 5. Here, a superimposed image S that is a planar image P ′ in which the planar image P is superimposed is included in the predetermined region image Q. The transmission / reception unit 71 transmits the data of the predetermined area image Q created by the display control unit 76 to the smartphone 5 (step S46). Thereby, the transmission / reception unit 51 of the smartphone receives the data of the predetermined area image Q.

  Next, with the smartphone 5. The display control unit 56 displays the predetermined area image Q including the superimposed image S on the display 517 (step S47).

<< Main effects of this embodiment >>
As described above, according to the present embodiment, the same effects as those of the first embodiment can be obtained.

  Further, in the present embodiment, the cooperative shooting process is performed by the smartphone 5, and the creation and superimposition process of the superimposed display metadata is performed by the image processing server 7. In addition, there is an effect that image shift can be suppressed.

[Supplement]
In each of the above embodiments, as shown in FIG. 14, the smartphone 5 has equirectangular projection image data, planar image data, and superimposed display parameter data, but the present invention is not limited to this. For example, a management server that can communicate via a communication network such as the Internet may store at least one of equirectangular projection image data, planar image data, and superimposed display parameter data.

  In each of the above embodiments, the planar image P is superimposed on the omnidirectional image CE, but the present invention is not limited to this. For example, a partial image of the omnidirectional image CE may be replaced with the planar image P, or a partial image of the omnidirectional image CE may be deleted and the planar image P may be inserted into the deleted portion. Good.

  In the second embodiment, the superimposing process is performed by the image processing server 7 (see step S45), but the present invention is not limited to this. For example, the superimposed display metadata may be transmitted from the image processing server 7 to the smartphone 5, and the overlapping process and the display process may be performed on the smartphone 5 side. In this case, in the image processing server 7, the metadata creation unit 75a shown in FIG. 44 creates metadata for superimposed display. On the other hand, in the smartphone 5, the superimposing unit 55b illustrated in FIG. 16 performs the superimposing process, and the display control unit 56 illustrated in FIG. 14 performs the displaying process.

  Moreover, although the case where a planar image is superimposed on an omnidirectional image has been described above, the superimposition is an example of synthesis. In addition to superposition, the composition includes pasting, fitting, superposition, and the like. The superimposed image is an example of a composite image. In addition to the superimposed image, the composite image includes a pasted image, a fitted image, a superimposed image, and the like. Furthermore, the image superimposing units 588 and 788 are examples of an image synthesizing unit.

1 Special shooting device (example of first shooting device)
3 General photographing device (an example of a second photographing device)
5 Smartphone (an example of an image processing device)
7 Image processing server (an example of an image processing device)
21 Block division unit (an example of block division means)
22 Matching part (an example of matching means)
23. Motion vector calculation unit (an example of motion vector calculation means)
24 motion vector correction unit (an example of motion vector correction means)
25 representative point correction unit (an example of representative point correction means)
51 Transmission / Reception Unit 52 Reception Unit 55a Metadata Creation Unit 55b Superimposition Unit 56 Display Control Unit 58 Near Field Communication Unit 59 Storage / Readout Unit (Example of Acquisition Unit)
72 Reception unit 75 Image / sound processing unit 75a Metadata creation unit 75b Superimposition unit 76 Display control unit 78 Short-range communication unit 79 Storage / reading unit (an example of an acquisition unit)
517 Display 550 Extraction Unit (Example of Extraction Unit)
552 1st corresponding area calculation part (an example of the 1st corresponding area calculation means)
554 Gaze point identification part (an example of gaze point identification means)
556 Projection method conversion unit (an example of projection method conversion means)
558 Second Corresponding Area Calculation Unit (Example of Corresponding Area Calculation Unit / Example of Second Corresponding Area Calculation Unit)
559 corresponding area correction unit (an example of corresponding area correction means)
560 area dividing unit (an example of area dividing means)
562 Projection method reverse conversion unit (an example of projection method reverse conversion means)
564 Shape converter (an example of shape converter)
566 Correction parameter creation unit (an example of correction information creation means)
570 Superimposed display metadata creation unit (an example of position calculation means)
582 Attachment area creation unit (an example of attachment area creation means)
584 Correction unit (an example of correction means)
586 Image creation unit (an example of image creation means)
588 Image superimposing unit (an example of image superimposing means)
590 Projection conversion unit (an example of projection conversion means)
750 extraction unit (an example of extraction means)
752 First corresponding area calculation unit (an example of first corresponding area calculation means)
754 Gaze point identification part (an example of gaze point identification means)
756 Projection method conversion unit (an example of projection method conversion means)
758 Second corresponding area calculation unit (an example of a corresponding area calculation unit / an example of a second corresponding area calculation unit)
760 area dividing unit (an example of area dividing means)
762 Projection method inversion unit (an example of projection method inversion unit)
764 Shape converter (an example of shape converter)
766 Correction parameter creation unit (an example of correction information creation means)
770 superimposed display metadata creation unit (an example of position calculation means)
782 Attachment region creation unit (an example of attachment region creation means)
784 Correction unit (an example of correction means)
786 Image creation unit (an example of image creation means)
788 image superimposing unit (an example of image superimposing means)
790 Projection conversion unit (an example of projection conversion means)
5000 storage unit (an example of storage means)
5001 Collaborative photographing apparatus DB (an example of cooperative photographing apparatus means)
7000 storage unit (an example of storage means)

Japanese Patent Laid-Open No. 2006-96487 JP 2012-178135 A

Claims (7)

Acquisition means for acquiring a first image of the first projection method and a second image of a second projection method different from the first projection method;
First projection method conversion means for converting the first image into the second projection method and generating a third image;
Extraction means for extracting a plurality of feature points from each of the second image and the third image;
A second corresponding region corresponding to the second image in the third image based on the plurality of feature points of the second image and the plurality of feature points of the third image respectively extracted by the extraction unit; Corresponding area calculation means for obtaining
Corresponding area correction means for correcting the second corresponding area based on a plurality of blocks in the third image obtained by dividing the second image into blocks and matching with the third image;
Second projection method conversion means for converting a plurality of predetermined points included in the corrected second corresponding region to the first projection method;
Position calculating means for obtaining position information in the first image of the predetermined plurality of points converted by the second projection method converting means;
Storage means for storing the position information obtained by the position calculation means in association with a plurality of points of the second image;
An image processing apparatus comprising: The corresponding area correction means includes
Block dividing means for dividing the second image into a plurality of blocks;
Matching means for matching each block divided by the block dividing means on the third image;
Motion vector calculating means for calculating a motion vector from a representative point in the second corresponding area for a predetermined point of the corresponding block on the third image matched by the matching means;
Representative point correcting means for correcting the second corresponding area by correcting the representative point in the second corresponding area based on the motion vector;
The image processing apparatus according to claim 1, further comprising: The image processing apparatus according to claim 2,
The corresponding area correction means includes
Motion vector correction means for correcting the motion vector calculated by the motion vector calculation means based on the similarity between the first image and the third image;
The representative point correcting unit corrects the second corresponding region by correcting the representative point in the second corresponding region based on the motion vector corrected by the vector correcting unit. The image processing apparatus according to claim 2. The image processing apparatus according to claim 2,
The corresponding area correction means includes
Motion vector correction means for correcting the motion vector calculated by the motion vector calculation means based on a variance value of luminance in the third image;
The representative point correcting unit corrects the second corresponding region by correcting the representative point in the second corresponding region based on the motion vector corrected by the vector correcting unit. The image processing apparatus according to claim 2.   The image processing apparatus according to claim 1, wherein the image processing apparatus is a smartphone, a tablet personal computer, a notebook personal computer, a desktop personal computer, or a server computer. An image processing apparatus according to claim 1;
A first imaging device that images a subject and obtains the first image of the first projection method;
A second imaging device that images a subject and obtains the second image of the second projection method;
An imaging system comprising: An acquisition step of acquiring a first image of the first projection method and a second image of a second projection method different from the first projection method;
A first projection method conversion step of converting the first image into the second projection method and generating a third image;
An extraction step of extracting a plurality of feature points from each of the second image and the third image;
A second corresponding region corresponding to the second image in the third image based on the plurality of feature points of the second image and the plurality of feature points of the third image respectively extracted in the extraction step; A corresponding area calculating step for obtaining
A corresponding area correction step for correcting the second corresponding area based on a plurality of blocks in the third image obtained by dividing the second image into blocks and matching the third image with the third image;
A second projection method conversion step of converting a plurality of predetermined points included in the corrected second corresponding region to the first projection method;
A position calculating step for obtaining position information in the first image of the predetermined plurality of points converted in the second projective method conversion step;
A storage step of storing the position information obtained by the position calculation step in association with a plurality of points of the second image;
The image processing method characterized by performing.