JP2019220994A - Video distribution method and server - Google Patents

Abstract

To provide a video distribution method capable of distributing proper video on the basis of multiple images.SOLUTION: The video distribution method is a video distribution method executed by a server 103 that distributes video signals on the basis of multiple images, which are obtained by picking up the same scene from different multiple viewpoints in successive multiple periods, to multiple terminal units 102. The video distribution method includes: a selection step (S301) of selecting images from multiple images for each of multiple periods on the basis of evaluation values of multiple images by multiple users; and a transmission step (S302) of transmitting video signals for continuously reproducing multiple images selected for multiple periods to the terminal unit 102.SELECTED DRAWING: Figure 18

Description

  The present invention relates to a video distribution method and the like for distributing video captured from a plurality of viewpoints.

  As a video distribution method, for example, a technique described in Patent Document 1 is known. Further, a video distribution method for distributing video captured from a plurality of viewpoints is known (for example, see Patent Document 2). According to such a video distribution method, a user can specify and view an arbitrary video from a plurality of videos obtained by shooting a specific scene from different viewpoints.

JP 2009-206625 A JP 2012-094990 A

  In such a video distribution method, it is desired that an appropriate video can be distributed based on a plurality of videos.

  Therefore, an object of the present invention is to provide a video distribution method or a video reception method that can distribute an appropriate video based on a plurality of videos.

  In order to achieve the above object, a video distribution method according to an aspect of the present invention provides a video distribution method based on a plurality of videos obtained by capturing the same scene from a plurality of different viewpoints in a plurality of continuous periods. A video distribution method by a server that distributes one video signal to a terminal device, wherein a first selection is made from the plurality of videos for each of the plurality of periods based on evaluation values of the plurality of videos by a plurality of users. A first selection step of selecting a video, and a first transmission of transmitting the first video signal for continuously reproducing the plurality of the first selection videos selected for the plurality of periods to the terminal device Steps.

  Further, the video receiving method according to one aspect of the present invention receives a video signal from a server based on a plurality of videos obtained by capturing the same scene from a plurality of different viewpoints in a plurality of continuous periods. A video receiving method by a terminal device, wherein for a plurality of periods, a plurality of selected videos selected from the plurality of videos based on evaluation values of the plurality of videos by a plurality of users are continuously reproduced. Receiving the video signal from the server, and reproducing the plurality of selected videos continuously based on the video signal.

  Note that these general or specific aspects may be realized by a system, a method, an integrated circuit, a computer program or a recording medium such as a computer-readable CD-ROM, and the system, the method, the integrated circuit, and the computer program. And any combination of recording media.

  The present invention can provide a video distribution method or a video reception method that can distribute an appropriate video based on a plurality of videos.

FIG. 1 is a diagram illustrating a configuration of a video distribution system according to a first embodiment. FIG. 3 is a block diagram of a terminal device according to Embodiment 1. 5 is a flowchart illustrating a process performed by the terminal device according to Embodiment 1. 6 is a flowchart illustrating a viewpoint video switching process performed by the terminal device according to Embodiment 1. 6 is a flowchart illustrating an evaluation adding process by the terminal device according to the first embodiment. 5 is a flowchart illustrating an information acquisition process performed by the terminal device according to Embodiment 1. FIG. 3 is a block diagram of a server according to Embodiment 1. 5 is a flowchart showing processing by the server according to the first embodiment. 5 is a flowchart illustrating a viewpoint video switching process performed by a server according to Embodiment 1. 5 is a flowchart illustrating an evaluation adding process by the server according to the first embodiment. 5 is a flowchart illustrating an information acquisition process performed by the server according to the first embodiment. FIG. 5 is a diagram showing an operation example of a favorite subject switching mode according to the first embodiment. FIG. 5 is a diagram showing an operation example of a recommended viewpoint video switching mode according to the first embodiment. FIG. 5 is a diagram illustrating an operation example of a recommended subject switching mode according to the first embodiment. FIG. 4 is a diagram showing an example of a display screen according to the first embodiment. FIG. 4 is a diagram showing an example of a display screen according to the first embodiment. FIG. 4 is a diagram showing an example of a display screen according to the first embodiment. 4 is a flowchart of a video distribution method by a server according to Embodiment 1. 5 is a flowchart of a video receiving method by the terminal device according to Embodiment 1. FIG. 1 is a diagram illustrating a configuration of a video information processing system. It is a figure showing an example of a notice screen displayed at the time of camera activation. 1 is an overall configuration diagram of a content supply system that realizes a content distribution service. 1 is an overall configuration diagram of a digital broadcasting system. It is a figure showing an example of a smart phone. It is a block diagram which shows the example of a structure of a smart phone.

  The video distribution method according to one aspect of the present invention distributes a first video signal to a terminal device based on a plurality of videos obtained by capturing the same scene from a plurality of different viewpoints in a plurality of continuous periods. Selecting a first selected video from the plurality of videos for each of the plurality of periods based on evaluation values of the plurality of videos by a plurality of users. And a first transmission step of transmitting the first video signal for continuously reproducing the plurality of first selected videos selected for the plurality of periods to the terminal device.

  Accordingly, the video distribution method can distribute an appropriate video based on evaluations of a plurality of users.

  For example, the evaluation value is an evaluation value of each of the plurality of videos, and in the first selecting step, for each of the plurality of periods, the evaluation value in the period from the plurality of images is the highest or An image higher than a predetermined value may be selected as the first selected image.

  Accordingly, the video distribution method can distribute a video in which a highly evaluated viewpoint video is continuously reproduced based on the evaluations of a plurality of users.

  For example, the evaluation value is an evaluation value of each of a plurality of subjects included in the plurality of videos, and in the first selection step, in each of the plurality of periods, the evaluation value is obtained from the plurality of videos. An image in which a recommended object having the highest or higher than a predetermined value is shown may be selected as the first selected image.

  Accordingly, the video distribution method can distribute a video that continuously reproduces a viewpoint video in which a subject with a high evaluation is captured, based on the evaluations of a plurality of users.

  For example, in the first selecting step, in each of the plurality of periods, an image having the largest area in which the recommended subject is shown may be selected from the plurality of images as the first selected image.

  Accordingly, the video distribution method can distribute a video that continuously reproduces a viewpoint video in which a subject with a high evaluation is largely displayed, based on the evaluations of a plurality of users.

  For example, when the data amount of the first video signal is larger than an upper limit value based on a reception band of the terminal device, the video distribution method further includes, based on the evaluation values of the plurality of videos, each of the plurality of periods. A second selection step of selecting a second selection image from the plurality of images; and a second selection step of continuously reproducing the second selection image selected for each of the plurality of periods. Transmitting a video signal to the terminal device, wherein at least one of the plurality of second selected videos selected for the plurality of periods is selected for the plurality of periods. Unlike the plurality of first selection images, the data amount of the second image signal may be smaller than the upper limit.

  Thus, the video distribution method can distribute a video signal with a small data amount when the reception band of the terminal device is narrow.

  For example, the plurality of videos are transmitted from the plurality of cameras to the server in real time, and when the transmission of the video corresponding to the first selected video from the camera is interrupted, the first selection step includes: A second selected video different from the one selected video may be selected, and in the first transmitting step, a second video signal for reproducing the second selected video may be transmitted to the terminal device.

  Accordingly, the video distribution method can suppress interruption of video distribution when transmission of video from the camera is interrupted.

  For example, the evaluation value is an evaluation value of each of the plurality of videos, and in the first selection step, the evaluation value in the period is predetermined from the plurality of videos for each of the plurality of periods. An image lower than the selected value may be selected as the first selected image.

  Thus, the video distribution method can reduce the occurrence of blind spots in, for example, monitoring applications.

  For example, in the first transmitting step, the video signal may be generated by joining the plurality of selected videos selected for the plurality of periods, and the video signal may be transmitted to the terminal device.

  For example, in the transmitting step, the plurality of videos and information for specifying the plurality of selected videos selected for the plurality of periods may be transmitted to the terminal device as the video signal. .

  Further, the video receiving method according to one aspect of the present invention receives a video signal from a server based on a plurality of videos obtained by capturing the same scene from a plurality of different viewpoints in a plurality of continuous periods. A video receiving method by a terminal device, wherein for a plurality of periods, a plurality of selected videos selected from the plurality of videos based on evaluation values of the plurality of videos by a plurality of users are continuously reproduced. Receiving the video signal from the server, and reproducing the plurality of selected videos continuously based on the video signal.

  Accordingly, the video receiving method can reproduce an appropriate video based on the evaluations of a plurality of users.

  For example, the evaluation value is an evaluation value of each of the plurality of videos, and the selected video is selected from the plurality of videos for each of the plurality of periods. The image may be high or higher than a predetermined value.

  Thus, the video receiving method can continuously reproduce a highly evaluated viewpoint video based on the evaluations of a plurality of users.

  For example, the evaluation value is an evaluation value of each of a plurality of subjects included in the plurality of videos, and the selected video is selected from the plurality of videos in each of the plurality of periods. It may be an image in which a subject that is the highest or higher than a predetermined value is shown.

  Accordingly, the video receiving method can continuously reproduce a viewpoint video in which a subject with a high evaluation is captured based on the evaluations of a plurality of users.

  For example, the video receiving method further includes a display step of displaying any of the plurality of videos, and in the display step, among the plurality of subjects included in the displayed video, the evaluation value is the highest. Alternatively, a subject higher than a predetermined value may be highlighted.

  Thus, the video receiving method can notify a user of a subject with a high evaluation.

  For example, the image receiving method further includes a first display step of displaying any of the plurality of images, and a step of selecting one of a plurality of subjects included in the displayed image by a user. A second display step of displaying a menu for evaluating the selected subject.

  Thus, the video receiving method can improve user convenience.

  For example, the image receiving method further includes a first display step of displaying any of the plurality of images, and a step of selecting one of a plurality of subjects included in the displayed image by a user. And a second display step of displaying a list of videos of a plurality of periods including the selected subject from among the plurality of videos and having an evaluation value higher than a predetermined value.

  Thus, the video receiving method can improve user convenience.

  For example, the image receiving method further includes a first display step of displaying any of the plurality of images, and a step of selecting one of a plurality of subjects included in the displayed image by a user. A second display step of displaying a tracking image, which is an image obtained by connecting the plurality of images and tracks the selected subject.

  Thus, the video receiving method can improve user convenience.

  For example, the video receiving method further includes a first screen on which any one of the plurality of videos is displayed, and a second screen on which the reproduced plurality of selected videos are smaller than the first screen and are displayed. And a second display step of displaying the plurality of selected images on the first screen when the second screen is selected by the user.

  Thus, the video receiving method can improve user convenience.

  For example, the evaluation value is an evaluation value of each of the plurality of videos, and the selected video is selected from the plurality of videos for each of the plurality of periods. The image may be lower than a predetermined value.

  Further, the server according to one embodiment of the present invention is a server that distributes, to a terminal device, video signals based on a plurality of videos obtained by capturing the same scene from a plurality of different viewpoints in a plurality of continuous periods. A selection unit for selecting a selected video from the plurality of videos for each of the plurality of periods based on evaluation values of the plurality of videos by a plurality of users; A transmitting unit that transmits the video signal for continuously reproducing the plurality of the first selected videos to the terminal device.

  Thereby, the server can distribute an appropriate video based on the evaluations of a plurality of users.

  In addition, a terminal device according to one embodiment of the present invention is a terminal that receives, from a server, a video signal based on a plurality of videos obtained by capturing the same scene from a plurality of different viewpoints in a plurality of continuous periods. The apparatus, for the plurality of periods, the video signal for continuously reproducing a plurality of selected videos selected from the plurality of videos based on evaluation values of the plurality of videos by a plurality of users, A receiving unit that receives from the server; and a reproducing unit that continuously reproduces the plurality of selected videos based on the video signal.

  Thereby, the terminal device can reproduce an appropriate video based on the evaluations of a plurality of users.

  Note that these general or specific aspects may be realized by a system, a method, an integrated circuit, a computer program or a recording medium such as a computer-readable CD-ROM, and the system, the method, the integrated circuit, and the computer program. And any combination of recording media.

  Hereinafter, embodiments will be specifically described with reference to the drawings. Each of the embodiments described below shows a specific example of the present invention. Numerical values, shapes, materials, constituent elements, arrangement positions and connection forms of constituent elements, steps, order of steps, and the like shown in the following embodiments are merely examples, and do not limit the present invention. Further, among the components in the following embodiments, components not described in the independent claims indicating the highest concept are described as arbitrary components.

(Embodiment 1)
In the video distribution system according to the present embodiment, a selected video is selected from a plurality of viewpoint videos based on evaluations by a plurality of users in each period, and video data obtained by joining the selected plurality of selected videos is transmitted to a terminal device. To deliver.

  First, the configuration of the video distribution system 100 according to the present embodiment will be described. FIG. 1 is a block diagram showing an overall configuration of a video distribution system 100 according to the present embodiment. The video distribution system 100 includes a plurality of cameras 101, a terminal device 102, and a server 103, each of which can communicate via a network 104A or 104B.

  The plurality of cameras 101 generate a plurality of viewpoint videos by shooting the same scene from different viewpoints in the same time zone. Each camera 101 is carried by each of a plurality of users. For example, the plurality of cameras 101 are owned by a plurality of spectators in a place such as a sports stadium. The plurality of viewpoint videos captured by the plurality of cameras 101 are transmitted to the server 103 via the network 104A. The camera 101 also acquires data such as position information used for calculating the degree of association between viewpoint videos, and transmits the data to the server 103. The camera 101 may be any device having at least a photographing function, and is, for example, a digital still camera, a digital video camera, a smartphone, a mobile terminal, or the like. Further, the plurality of cameras 101 may include not only cameras owned by spectators but also fixed cameras or broadcast cameras.

  The networks 104A and 104B are, for example, the Internet. Although the networks 104A and 104B are individually illustrated in FIG. 1, a plurality of cameras 101, the terminal device 102, and the server 103 may be connected via a single network. Further, part or all of the communication between the devices may be directly performed without going through the network 104A or 104B. The connection between the devices may be either a wired connection or a wireless connection.

  The terminal device 102 is a terminal used by the user 106, and is connected to a monitor 105 that displays an image. For example, the terminal device 102 is a smartphone, a mobile terminal, a personal computer, or the like. Note that the terminal device 102 has the same function as the camera 101, and the user may be included in the audience, or the user may view video from a place other than the stadium.

  The terminal device 102 receives the viewpoint video from the server 103 according to the instruction of the user 106, and outputs the viewpoint video to the monitor 105 by a method according to the instruction of the user 106. Note that the monitor 105 may be any output device such as a PC monitor, a tablet terminal, a smartphone, a mobile phone, or a monitor of a notebook PC. Although the terminal device 102 and the monitor 105 are separately described here, the monitor 105 may be included in the terminal device 102.

  Further, the instruction from the user is performed by, for example, a screen touch operation or a mouse, but any input device may be used. The instructions of the user 106 include selecting a viewpoint video to be viewed, leaving a comment on the viewpoint video, evaluating the viewpoint video, leaving a comment on the subject shown in the viewpoint video, evaluating the subject, and examining information on the subject. And instructions for ordering related goods relating to the subject. A part of the user's instruction is transmitted to the server 103.

  The server 103 temporarily stores a plurality of viewpoint videos transmitted from the plurality of cameras 101. In addition, the server 103 transmits a part or all of the plurality of held viewpoint videos to the terminal device 102 according to a request from the terminal device 102. In addition, the server 103 performs processing such as switching of viewpoint video according to a request from the terminal device 102, and transmits video data obtained by the processing to the terminal device 102. Further, the server 103 receives, from the terminal device 102, the user's evaluation of the viewpoint video or the subject in the viewpoint video, tallies the comment and the evaluation of the viewpoint video or the subject, and stores the result in the database.

  Further, the server 103 transmits information to the terminal device 102 in accordance with the viewpoint video or the information presentation request for the subject from the terminal device 102. Further, the server 103 uses the information on the subject specified by the user transmitted from the terminal device 102 to specify the subject included in the plurality of viewpoint videos captured by the camera 101. Specifically, the server 103 stores the subject information (name, uniform number, etc.) designated by the user, or the subject in the viewpoint video designated by the user through a touch operation, a mouse operation, a gesture operation, a line-of-sight operation, or a voice operation. The subject is specified by image processing such as feature amount matching between a plurality of viewpoint videos captured by the camera 101 and the received image information. Note that the server 103 may create a combined image by combining a plurality of viewpoint videos captured by the plurality of cameras 101. Further, the composite image may be included in a plurality of viewpoint videos transmitted to the terminal device 102.

  Note that a plurality of viewpoint videos may be transmitted in real time from the plurality of cameras 101, and the user 106 may view the viewpoint videos in real time using the terminal device 102, or at least one of transmission and viewing of the video may be performed in real time. May not be performed. That is, the user 106 may view the viewpoint video captured in the past at an arbitrary timing. Further, transmission and reception of a video signal (video) described below mainly mean stream transmission and reception in which a video signal is continuously transmitted or received.

  Hereinafter, the configuration of each device will be described. FIG. 2 is a block diagram showing the configuration of the terminal device 102. The terminal device 102 includes a receiving unit 111, a storage unit 112, an image processing unit 113, an output unit 114, a transmitting unit 115, and a control unit 116.

  The receiving unit 111 receives the video data transmitted from the server 103. Specifically, the video data includes a viewpoint video specified by the user. Further, the video data includes, as sub-data, a video having a high degree of association with the viewpoint video. For example, this video is another viewpoint video whose shooting position is close to the viewing viewpoint video being viewed, another viewpoint video including a subject in the viewing viewpoint video being viewed, or the like. In this way, the server 103 may transmit a video having a high degree of association with the viewpoint video being viewed to the terminal device 102 at the same time as the viewing video. The video data includes not only the viewpoint video specified by the user, but also comments, evaluations, related information, and the like on the viewpoint video or the subject. Further, the server 103 transmits information on the viewpoint video with the largest number of views counted by the server 103 and the like.

  The storage unit 112 temporarily stores the video data received by the reception unit 111. This enables seamless display when the viewpoint is switched.

  The image processing unit 113 performs a decoding process when the viewpoint video is encoded. Further, the image processing unit 113 generates an output video to be presented to the user 106 using the viewpoint video, the information on the viewpoint video received from the server 103, and the information on the subject. Note that, at this time, the image processing unit 113 separately obtains a viewpoint video with a large number of views and statistical information of a subject with a large number of comments or a high evaluation from the server 103 and outputs an output video that emphasizes the information. May be created. For example, the image processing unit 113 may emphasize a selection icon of a viewpoint video having a large number of views by illuminating a selection icon of the viewpoint video or illuminating a contour of a subject with a high evaluation.

  The output unit 114 outputs the output video generated by the image processing unit 113 to the monitor 105.

  Control unit 116 receives a user instruction. Here, the user's instruction includes (1) a viewpoint video switching request, (2) an evaluation addition request, and (3) an information acquisition request.

  The (1) viewpoint video switching request is a request for switching a viewpoint video to be viewed. (1-1) favorite viewpoint video switching mode, (1-2) favorite subject switching mode, and (1-3) recommended viewpoint video. A switching mode and a (1-4) recommended subject switching mode are included.

  (1-1) The favorite viewpoint video switching mode is a mode in which the user 106 selects a favorite viewpoint video registered in advance. (1-2) The favorite subject switching mode is a mode for selecting a viewpoint video including a favorite subject registered in advance by the user 106. In the favorite viewpoint video switching mode and the favorite subject switching mode, the user can select a viewpoint video or a subject at any time while watching the video, as well as the favorite viewpoint video or the viewpoint video including the subject registered in advance by the user.

  (1-3) The recommended viewpoint video switching mode is a mode for selecting viewpoint videos that are highly evaluated by a plurality of users. (1-4) The recommended subject switching mode is a mode for selecting a viewpoint video including a subject highly evaluated by a plurality of users.

  The (2) evaluation addition request is a request for evaluating a viewpoint video or a subject, and includes (2-1) a viewpoint video evaluation addition mode and (2-2) a subject evaluation addition mode. (2-1) The viewpoint video evaluation addition mode is a mode for the user 106 to evaluate the viewpoint video. (2-2) The subject evaluation addition mode is a mode for the user 106 to evaluate a subject included in the viewpoint video.

  (3) The information acquisition request is a request for acquiring viewpoint video or subject information, and includes (3-1) viewpoint video information acquisition mode and (3-2) subject information acquisition mode. (3-1) The viewpoint video information acquisition mode is a mode for acquiring viewpoint video information. (3-2) The subject information acquisition mode is a mode for acquiring information on a subject included in the viewpoint video.

  When the user's instruction is (1) a viewpoint video switching request, the control unit 116 transmits information on the specified viewpoint video position or information on the specified subject position to the server 103 via the transmission unit 115.

  Further, when the user's instruction is (2) an evaluation addition request, the control unit 116 determines whether the comment or evaluation information for the viewpoint video and the time information at the time of the evaluation addition, or the comment or evaluation information for the designated subject and the The time information and the like at the time of evaluation addition are transmitted to the server 103 via the transmission unit 115.

  When the user's instruction is (3) an information acquisition request, the control unit 116 transmits the information on the position of the designated viewpoint video, the information on the designated subject position, the information on the information acquisition content, and the like to the server 103. Via the transmission unit 115. Here, the information acquisition content is information on the viewpoint video and the subject, such as (1) comments or evaluations on the viewpoint video or the subject, (2) the profile of the photographer or the subject of the viewpoint video, or (3) the viewpoint. The information is information on other content shot by the video photographer, or related goods information on the subject.

  When the user specifies the subject as in (1-2) favorite subject switching mode, (2-2) subject evaluation addition mode, or (3-2) subject information acquisition mode, In order to specify the subject designated by the user on the server 103 side, subject information (name or uniform number, etc.) designated by the user, or a touch operation, a mouse operation, a gesture operation, a line-of-sight operation, a voice operation, etc. The pixel information of the subject in the specified viewpoint video is transmitted to the server 103.

  Hereinafter, the operation of the terminal device 102 will be described. FIG. 3 is a flowchart of the entire process performed by the terminal device 102.

  First, the terminal device 102 determines whether the user instruction is (1) a viewpoint video switching request, (2) an evaluation addition request, or (3) an information acquisition request (S101, S103, S105). .

  When the user instruction is a viewpoint video switching request (Yes in S101), the terminal device 102 performs a viewpoint video switching process (S102). When the user's instruction is an evaluation addition request (Yes in S103), the terminal device 102 performs an evaluation addition process (S104). When the user's instruction is an information acquisition request (Yes in S105), the terminal device 102 performs an information acquisition process (S106).

  Note that the terminal device 102 does not necessarily need to perform all of the above determinations. For example, the terminal device 102 may treat the above determinations as options, such as determining whether the user's instruction is a viewpoint video switching request. Absent.

  Hereinafter, details of the viewpoint video switching process (S102), the evaluation adding process (S104), and the information obtaining process (S106) will be described.

  First, the viewpoint video switching process (S102) will be described. FIG. 4 is a flowchart of the viewpoint video switching process performed by the terminal device 102.

  First, the terminal device 102 determines whether the viewpoint video switching mode is (1-1) the favorite viewpoint video switching mode, (1-2) the favorite subject switching mode, or (1-3) the recommended viewpoint video switching mode. Or (1-4) recommended subject switching mode (S121, S123 and S126).

  When the viewpoint video switching mode is the favorite viewpoint video switching mode (Yes in S121), the terminal device 102 requests the server 103 to distribute the viewpoint video at the viewpoint video position designated by the user 106 (S122).

  When the viewpoint video switching mode is the favorite subject switching mode (Yes in S123), the terminal device 102 transmits information on the subject designated by the user to the server 103 (S124). Here, the subject information is, for example, subject information such as a subject name or a uniform number, or pixel information of a subject in a viewpoint video specified by a user through a touch operation or a mouse operation. Acquisition of subject information (name or uniform number, etc.) is performed, for example, by the user specifying subject information output on the screen by a touch operation, a mouse operation, a voice, or the like. Further, the pixel information of the subject in the viewpoint video includes, for example, the pixel position or the pixel information of the subject in the viewpoint video.

  In addition, the terminal device 102 requests the server to distribute the tracking video of the subject designated by the user (S125).

  When the viewpoint video switching mode is the recommended viewpoint video switching mode (Yes in S126), the terminal device 102 requests the server 103 to distribute the recommended viewpoint video generated by the server 103 (S127).

  When the viewpoint video switching mode is the recommended subject switching mode (No in S126), the terminal device 102 requests the server 103 to distribute the recommended subject tracking video generated by the server 103 (S128).

  After steps S122, S125, S127, or S128, the terminal device 102 receives the video data transmitted from the server 103, and stores the received video data in the storage unit 112 (S129). The video data received from the server 103 may include, in addition to the viewpoint video, a video produced by processing or editing on the server 103 side.

  Further, the terminal device 102 does not need to wait until the reception of all the video data received from the server 103 is completed, and may start the reproduction from a point in time when a certain amount of data is received. Further, the terminal device 102 may receive the video data from the server 103, and may reproduce the video data that has been reproduced before until the reproduction can be started.

  Next, the terminal device 102 extracts video data to be displayed from the storage unit 112, and decodes the video data (S130). Next, the terminal device 102 generates a synthesized video (output video) by synthesizing the decoded video data and the display user interface (S131).

  Next, the terminal device 102 outputs the synthesized video to the monitor 105 such as a display (S132). Thus, the composite video is displayed on the monitor 105.

  Hereinafter, the evaluation adding process (S104) will be described. FIG. 5 is a flowchart of the evaluation adding process by the terminal device 102.

  First, the terminal device 102 determines whether the evaluation addition mode is the (2-1) viewpoint video evaluation addition mode or (2-2) the subject evaluation addition mode (S141).

  When the evaluation addition mode is the viewpoint video evaluation addition mode (Yes in S141), the terminal device 102 transmits the evaluation information for the viewpoint video being viewed by the user 106 and the time information at the time of the evaluation addition to the server 103 (S142). . Here, the evaluation information is a comment or an evaluation on the viewpoint video. The evaluation may be information indicating whether the viewpoint video is good or bad, or information indicating the degree of good or bad.

  When the evaluation addition mode is the subject evaluation addition mode (No in S141), the terminal device 102 transmits information on the subject specified by the user to the server 103 (S143). Note that the content of the subject information and the method of acquiring the information are the same as in step S124 described above.

  Acquisition of subject information (name or uniform number, etc.) is performed, for example, by the user specifying subject information output on the screen by a touch operation, a mouse operation, a voice, or the like. Further, the pixel information of the subject in the viewpoint video includes, for example, the pixel position or the pixel information of the subject in the viewpoint video.

  Next, the terminal device 102 transmits the evaluation information for the subject designated by the user and the time information at the time of the evaluation addition to the server 103 (S144).

  Hereinafter, the information acquisition process (S106) will be described. FIG. 6 is a flowchart of the information acquisition processing by the terminal device 102.

  First, the terminal device 102 determines whether the information acquisition mode is the (3-1) viewpoint video information acquisition mode or (3-2) the subject information acquisition mode (S161).

  When the information acquisition mode is the viewpoint video information acquisition mode (Yes in S161), the terminal device 102 transmits a request to acquire information on the viewpoint video being viewed by the user to the server 103 (S162). Here, the information related to the viewpoint video is, for example, a comment or an evaluation on the viewpoint video, a profile of a photographer of the viewpoint video, or information of another content photographed by the photographer of the viewpoint video.

  When the information acquisition mode is the subject information acquisition mode (No in S161), the terminal device 102 transmits information on the subject designated by the user to the server 103 (S163). Note that the content of the subject information and the method of acquiring the information are the same as in step S124 described above.

  Next, the terminal device 102 transmits a request to acquire information on the subject designated by the user to the server 103 (S164). Here, the subject information is, for example, a comment or evaluation on the subject, a profile of the subject, or related goods information of the subject.

  After step S162 or S164, the terminal device 102 receives the viewpoint video or the information on the subject transmitted from the server (S165). Next, the terminal device 102 adds the received viewpoint video or the information regarding the subject to the display user interface, and generates a composite video by synthesizing the interface with the video data being reproduced (S166). Note that the terminal device 102 may superimpose the received viewpoint video or information on the subject on the video data being reproduced.

  Next, the terminal device 102 outputs the generated composite video to the monitor 105 (S167). Thereby, the composite video is displayed on the monitor 105.

  Hereinafter, the configuration of the server 103 will be described. FIG. 7 is a block diagram illustrating the configuration of the server 103. The server 103 illustrated in FIG. 7 includes a receiving unit 121, a storage unit 122, an image processing unit 123, a transmitting unit 124, and a control unit 126.

  The receiving unit 121 receives, from the plurality of cameras 101, a plurality of viewpoint videos obtained by the plurality of cameras 101 shooting the same scene from different viewpoints in the same time zone, and shooting camera information. The photographing camera information includes, for example, position information of the camera 101, photographing time information, photographer profile information, and the like. Here, the photographing camera position information is, for example, information obtained by a sensor such as a GPS built in the photographing camera. Note that the photographing camera position information may be wireless information such as WiFi (registered trademark) or position information estimated from a feature amount of a photographed video. Further, the server 103 may estimate the position of the camera 101 from wireless information such as WiFi or a feature amount of a captured video.

  The storage unit 122 stores the plurality of viewpoint videos and photographing camera information received by the receiving unit 121. The storage unit 122 stores comments or evaluations added by the user to the viewpoint video or the subject, together with time information to which the comments or evaluations were added. Further, the storage unit 122 stores collected data such as statistics on a viewpoint video or a subject having a large number of comments or a high evaluation. The storage unit 122 may store information on the viewpoint video and the subject, such as the profile of the photographer or the subject of the viewpoint video, or information on the related goods of the subject.

  The image processing unit 123 generates video data by processing or editing the viewpoint video in response to the viewpoint video switching request from the terminal device 102, and transmits the generated video data to the terminal device 102 via the transmission unit 124. Send. Further, the image processing unit 123 includes a subject indicated by the image information by performing feature amount matching between the pixel information of the subject in the viewpoint video received from the terminal device 102 and a plurality of viewpoint videos captured by the camera 101, or the like. The viewpoint video is specified, and the specified result is sent to the control unit 126.

  The transmission unit 124 transmits the video data generated by the image processing unit 123.

  The control unit 126 receives data from the terminal device 102 and the Internet (network 104A or 104B). Specifically, the control unit 126 receives a viewpoint video switching request, an evaluation addition request, and an information acquisition request from the terminal device 102. Further, the control unit 126 creates a search key for the subject, acquires a profile of the subject, information on related goods, and the like by Internet search, and stores it in the storage unit 122 or transmits it to the terminal device 102. The control unit 126 transmits and receives data to and from a web store for purchasing related goods.

  Hereinafter, the operation of the server 103 will be described. FIG. 8 is a flowchart of the entire process performed by the server 103.

  First, the server 103 receives the plurality of viewpoint videos transmitted from the plurality of cameras 101 and stores the plurality of viewpoint videos in the storage unit 122 together with the shooting camera information (S201).

  Next, the server 103 determines whether a viewpoint video switching request has been received from the terminal device 102 (S202). When the viewpoint video switching request is received (Yes in S202), the server 103 performs a viewpoint video switching process (S203).

  The server 103 determines whether an evaluation addition request has been received from the terminal device 102 (S204). When the evaluation addition request is received (Yes in S204), the server 103 performs an evaluation addition process (S205).

  The server 103 determines whether an information acquisition request has been received from the terminal device 102 (S206). When receiving the information acquisition request (Yes in S206), the server 103 performs an information acquisition process (S207).

  Note that the processing order illustrated in FIG. 8 is an example, and the order of the determination processing in steps S202, S204, and S206 may be arbitrary.

  Note that the evaluation information to be added to the viewpoint video or the subject is not necessarily limited to information obtained directly from the user 106, and may be obtained separately. For example, these pieces of evaluation information may be separately acquired from an SNS or the like.

  Hereinafter, details of the viewpoint video switching process (S203), the evaluation addition process (S205), and the information acquisition process (S207) will be described.

  First, the viewpoint video switching process (S203) will be described. FIG. 9 is a flowchart of the viewpoint video switching process performed by the server 103.

  First, the server 103 determines whether the viewpoint video switching mode is (1-1) the favorite viewpoint video switching mode, (1-2) the favorite subject switching mode, or (1-3) the recommended viewpoint video switching mode. It is determined whether or not there is (1-4) the recommended subject switching mode (S221, S223, and S226).

  When the viewpoint video switching mode is the favorite viewpoint video switching mode (Yes in S221), the server 103 selects the viewpoint video at the received viewpoint video position from the storage unit 122, and sets the viewpoint video in the video data to be transmitted to the terminal device 102. (S222).

  When the viewpoint video switching mode is the favorite subject switching mode (Yes in S223), the server 103 specifies the subject specified by the user using the subject information received from the terminal device 102 (S224). Specifically, the server 103 specifies the subject by feature matching or the like using the subject information (name or uniform number, etc.) or the pixel information of the subject in the viewpoint video and a plurality of viewpoint videos of a group of photographing cameras. .

  Next, the server 103 selects a plurality of viewpoint videos including the identified subject from the storage unit 122 and transmits the viewpoint videos to the terminal device 102 by switching the viewpoint videos every time according to the ratio of the area occupied by the subject in the video. Then, video data to be generated is generated (S225). The server 103 may preferentially select not only the ratio of the area occupied by the subject in the video but also, for example, a viewpoint video in which the frontal face of the subject is always taken, or the subject is always displayed in the middle. The selected viewpoint video may be preferentially selected. In other words, the server 103 may select a video with a good image of the subject according to the shooting situation.

  If the viewpoint video switching mode is the recommended viewpoint video switching mode (Yes in S226), the server 103 automatically switches the viewpoint video every time in accordance with the statistical data of the viewpoint video having a high evaluation, and thereby the terminal device 102 Then, video data to be transmitted to is generated (S227).

  Here, the viewpoint video with a high evaluation is a viewpoint video that has been highly evaluated by the user. For example, a menu for giving a high evaluation is prepared for each viewpoint video, and when the user selects the menu, the high evaluation of the viewpoint video increases by one. The viewpoint video with a high evaluation is a viewpoint video with the highest number of high evaluations or with a number higher than a predetermined value. Further, a menu for giving a low evaluation may be prepared. In this case, the evaluation of the viewpoint video may be determined based on the difference between the high evaluation and the low evaluation.

  The evaluation given by the user may be a stepwise numerical value such as 1 to 5. For example, 5 indicates the best and 1 indicates the worst. In such a case, for example, the evaluation of the viewpoint video may be determined according to the average value of the evaluation.

  In addition, the number of evaluations (both high evaluation and low evaluation) may be added to the determination. Also, it may be determined that a viewpoint video having a large number of reproductions (viewing times) has a high evaluation or a viewpoint video having a long reproduction time (viewing time) has a high evaluation.

  Further, a menu for the user to add a comment to the viewpoint video may be prepared, and it may be determined that the higher the number of comments by the user, the higher the evaluation. Further, the content of the comment may be analyzed, and a high evaluation, a low evaluation, or a graded numerical value may be determined according to the content of the comment.

  Alternatively, it may be determined that the viewpoint video having the larger number of favorite registrations by the user has higher evaluation.

  Further, two or more of the above determination methods may be combined.

  Further, the statistical data may be statistical data of the entire shooting scene or statistical data of each period in which the shooting scene is divided. For example, in the case of a sports game, the statistical data may be statistical data for the entire period of the game, or may be statistical data for the first half and the second half, or for each predetermined period. Further, the statistical data may be statistical data for a long period beyond the shooting scene. For example, in the case of a sports game, the statistical data may be all statistical data of a plurality of games performed in one week or one month, or may be statistical data in one or more seasons. Good.

  If the viewpoint video switching mode is the recommended subject switching mode (No in S226), the server 103 specifies the subject from the statistical data of the subject with a high evaluation (S228). It should be noted that a subject with a high evaluation is determined by a method similar to that for a viewpoint video with a high evaluation.

  Next, the server 103 selects a plurality of viewpoint videos including the identified subject from the storage unit 122, and switches the viewpoint videos every time according to the ratio of the area occupied by the subject in the video, thereby providing the terminal device 102 with the viewpoint video. The video data to be transmitted is generated (S229). The server 103 may preferentially select not only the ratio of the area occupied by the subject in the video but also, for example, a viewpoint video in which the frontal face of the subject is always taken, or the subject is always displayed in the middle. The selected viewpoint video may be preferentially selected. In other words, the server 103 may select a video with a good image of the subject according to the shooting situation.

  After steps S222, S225, S227 or S229, the server 103 transmits the generated video data to the terminal device 102 (S230).

  Note that, in the favorite viewpoint video switching mode or the favorite subject switching mode, the server 103 does not include a moving subject in the viewpoint video, or interrupts shooting due to lack of a battery of a camera that is capturing the viewpoint video. When a specific event has occurred, the event may be detected, and the video to be automatically viewed may be switched to another viewpoint video whose shooting position is close to the viewpoint video viewed by the user 106. The server 103 may switch the switching mode to the recommended viewpoint video switching mode or the recommended subject switching mode. Conversely, the server 103 may switch the switching mode to the favorite viewpoint video switching mode or the favorite subject switching mode when a specific event occurs in the recommended viewpoint video switching mode or the recommended subject switching mode. Good.

  Also, the number of comments and the evaluation value for the subject need not necessarily be limited to those added by the user, and the server 103 may separately acquire them. For example, the server 103 may set an initial value in advance based on the number of search results of the Web search using the name of the subject.

  Hereinafter, the evaluation adding process (S205) will be described. FIG. 10 is a flowchart of the evaluation adding process by the server 103.

  First, the server 103 determines whether the evaluation addition mode is the (2-1) viewpoint video evaluation addition mode or (2-2) the subject evaluation addition mode (S241).

  When the evaluation addition mode is the viewpoint video evaluation addition mode (Yes in S241), the server 103 receives, from the terminal device 102, evaluation information that is a comment or an evaluation on the viewpoint video being viewed by the user and time information at the time of the evaluation addition. Then, the received evaluation information of the viewpoint video is stored in the storage unit 122 (S242).

  If the evaluation addition mode is the subject evaluation addition mode (No in S241), the server 103 specifies the subject using the information on the subject received from the terminal device 102 (S243). The contents of the subject information and the method of specifying the subject are the same as in step S224 described above.

  Next, the server 103 receives the evaluation information for the subject designated by the user and the time information at the time of adding the evaluation from the terminal device 102, and stores the received subject evaluation information in the storage unit 122 (S244).

  After step S <b> 242 or S <b> 244, the server 103 periodically updates the viewpoint video or the statistic information of the subject with high evaluation using the viewpoint information or the evaluation information for the subject stored in the storage unit 122 and the time information when the evaluation is added. Then, the updated statistical information is stored in the storage unit 122 (S245).

  Hereinafter, the information acquisition process (S207) will be described. FIG. 11 is a flowchart of the information acquisition process by the server 103.

  First, the server 103 determines whether the information acquisition mode is (3-1) viewpoint video information acquisition mode or (3-2) subject information acquisition mode (S261).

  When the information acquisition mode is the viewpoint video information acquisition mode (Yes in S261), the server 103 reads from the storage unit 122 information related to the viewpoint video being viewed by the user, and transmits the read information to the terminal device 102 (S262). ). Here, the information related to the viewpoint video is, for example, a comment or an evaluation on the viewpoint video, a profile of a photographer of the viewpoint video, or information of another content photographed by the photographer of the viewpoint video.

  When the information acquisition mode is the subject information acquisition mode (No in S261), the server 103 specifies the subject designated by the user 106 using the subject information received from the terminal device 102 (S263). Note that the content of the subject information and the method of obtaining the information are the same as in step S224 described above. In addition, the server 103 may narrow down the subjects to a few people by feature amount matching or the like, and allow the user 106 to select a target subject from among them.

  Next, the server 103 reads out information on the subject designated by the user 106 from the storage unit 122, and transmits the read out information to the terminal device 102 (S264). Here, the subject information is, for example, a comment or evaluation on the subject, a profile of the subject, or related goods information of the subject.

  Hereinafter, an operation example of the viewpoint video switching process will be described. FIG. 12 is a diagram illustrating an operation example of the favorite subject switching mode.

  In this example, the server 103 selects the viewpoint video A, the viewpoint video B, and the viewpoint video C including the specified subject from the storage unit 122, and selects the viewpoint video for each time according to the ratio of the area occupied by the subject in the video. By switching the video, video data to be transmitted to the terminal device 102 is generated. In the first period, the ratio of the area occupied by the subject specified by the user in the viewpoint video A is large. Therefore, in this period, the viewpoint video A is selected. In the next period, the ratio of the area occupied by the subject specified by the user in the viewpoint video C is large. Therefore, in this period, the viewpoint video C is selected. In the last period, the proportion of the area occupied by the subject specified by the user in the viewpoint video B is large. Therefore, in this period, the viewpoint video B is selected. In this manner, an image in which a plurality of viewpoint images are connected according to the area of the subject is generated.

  If the continuous reproduction time of a certain viewpoint video does not exceed the threshold, the server 103 may perform control so that the viewpoint video is not switched until the continuous playback time exceeds the threshold. This can prevent frequent switching of viewpoint videos. For example, even if the area of the subject is larger in the viewpoint video C than in the viewpoint video A, if the continuous reproduction time of the viewpoint video A does not exceed 5 seconds, for example, the server 103 And play. Further, at the timing of switching to the viewpoint video C, the server 103 switches only when the subject is captured in the viewpoint video C for, for example, two seconds or more.

  FIG. 13 is a diagram illustrating an operation example of the recommended viewpoint video switching mode. As illustrated in FIG. 13, the server 103 generates video data to be transmitted to the terminal device 102 by switching viewpoint videos at intervals according to statistical data of viewpoint videos having a high evaluation. Note that the server 103 may perform switching control according to the continuous playback time, as in the case of the favorite subject switching mode.

  FIG. 14 is a diagram illustrating an operation example of the recommended subject switching mode. As shown in FIG. 14, the server 103 selects a plurality of viewpoint videos including a subject with a high evaluation from the storage unit 122, and switches the viewpoint videos every time according to the ratio of the area occupied by the subject in the video. , And generates video data to be transmitted to the terminal device 102.

  Note that the server 103 may perform switching control according to the continuous playback time, as in the case of the favorite subject switching mode.

  Further, in the above description, when performing the above-described switching, an example has been described in which a single video stream is generated by performing the switching in the server 103 and the video stream is transmitted to the terminal device 102. Part of the processing may be performed by the terminal device 102. For example, the server 103 transmits a plurality of viewpoint videos and information indicating switching timing of the viewpoint videos to the terminal device 102, and the terminal device 102 reproduces (displays) the plurality of viewpoint videos based on the information. The viewpoint video to be switched may be switched.

  Hereinafter, an example of a display screen on the terminal device 102 will be described. FIG. 15 is a diagram illustrating an example of an output screen of the terminal device 102. FIG. 15 shows a state where a live video is displayed.

  The output screen 300A illustrated in FIG. 15 includes a viewpoint video display screen 301, a viewpoint selection screen 302, operation buttons 303A to 303C, 304A and 304B, and information display screens 305 and 306.

  The viewpoint video display screen 301 displays a viewpoint video selected by the user or video data generated by the server 103 in the mode selected by the user.

  When the user selects a player on the viewpoint video display screen 301 by touching or clicking the mouse, a circle-shaped operation menu 307C is displayed. The operation menu 307C includes a plurality of menus, and menus of “high evaluation”, “low evaluation”, “player information”, and “favorite” are arranged clockwise from the top. When the user selects the “high evaluation” or “low evaluation” menu, an evaluation (high evaluation or low evaluation) is added to the selected player (subject evaluation addition mode).

  In addition, the evaluation may be performed by a method of adding points by 1 to 6 points, or by any method such as a method of adding by a comment such as “very good” or “good”. Further, as the evaluation method, not only text or voice but also a method of transmitting a comment by a picture, for example, may be used.

  The value of points may be improved by setting an upper limit on the number of points that can be added by each user. Further, the upper limit of points that can be added may be increased by paying money by the user.

  In addition, when the user selects the “player information” menu, information about the players such as comments, evaluations, related goods information, or scores on the players is displayed (subject information acquisition mode).

  In addition, when the user selects the “favorite” menu, the viewpoint video at or before and after the selection may be registered as a favorite play of the selected player (subject evaluation addition mode).

  Further, the server 103 or the terminal device 102 may access a website or the like related to related goods information, scores, or information on players, and the terminal device 102 may display the content. The server 103 or the terminal device 102 may access a Web site or the like where related goods can be purchased, and the terminal device 102 may display the content. This allows the viewer to easily purchase goods while watching the game. Therefore, the present service provider can connect the viewer with the goods selling company through the present service, and can expect the effect of obtaining the margin from the goods selling company.

  In addition, when the user selects the “favorite” menu, the selected player is registered as the user's favorite player. Then, the player registered in the favorite subject switching mode is used as a favorite subject.

  Here, an example is shown in which only one viewpoint video display screen 301 exists, but a plurality of viewpoint video display screens may be displayed simultaneously. For example, the output screen includes three viewpoint video display screens, the first viewpoint video display screen displays a viewpoint video of the entire court, and the second viewpoint video display screen specifies the viewpoint video. Is displayed (favorite viewpoint video switching mode), and a viewpoint video (favorite subject switching mode) of the subject designated by the user is displayed on the third viewpoint video display screen. Further, the viewpoint image display screen 301 may display an image created by combining a plurality of viewpoint images. In this case, for example, viewing can be performed from multiple angles by changing the angle left, right, up, and down while the user touches the screen.

  In the viewpoint video display screen 301, a player with a high evaluation may be highlighted. For example, the outline of a player with a high evaluation may be displayed shining.

  The viewpoint selection screen 302 is a diagram showing the positions of the plurality of cameras 101 (the viewpoint positions of the viewpoint videos). For example, in the example shown in FIG. 15, the positions of the plurality of cameras 101 are indicated by circles on the top view of the match venue.

  The positions of the hatched circles indicate the viewpoint positions of the viewpoint video currently displayed on the viewpoint video display screen 301. When the user selects the circle at each viewpoint position by touching or clicking the mouse, the displayed viewpoint image is switched to the viewpoint image at that position (favorite viewpoint image switching mode).

  Note that, on the viewpoint selection screen 302, the viewpoint position of a viewpoint video with a high evaluation may be highlighted. Further, the server 103 may analyze voice data such as cheers measured at each viewpoint position, and the viewpoint position at which cheers occur most at that time may be highlighted on the viewpoint selection screen 302. Further, the viewpoint videos in which the cheers are most likely may be switched and displayed in order.

  In addition, when the user selects the viewpoint position on the viewpoint selection screen 302 by touching or long-pressing with a mouse click or the like, a circle-shaped operation menu 307A is displayed. The operation relating to the operation menu 307A is the same as the operation relating to the operation menu 307C described above, except that the target is changed from a player (subject) to a viewpoint video.

  Specifically, the operation menu 307A includes a plurality of menus, and a menu of “high evaluation”, “low evaluation”, “viewpoint video information”, and “favorite” is arranged clockwise from the top. When the user selects the “high evaluation” or “low evaluation” menu, an evaluation (high evaluation or low evaluation) is added to the selected viewpoint video (view video evaluation addition mode). In addition, the evaluation may be performed by a method of adding points by 1 to 6 points, or by any method such as a method of adding by a comment such as “very good” or “good”.

  When the user selects the “viewpoint video information” menu, a comment, an evaluation, a photographer's profile, or other content information of the photographer on the viewpoint video are displayed (viewpoint video information acquisition mode).

  Further, when the user selects the menu of “favorite”, the viewpoint video at the time of selection or before and after the selection may be registered as a favorite time zone of the selected viewpoint video. (Viewpoint video evaluation addition mode). The registered favorite time zone is used at the time of recording and viewing. For example, at the time of recording and viewing, another viewpoint video of the favorite time zone is displayed.

  When the user selects the “favorite” menu, the selected viewpoint video is registered as a favorite video. Then, the registered favorite video is used as the viewpoint video in the favorite viewpoint video switching mode.

  The operation buttons 303A to 303C are buttons for selecting a favorite subject switching mode, a recommended subject switching mode, and a recommended viewpoint video switching mode, respectively.

  The operation buttons 304A and 304B are buttons for switching between live video and recorded video. For example, when the user selects the operation button 304A, a live video is displayed on the viewpoint video display screen 301. When the user selects the operation button 304B, the recorded video is displayed on the viewpoint video display screen 301.

  The information display screen 305 displays information on the current game and the like. For example, a circle with a number shown in FIG. 15 indicates a player's uniform number. Also, for example, the upper number is the number of the player who is currently participating in the game, and the lower number is the number of the reserve player.

  When the user selects the player's uniform number by touching or clicking the mouse, a circle-shaped operation menu 307B is displayed. The operation relating to the operation menu 307B is the same as the operation relating to the operation menu 307C described above.

  The information display screen 306 displays a total of evaluations added to the subject (player) by the user, and the like. Similarly to the information display screen 305, a circle with a number indicates a player's uniform number. On the right side of each uniform number, the evaluation of a plurality of users for the player with the uniform number is shown. Specifically, the upper row shows the number of high evaluations, and the lower row shows the number of low evaluations. In the example shown in FIG. 15, six levels (1 to 6) are evaluated for each of the high evaluation and the low evaluation, and the numbers to which the evaluations of each level are given are described. In this evaluation, values totalized for each predetermined period are displayed.

  At the left end of the information display screen 306, the time registered as a favorite scene with the vertical direction as a time axis is indicated by an asterisk.

  When the user clicks on the player's uniform number, the details of the comment and the evaluation added to the player are displayed.

  Further, the total value displayed on the information display screen 306 may be updated at regular intervals. Alternatively, this total value may be updated each time an evaluation is added by the user. Further, the points added to the subject may be fed back to the subject, for example, the points are reflected in the annual salary of the subject.

  FIG. 16 is a diagram illustrating an example of an output screen of the terminal device. FIG. 16 shows a state where the recorded video is displayed. For example, in the output screen 300A shown in FIG. 15, when the user selects the operation button 304B, the output screen 300B shown in FIG. 16 is displayed.

  The output screen 300B shown in FIG. 16 includes an information display screen 308 instead of the information display screen 305 in the output screen 300A shown in FIG.

  The horizontal direction of the output screen 300B indicates the time axis, and the degree of excitement (for example, the volume of cheers) is displayed by a waveform. In addition, the time of favorite registration is indicated by a star in the lower row, and the time of a recommended scene is indicated by a circle in the middle. The lower circle indicates the number of the player who has scored one team, and the numerical value above it indicates the score. Similarly, the upper circle indicates the number of the player who scored the score of the other team, and the lower numerical value indicates the score.

  When a viewpoint video developed as a recommendation is viewed by more people, points may be returned to the photographer of the viewpoint video.

  Further, at the time of displaying the recorded video, the viewing point may be displayed in accordance with the viewpoint video or the subject to which the user has added a favorite, the viewpoint video or the subject with a high evaluation, or audio data such as cheers.

  When a certain subject is selected, a play collection according to the evaluation added to the subject may be displayed in a ranking format or the like, and the play collection may be reproduced. For example, when the user selects the player with the uniform number 5, a play collection in which high points (for example, 6 points) are added to the player with the uniform number 5 is displayed. Then, one play selected from among them is reproduced.

  Further, at the time of displaying a comment of each evaluation, a list of viewpoint videos taken by the user who posted the comment may be displayed at the same time. The photographer may be registered as a favorite or the like. In this case, there may be a mode in which only a viewpoint video taken by a favorite registered photographer can be viewed.

  FIG. 17 is a diagram showing another example of the output screen according to the present embodiment. The output screen 300C shown in FIG. 17 includes a main screen 311, a first sub screen 312A, a second sub screen 312B, and a viewpoint selection screen 313.

  On the main screen 311, a viewpoint video specified by the user is displayed (favorite viewpoint video switching mode). The recommended viewpoint video generated in the recommended viewpoint video switching mode described above is displayed on the first sub-screen 312A. On the second sub-screen 312B, a recommended subject image generated in the recommended subject switching mode described above is displayed.

  In this state, when the user clicks the first sub-screen 312A, the recommended viewpoint video is displayed on the main screen 311. Then, a viewpoint video recommended next is displayed on the first sub-screen 312A.

  Similarly, when the user clicks the second sub-screen 312B, the recommended subject video is displayed on the main screen 311. Then, the next recommended subject image is displayed on the second sub-screen 312B.

  When the user designates a subject in the viewpoint video displayed on the main screen 311 by clicking or the like, the mode shifts to a favorite subject switching mode, and a tracking video of the designated subject is displayed. Furthermore, if the user clicks the main screen 311 while the favorite subject video is being displayed, the display is switched to the previously viewed viewpoint video.

  Here, an example is shown in which the recommended viewpoint video and the recommended subject video are displayed on the first sub-screen 312A and the second sub-screen 312B, but the first recommended viewpoint video is displayed on the first sub-screen 312A. The second recommended viewpoint video may be displayed on the second sub-screen 312B. Similarly, the first recommended subject image may be displayed on the first sub-screen 312A, and the second recommended subject image may be displayed on the second sub-screen 312B.

  Alternatively, the information display screen 305, 306, or 308 shown in FIG. 15 or FIG. 16 may be displayed with the display of the first sub screen 312A and the second sub screen 312B turned off.

  When the user double-clicks the main screen 311 or the like, only the main screen 311 may be displayed on the full screen on the display.

  Note that the viewpoint selection screen 313 is similar to the viewpoint selection screen 302 shown in FIGS.

  Hereinafter, a method of delivering the video (tracking video) edited by the server 103 to the terminal device 102 will be described. First, the server 103 generates a first tracking video of the subject A and a second tracking video of the subject B. Although an example in which two tracking images are generated is described here for simplification of description, three or more tracking images may be generated. Further, the video transmitted from the server 103 is not limited to the tracking video, and may be one of viewpoint videos, a favorite viewpoint video, a recommended viewpoint video, or the like.

  The server 103 transmits a plurality of videos generated by multicast (broadcast) to the terminal device 102. Specifically, the server 103 distributes the first tracking video on 1ch and the second tracking video on 2ch. The terminal device 102 switches the video to be displayed by performing channel switching according to the user's request.

  By using this method, the required band can be made constant. This makes it possible to easily cope with a case where the number of users (the number of terminal devices 102) is large.

  Although there is a problem that it is difficult to cope with the case where the number of users is large because the required bandwidth fluctuates, transmission and reception of one video requested by the user from the server 103 to a certain terminal device 102 or between terminal devices is performed. Is also good. When the number of users is small, there is also an advantage that the required bandwidth can be reduced.

  Hereinafter, processing when a communication error occurs will be described.

  Ideally, the server 103 generates a recommended viewpoint video or a recommended subject video according to the evaluation result of the user. On the other hand, in the following cases, these images cannot be generated or transmitted.

  First, the following problem may occur between the server 103 and the terminal device 102.

  (1) When the amount of data of the transmission video (recommended viewpoint video or recommended subject video, etc.) is larger than the receivable band of the terminal device 102. In such a case, any of the following methods can be used.

  (1-1) The server 103 reduces the resolution or frame rate of the transmission video, re-encodes the video, and transmits the obtained video to the terminal device 102. As described above, the transmission video whose data amount is reduced is used.

  Note that the server 103 may generate video signals of a plurality of bit rates by converting the resolution or frame rate of the viewpoint video transmitted from the camera 101, and store the plurality of video signals.

  Further, when the bandwidth available to the terminal device 102 fluctuates during viewing, the server 103 may switch the transmission video according to the available bandwidth.

  Further, the server 103 may determine the number of videos to be transmitted according to a communication band that can be used by the terminal device 102. Specifically, the server 103 increases the number of transmission videos as the communication band is wider.

  (1-2) The server 103 substitutes another image close to the target image (for example, a shooting position is close, an image is close, etc.) with a small data amount.

  (2) A case in which an image or an area that is not to be distributed is shown in the transmission image or the viewpoint image used for generating the transmission image. In such a case, any of the following methods can be used.

  (2-1) The server 103 substitutes a video having no problem in another video close to the target video (for example, a shooting position is close, a reflected image is close, and the like).

  (2-2) The server 103 substitutes another highly evaluated video. For example, the server 103 substitutes the video with the second highest evaluation if there is no similar problem.

  Further, the following problem may occur between the camera 101 (photographer) and the server 103.

  (3-1) The video from the camera 101 does not reach the server 103 due to packet loss or the like. (3-2) The case where the band of the camera 101 is unstable and the data transfer to the server 103 is delayed. (3-3) When the photographer suddenly interrupts photographing (insufficient battery, thermal problem, etc.). In such a case, any of the following methods can be used.

  (4-1) The server 103 substitutes a viewpoint video with high evaluation or another video close to the subject video (for example, a shooting position is close, a reflected image is close, or the like).

  (4-2) The server 103 substitutes another highly evaluated video. For example, the server 103 substitutes the video with the second highest evaluation.

  Hereinafter, a method of transmitting a video other than a video with a high user evaluation will be described.

  Here, a viewpoint video that is not popular is a video that is not much viewed by people, that is, a video that is not noticed. Note that the unpopular viewpoint video is a video with a small number of reproductions, a video with a low evaluation, a video with a small number of evaluations or comments, and the like. However, such a viewpoint video may also be useful for some people, and the server 103 may connect and distribute viewpoint videos that have not received much attention. For example, the above system can be applied to a surveillance application (for example, a surveillance system including a plurality of surveillance cameras). In such a case, it is assumed that a plurality of people monitor images of a plurality of monitoring cameras. In this case, the viewpoint video viewed by a plurality of persons is sufficiently monitored, but the viewpoint video that is rarely viewed may become a blind spot. Therefore, the server 103 connects and distributes viewpoint videos that have not received much attention (viewpoint videos with a small number of viewers). This image is displayed on a separate screen beside the viewpoint image that is being watched by the observer. As a result, blind spots that cannot be monitored by the observer can be reduced.

  In addition, when there is a viewpoint video that has not been watched for a certain period of time, the monitoring staff may be notified or warned of that, or information indicating the viewpoint video that has not been watched may be notified to the monitor. Here, the viewpoint video that has not been noticed for a certain period of time is, for example, a viewpoint video that has not been reproduced for a certain period of time. When a plurality of viewpoint videos are displayed simultaneously, a viewpoint video that has not been watched for a certain period of time is a viewpoint video that has not been selected as a video to be displayed on the main screen for a certain period of time, or a viewpoint video that has been evaluated for a certain period of time. It may be a viewpoint video that is not taken. Further, the viewpoint video that has not been noticed for a certain period of time may be a video in which the number of times of reproduction, the number of selections, or the number of evaluations is equal to or less than a predetermined threshold.

  For example, on the viewpoint selection screen 302 shown in FIG. 15, a viewpoint position of a viewpoint video that has not been noticed for a certain time may be highlighted. For example, the color of the viewpoint position may be changed to a color different from the colors of other viewpoint positions. Accordingly, it is possible to encourage the observer to monitor a viewpoint video that is not noticed.

  As described above, server 103 according to the present embodiment performs the processing shown in FIG.

  The server 103 distributes a first video signal based on a plurality of videos obtained by capturing the same scene from a plurality of different viewpoints in a plurality of continuous periods to the terminal device 102.

  First, the server 103 selects a first selected video from a plurality of videos for each of a plurality of periods based on evaluation values of a plurality of videos by a plurality of users (S301). Specifically, the server 103 selects a video having a high evaluation value as a first selected video.

  More specifically, as described in the recommended viewpoint video switching mode described above, the evaluation value is an evaluation value of each of the plurality of videos, and the server 103 transmits the plurality of videos for each of the plurality of periods. The video having the highest evaluation value or higher than a predetermined value in the period is selected as the first selected video.

  Alternatively, as described in the recommended subject switching mode described above, the evaluation value is an evaluation value of each of the plurality of subjects included in the plurality of videos, and the server 103 performs the processing from the plurality of videos in each of the plurality of periods. Then, an image in which the recommended subject having the highest evaluation value or higher than a predetermined value is shown is selected as the first selection image. For example, in each of the plurality of periods, the server 103 selects, from the plurality of videos, the video having the largest area in which the recommended subject is shown as the first selected video.

  Note that, as described in the method of transmitting a video other than a video having a high user evaluation, the evaluation value is an evaluation value of each of a plurality of videos, and the server 103 transmits a plurality of videos for each of a plurality of periods. Therefore, an image in which the evaluation value is lower than a predetermined value in the period may be selected as the first selected image.

  Next, the server 103 transmits, to the terminal device 102, a first video signal for continuously reproducing a plurality of first selected videos selected for a plurality of periods (S302). Specifically, the server 103 generates a video signal by connecting a plurality of selected videos selected for a plurality of periods, and transmits the video signal to the terminal device 102. Alternatively, the server 103 transmits, to the terminal device 102, a plurality of videos and information for specifying the selected video selected for each of the plurality of periods as a video signal. In this case, the terminal device 102 generates a video signal by connecting a plurality of selected videos selected for a plurality of periods using the above information.

  Also, as described in the processing when a communication error occurs, when the data amount of the first video signal is larger than the upper limit based on the reception band of the terminal device 102, the server 103 further evaluates a plurality of videos. A second selection video from the plurality of videos for each of the plurality of periods based on the value, and a second selection video for continuously reproducing the plurality of second selection videos selected for the plurality of periods. The video signal is transmitted to the terminal device 102. Here, at least one of the second selection images selected for each of the plurality of periods is different from the first selection image selected for each of the plurality of periods, and the data amount of the second image signal is It is smaller than the above upper limit.

  Also, the plurality of videos are transmitted from the plurality of cameras 101 to the server 103 in real time, and the server 103 differs from the first selected video when the transmission of the video corresponding to the first selected video from the camera 101 is interrupted. The second selected video is selected, and a second video signal for reproducing the second selected video is transmitted to the terminal device 102.

  Further, terminal apparatus 102 according to the present embodiment performs the processing shown in FIG.

  The terminal device 102 receives from the server 103 video signals based on a plurality of videos obtained by capturing the same scene from a plurality of different viewpoints in a plurality of continuous periods.

  First, the terminal device 102 transmits, to the server 103, a video signal for continuously reproducing a plurality of selected videos selected from the plurality of videos based on the evaluation values of the plurality of videos by the plurality of users for a plurality of periods. (S311). Specifically, the selected video is a video having a high evaluation value.

  More specifically, as described in the recommended viewpoint video switching mode described above, the evaluation value is an evaluation value of each of the plurality of videos, and the selected video is a plurality of videos from each of the plurality of periods. The selected video has the highest evaluation value in the period or is higher than a predetermined value.

  Alternatively, as described in the recommended subject switching mode described above, the evaluation value is an evaluation value of each of a plurality of subjects included in a plurality of videos, and the selected video is selected from a plurality of videos in each of a plurality of periods. This is the video showing the subject having the highest evaluation value or higher than a predetermined value.

  Alternatively, as described in the method of transmitting a video other than a video having a high user evaluation, the evaluation value is an evaluation value of each of a plurality of videos, and the selected video is a plurality of videos for each of a plurality of periods. Is an image whose evaluation value in the period is lower than a predetermined value.

  Next, the terminal device 102 continuously reproduces a plurality of selected videos based on the received video signal (S312). Specifically, the terminal device 102 receives a video signal generated by connecting a plurality of selected videos selected for each of a plurality of periods in the server 103, and reproduces the video signal. Alternatively, the terminal device 102 receives, from the server 103, a plurality of videos and information for specifying the selected video selected for each of the plurality of periods, and A video signal is generated by joining a plurality of selected videos selected for the video signal, and the video signal is reproduced.

  In addition, the terminal device 102 displays any of the plurality of videos, and highlights the subject having the highest evaluation value or higher than a predetermined value among the plurality of subjects included in the displayed video. Is also good.

  Further, as shown in FIG. 15, the terminal device 102 displays any of the plurality of videos, and when any of the plurality of subjects included in the displayed video is selected by the user, the selection is made. An operation menu 307C for evaluating the performed subject may be displayed.

  In addition, the terminal device 102 displays any of the plurality of videos, and when any of the plurality of subjects included in the displayed video is selected by the user, the terminal device 102 selects the plurality of videos. May be displayed in a list for a plurality of periods in which the subject is included and the evaluation value is higher than a predetermined value.

  Also, as shown in FIG. 17, the terminal device 102 displays any of the plurality of videos, and when any of the plurality of subjects included in the displayed video is selected by the user, And a tracking image for tracking the selected subject may be displayed.

  Also, as shown in FIG. 17, the terminal device 102 displays a first screen (main screen 311) on which any of a plurality of videos is displayed, and a plurality of reproduced selected videos smaller than the first screen. And an output screen 300C including a second screen (a first sub-screen 312A or a second sub-screen 312B). When the user selects the second screen, a plurality of selected images are displayed on the first screen. May be.

  Note that part of the processing performed by the server 103 may be performed by the terminal device 102, and part of the processing performed by the terminal device 102 may be performed by the server 103. That is, the video distribution system 100 according to the present embodiment reproduces (displays) a video signal based on a plurality of videos obtained by capturing the same scene from a plurality of different viewpoints in a plurality of consecutive periods. I do. The video distribution system 100 selects a selected video from a plurality of videos for each of a plurality of periods based on evaluation values of a plurality of videos by a plurality of users. The video distribution system 100 generates a video signal by joining a plurality of selected videos selected for a plurality of periods, and reproduces (displays) the video signal.

  As described above, the video distribution method, the video reception method, and the video distribution system according to the embodiment have been described, but the present invention is not limited to this embodiment.

  Further, each processing unit included in each device included in the video distribution system according to the above-described embodiment is typically realized as an LSI which is an integrated circuit. These may be individually integrated into one chip, or may be integrated into one chip so as to include some or all of them.

  Further, the integrated circuit is not limited to the LSI, and may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after manufacturing the LSI, or a reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI may be used.

  In each of the above embodiments, each component may be configured by dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

  In other words, each device included in the video distribution system includes a processing circuit (processing circuit) and a storage device (storage) electrically connected to the processing circuit (accessible from the processing circuit). The processing circuit includes at least one of dedicated hardware and a program execution unit. When the processing circuit includes a program execution unit, the storage device stores a software program executed by the program execution unit. The processing circuit executes the video distribution method or the wave video reception method according to the above-described embodiment using the storage device.

  Further, the present invention may be the software program or a non-transitory computer-readable recording medium on which the program is recorded. Needless to say, the program can be distributed via a transmission medium such as the Internet.

  In addition, the numbers used above are all examples for specifically describing the present invention, and the present invention is not limited to the illustrated numbers.

  The order in which the steps included in the video distribution method or the video reception method are performed is merely an example for specifically describing the present invention, and may be any other order. Also, some of the above steps may be performed simultaneously (in parallel) with other steps.

  As described above, the video distribution method, the video reception method, the video distribution system, the server, and the terminal device according to one or more aspects of the present invention have been described based on the embodiments. However, the present invention is not limited to this embodiment. It is not limited. Unless departing from the spirit of the present invention, various modifications conceivable to those skilled in the art may be applied to the present embodiment, and a form constructed by combining components in different embodiments may be one or more of the present invention. It may be included within the scope of the embodiment.

(Embodiment 2)
Another application example of the configuration of the image processing method and apparatus described in each of the above embodiments and a system using the same will be described. This system can be applied to a video system that is becoming more intelligent and a wider target space. For example, (1) a security system installed in a store or factory security camera, or a police car-mounted camera, (2) an individual A traffic information system using a proprietary camera or each on-vehicle camera, or a camera provided on a road; (3) an environmental investigation or delivery system using a device that can be remotely controlled or automatically controlled such as a drone; and (4) entertainment The present invention can be applied to a content transmission / reception system such as an installation camera in a facility or a stadium, a mobile camera such as a drone, or a video using a personally owned camera.

  FIG. 20 is a diagram illustrating a configuration of a video information processing system ex100 according to the present embodiment. In the present embodiment, an example of preventing occurrence of a blind spot and an example of prohibiting shooting of a specific area will be described.

  The video information processing system ex100 illustrated in FIG. 20 includes a video information processing device ex101, a plurality of cameras ex102, and a video receiving device ex103. Note that the video receiving device ex103 does not necessarily need to be included in the video information processing system ex100.

  The video information processing device ex101 includes a storage unit ex111 and an analysis unit ex112. Each of the N cameras ex102 has a function of capturing a video and a function of transmitting captured video data to the video information processing apparatus ex101. In addition, the camera ex102 may have a function of displaying a video being captured. Note that the camera ex102 converts the captured video signal into HEVC or H.264. H.264 may be encoded and then transmitted to the video information processing apparatus ex101, or unencoded video data may be transmitted to the video information processing apparatus ex101.

  Here, each camera ex102 is a fixed camera such as a surveillance camera, a mobile camera mounted on an unmanned flight radio control, a car, or the like, or a user camera owned by a user.

  The mobile camera receives the instruction signal transmitted from the video information processing apparatus ex101, and changes the position or the shooting direction of the mobile camera according to the received instruction signal.

  In addition, the times of the plurality of cameras ex102 are calibrated using the time information of the server or the reference camera before the start of shooting. Further, the spatial positions of the plurality of cameras ex102 are calibrated based on how the object in the space to be photographed is imaged or the relative position from the reference camera.

  The storage unit ex111 included in the information processing device ex101 stores the video data transmitted from the N cameras ex102.

  The analysis unit ex112 detects a blind spot from the video data stored in the storage unit ex111, and transmits an instruction signal indicating an instruction to the mobile camera to prevent the occurrence of the blind spot to the mobile camera. The moving camera moves according to the instruction signal and continues shooting.

  The analysis unit ex112 performs blind spot detection using, for example, SfM (Structure from Motion). SfM is a technique for restoring the three-dimensional shape of a subject from a plurality of videos photographed from different positions, and is widely known as a shape restoring technique for simultaneously estimating the subject shape and the camera position. For example, the analysis unit ex112 uses SfM to restore the three-dimensional shape in the facility or stadium from the video data stored in the storage unit ex111, and detects an area that cannot be restored as a blind spot.

  When the position and the shooting direction of the camera ex102 are fixed, and the information on the position and the shooting direction is known, the analysis unit ex112 may perform SfM using the known information. When the position and the shooting direction of the mobile camera can be obtained by the GPS and the angle sensor provided in the mobile camera, the mobile camera transmits information on the position and the shooting direction of the mobile camera to the analysis unit ex112, and the analysis unit ex112 The ex112 may perform SfM using the transmitted information on the position and the shooting direction.

  The method of detecting a blind spot is not limited to the method using SfM described above. For example, the analysis unit ex112 may use the information of a depth sensor such as a laser range finder to grasp the spatial distance of the object to be photographed. In addition, the analysis unit ex112 detects information such as a camera position, a shooting direction, and a zoom magnification based on whether a preset marker or a specific object in the space is included in the image or, if the marker or the specific object is included, the size or the like. Is also good. As described above, the analysis unit ex112 detects the blind spot using any method that can detect the imaging region of each camera. In addition, the analysis unit ex112 acquires information such as a positional relationship between a plurality of imaging targets from video data or a proximity distance sensor or the like, and specifies an area where a blind spot is likely to occur based on the acquired positional relationship. May be.

  Here, the blind spot includes not only a portion where an image does not exist in a region to be photographed, but also a portion having poor image quality as compared with other portions, a portion where a predetermined image quality is not obtained, and the like. The part to be detected may be appropriately set according to the configuration or purpose of the system. For example, the required image quality may be set high for a specific subject in the space where the image is taken. Conversely, for a specific area in the shooting space, the required image quality may be set low, or the setting may be made so that a blind spot is not determined even when a video is not shot.

  The image quality described above includes various information related to the video such as the area (for example, the number of pixels) occupied by the subject to be captured in the video or whether the subject to be captured is in focus. It may be determined whether or not the blind spot is based on the information or the combination thereof.

  In the above description, the detection of an area that is actually a blind spot has been described. However, the area that needs to be detected in order to prevent the occurrence of a blind spot is not limited to the area that is actually a blind spot. For example, when a plurality of imaging targets exist and at least a part thereof is moving, a new blind spot may be generated when another imaging target enters between a certain imaging target and a camera. On the other hand, the analysis unit ex112 detects movement of a plurality of shooting targets from, for example, shot video data and the like, and newly generates a blind spot based on the detected movements of the plurality of shooting targets and position information of the camera ex102. A region which may become a possibility may be estimated. In this case, the video information processing apparatus ex101 may transmit an instruction signal to the moving camera to capture an image of a region that may be a blind spot, and may prevent the occurrence of a blind spot.

  When there are a plurality of mobile cameras, the video information processing apparatus ex101 needs to select a mobile camera that transmits an instruction signal in order to photograph a blind spot or an area that may be a blind spot. If there are a plurality of mobile cameras and a plurality of blind spots or areas that may be blind spots, the video information processing apparatus ex101 determines which blind spots or blind spots are available for each of the plurality of mobile cameras. It is necessary to decide whether to shoot. For example, the video information processing apparatus ex101 selects a moving camera closest to a blind spot or a blind spot area based on an area that may be a blind spot or a blind spot and a position of an area where each moving camera is capturing an image. I do. Further, the video information processing apparatus ex101 determines, for each mobile camera, whether or not a new blind spot occurs when the video data currently being captured by the mobile camera cannot be obtained. A moving camera that is determined not to generate a blind spot even if it is not obtained may be selected.

  With the above configuration, the video information processing apparatus ex101 detects a blind spot and transmits an instruction signal to the mobile camera so as to prevent the blind spot, thereby preventing occurrence of a blind spot.

(Modification 1)
In the above description, an example in which the instruction signal for instructing the mobile camera to move has been described, but the instruction signal may be a signal for instructing the user of the user camera to move. For example, the user camera displays an instruction image that instructs the user to change the camera direction based on the instruction signal. In addition, the user camera may display an instruction image indicating a movement route on a map as an instruction of the user's movement. In addition, the user camera may display detailed shooting instructions such as a shooting direction, an angle, an angle of view, image quality, and movement of a shooting area in order to improve the quality of an obtained image. If control is possible on the ex101 side, the video information processing apparatus ex101 may automatically control the feature amount of the camera ex102 regarding such shooting.

  Here, the user camera is, for example, a smartphone, a tablet terminal, a wearable terminal, or an HMD (Head Mounted Display) possessed by a spectator in the stadium or a security guard in the facility.

  Further, the display terminal that displays the instruction image does not need to be the same as the user camera that shoots the video data. For example, the user camera may transmit an instruction signal or an instruction image to a display terminal previously associated with the user camera, and the display terminal may display the instruction image. Further, information on the display terminal corresponding to the user camera may be registered in the video information processing apparatus ex101 in advance. In this case, the video information processing apparatus ex101 may cause the display terminal to display the instruction image by directly transmitting the instruction signal to the display terminal corresponding to the user camera.

(Modification 2)
The analysis unit ex112 generates a free viewpoint video (three-dimensional reconstruction data) by restoring a three-dimensional shape in a facility or a stadium from video data stored in the storage unit ex111 using, for example, SfM. Good. This free viewpoint video is stored in the storage unit ex111. The video information processing device ex101 reads video data corresponding to the view information (and / or viewpoint information) transmitted from the video receiving device ex103 from the storage unit ex111, and transmits the video data to the video receiving device ex103. Note that the video receiving device ex103 may be one of the plurality of cameras 111.

(Modification 3)
The video information processing apparatus ex101 may detect a shooting prohibited area. In this case, the analysis unit ex112 analyzes the captured image, and transmits an image capturing inhibition signal to the mobile camera when the mobile camera is capturing an image in the image capturing prohibited area. The mobile camera stops shooting while receiving the shooting prohibition signal.

  The analysis unit ex112 performs matching between the captured image and the three-dimensional virtual space restored using SfM, for example, to determine whether a mobile camera set in advance in the space is capturing an imaging prohibited area. judge. Alternatively, the analysis unit ex112 determines whether the moving camera is capturing an image-capturing prohibited area using a marker or a characteristic object arranged in the space as a trigger. The shooting prohibited area is, for example, a toilet in a facility or a stadium.

  Further, when the user camera is capturing an image in the shooting prohibited area, the user camera may display a message on a display or the like connected wirelessly or via a wire, or output sound or voice from a speaker or earphone. Then, the user may be notified that the current location is a shooting prohibited location.

  For example, the message indicating that the direction in which the camera is currently facing is prohibited is displayed as the message. Alternatively, the shooting prohibited area and the current shooting area are shown on the displayed map. The restart of shooting is automatically performed, for example, when the shooting prohibition signal is no longer output. Alternatively, the shooting may be restarted when the shooting prohibition signal is not output and the user performs an operation to restart the shooting. Further, when the stop and restart of the shooting have occurred a plurality of times in a short period, the calibration may be performed again. Alternatively, a notification for confirming the current position or prompting the user to move may be given.

  Further, in the case of a special business such as police, a passcode or fingerprint authentication for turning off such a function for recording may be used. Further, even in such a case, if the image of the shooting prohibited area is displayed or stored outside, image processing such as mosaic may be automatically performed.

  With the above configuration, the video information processing apparatus ex101 determines that shooting is prohibited, and notifies the user to stop shooting, thereby setting a certain area to be prohibited.

(Modification 4)
In order to construct a three-dimensional virtual space from videos, it is necessary to collect videos from a plurality of viewpoints. Therefore, the video information processing system ex100 sets an incentive for the user who has transferred the captured video. For example, the video information processing apparatus ex101 distributes a video to a user who has transmitted a video for free or at a discounted rate, or has a monetary value that can be used in an online or offline store or a game, or a game. Give non-monetary value points such as social status in virtual space. In addition, the video information processing apparatus ex101 gives a particularly high point to a user who has transferred a captured video of a valuable field of view (and / or viewpoint) such as a large number of requests.

(Modification 5)
The video information processing device ex101 may transmit the additional information to the user camera based on the analysis result of the analysis unit ex112. In this case, the user camera superimposes the additional information on the captured video and displays it on the screen. The additional information is, for example, when a game in a stadium is being photographed, is information on a player such as a player name or height, and the name or face photograph of the player is associated with each player in the video. Is displayed. Note that the video information processing apparatus ex101 may extract the additional information by searching via the Internet based on a part or the entire area of the video data. The camera ex102 receives such additional information by short-range wireless communication such as Bluetooth (registered trademark) or visible light communication from lighting such as a stadium, and maps the received additional information to video data. Is also good. Also, the camera ex102 is a table held in a storage unit connected to the camera ex102 by wire or wireless, such as a table indicating the correspondence between information obtained by visible light communication technology and additional information. May be performed based on the predetermined rule, or may be performed using the result of the most probable combination by Internet search.

  Further, in the monitoring system, for example, information of a person of interest is superimposed on a user camera of a security guard in the facility, so that the accuracy of the monitoring system can be improved.

(Modification 5)
The analysis unit ex112 may determine which area in the facility or stadium the user camera is shooting by matching the free viewpoint video with the video shot by the user camera. Note that the method of determining a shooting region is not limited to this, and the various methods of determining a shooting region described in the above embodiments or other methods of determining a shooting region may be used.

  The video information processing apparatus ex101 transmits a past video to the user camera based on the analysis result of the analysis unit ex112. The user camera superimposes the past image on the photographed image or replaces the photographed image with the past image and displays the image on the screen.

  For example, during the half time, the highlight scene of the first half is displayed as the past video. Thereby, the user can enjoy the highlight scene of the first half as a video in the direction in which he or she is watching during the half time. The past video is not limited to the highlight scene of the first half, but may be a highlight scene of a past game played at the stadium. Further, the timing at which the video information processing apparatus ex101 distributes the past video is not limited to the half-time, and may be, for example, after the end of the match or during the match. Particularly during a match, the video information processing apparatus ex101 may distribute important scenes that the user has overlooked based on the analysis result of the analysis unit ex112. Further, the video information processing apparatus ex101 may distribute the past video only when requested by the user, or may distribute a distribution permission message before distributing the past video.

(Modification 6)
The video information processing apparatus ex101 may transmit the advertisement information to the user camera based on the analysis result of the analysis unit ex112. The user camera superimposes the advertisement information on the captured video and displays it on the screen.

  The advertisement information may be distributed, for example, during the half time or immediately before the distribution of the past video after the match, as shown in Modification Example 5. Thus, the distributor can obtain the advertisement fee from the advertiser, and can provide the user with a cheap or free video distribution service. In addition, the video information processing apparatus ex101 may distribute the advertisement distribution permission message immediately before the distribution of the advertisement information, may provide a service free of charge only when the user has viewed the advertisement, and may view the advertisement. If not, the service may be provided at a lower cost.

  In addition, if the user clicks "order now" or the like according to the advertisement, the staff or the automated delivery system at the venue, which knows the user's location based on the system or some kind of location information, takes the ordered drink to the seat. Will deliver. The decision may be handed to the staff or may be made based on credit card information set in advance in an application or the like of the mobile terminal. Further, the advertisement may include a link to an e-commerce site, and may be in a state where online shopping such as ordinary home delivery is possible.

(Modification 7)
The video receiving device ex103 may be one of the cameras ex102 (user camera). In this case, the analysis unit ex112 determines which area in the facility or stadium the user camera is shooting by matching the free viewpoint video with the video shot by the user camera. Note that the method of determining the shooting area is not limited to this.

  For example, when the user performs a swipe operation in the direction of the arrow displayed on the screen, the user camera generates viewpoint information indicating that the viewpoint is moved in that direction. The video information processing apparatus ex101 reads, from the storage unit ex111, video data obtained by capturing an area moved by the viewpoint information from the capturing area of the user camera determined by the analysis unit ex112, and transmits the video data to the user camera. Start. Then, the user camera displays not the captured video but the video distributed from the video information processing apparatus ex101.

  As described above, the user in the facility or the stadium can view a video from a desired viewpoint by a simple operation such as swiping the screen. For example, a spectator watching a game on the third base of the baseball stadium can view an image from the viewpoint of the first base. In the monitoring system, a security guard in the facility uses a simple operation such as a swipe of a screen to view a video or the like that the user wants to check or an image to be watched as an interruption from the center while changing the viewpoint adaptively. Therefore, the accuracy of the monitoring system can be improved.

  In addition, video distribution to a user in a facility or a stadium is also effective, for example, when there is an obstacle between the user camera and the shooting target and there is an invisible area. In this case, the user camera may switch and display the video of a part of the shooting area of the user camera including the obstacle from the captured video to the distribution video from the video information processing apparatus ex101, The entire screen may be switched from the captured video to the distribution video and displayed. Further, the user camera may display the video in which the viewing target is seen through the obstacle by combining the captured video and the distribution video. According to this configuration, the video distributed from the video information processing apparatus ex101 can be viewed even when the shooting target cannot be seen from the position of the user due to the influence of the obstacle, so that the influence of the obstacle can be reduced. it can.

  In the case where the distribution video is displayed as a video in an area invisible due to an obstacle, display switching control different from the display switching control according to the input processing by the user, such as the above-described screen swipe, may be performed. . For example, when it is determined that an obstacle is included in the shooting area based on the information on the movement and the shooting direction of the user camera and the position information of the obstacle obtained in advance, the display from the shot video to the distribution video is performed. The switching may be performed automatically. In addition, when it is determined from the analysis of the captured video data that an obstacle that is not a capturing target is reflected, the display may be automatically switched from the captured video to the distribution video. In addition, when the area of an obstacle (for example, the number of pixels) included in the captured image exceeds a predetermined threshold, or when the ratio of the area of the obstacle to the area of the imaging target exceeds a predetermined ratio, Switching of the display to the distribution video may be automatically performed.

  In addition, the switching of the display from the captured video to the distribution video and the switching of the display from the distribution video to the captured video may be performed according to the input processing of the user.

(Modification 8)
The speed at which the video data is transferred to the video information processing apparatus ex101 may be instructed based on the importance of the video data captured by each camera ex102.

  In this case, the analysis unit ex112 determines the video data stored in the storage unit ex111 or the importance of the camera ex102 that has captured the video data. The determination of the importance here is performed based on information such as the number of people or the number of moving objects included in the video, the image quality of the video data, or a combination thereof.

  Further, the determination of the importance of the video data may be based on the position of the camera ex102 where the video data is captured or the area where the video data is captured. For example, when there are a plurality of other cameras ex102 being photographed near the target camera ex102, the importance of the video data photographed by the target camera ex102 is reduced. Also, when there is a plurality of other cameras ex102 capturing the same area even if the position of the target camera ex102 is far from the other camera ex102, the importance of the video data captured by the target camera ex102 is determined. make low. The determination of the importance of the video data may be performed based on the number of requests in the video distribution service. The method of determining the importance is not limited to those described above and combinations thereof, and may be any method according to the configuration or purpose of the monitoring system or the video distribution system.

  Further, the determination of the importance may not be based on the captured video data. For example, the importance of the camera ex102 that transmits video data to a terminal other than the video information processing apparatus ex101 may be set high. Conversely, the importance of the camera ex102 that transmits video data to a terminal other than the video information processing apparatus ex101 may be set to be low. Thus, for example, when a plurality of services that require transmission of video data share a communication band, the degree of freedom in controlling the communication band according to the purpose or characteristic of each service increases. This can prevent the quality of each service from deteriorating due to the inability to obtain necessary video data.

  The analysis unit ex112 may determine the importance of the video data using the free viewpoint video and the video captured by the camera ex102.

  The video information processing device ex101 transmits a communication speed instruction signal to the camera ex102 based on the determination result of the importance performed by the analysis unit ex112. The video information processing apparatus ex101 instructs, for example, a high communication speed to a camera ex102 that is capturing a video with high importance. Further, the video information processing apparatus ex101 may transmit not only the speed control but also a signal instructing a method of transmitting important information a plurality of times in order to reduce disadvantages due to missing information. This allows efficient communication within the facility or the entire stadium. Communication between the camera ex102 and the video information processing device ex101 may be wired communication or wireless communication. Further, the video information processing apparatus ex101 may control only one of the wired communication and the wireless communication.

  The camera ex102 transmits the captured video data to the video information processing device ex101 at the communication speed according to the communication speed instruction signal. If retransmission of the camera ex102 fails a predetermined number of times, retransmission of the captured video data may be stopped, and transfer of the next captured video data may be started. Thus, communication within the facility or the entire stadium can be efficiently performed, and the processing speed in the analysis unit ex112 can be increased.

  If the communication speed assigned to each of the cameras ex102 is not a band sufficient to transfer the captured video data, the camera ex102 converts the captured video data to a bit rate video data capable of being transmitted at the allocated communication speed. And the converted video data may be transmitted, or the transfer of the video data may be stopped.

  Also, when video data is used to prevent the occurrence of blind spots as described above, only a part of the shooting area included in the shot video data may be necessary to fill the blind spot. There is. In this case, the camera ex102 generates extracted video data by extracting at least only a region required to prevent the occurrence of blind spots from the video data, and generates the extracted video data by using the video information processing apparatus. It may be transmitted to ex101. According to this configuration, the occurrence of blind spots can be suppressed with a smaller communication band.

  Also, for example, when superimposed display of additional information or video distribution is performed, the camera ex102 needs to transmit the position information and the shooting direction information of the camera ex102 to the video information processing apparatus ex101. In this case, the camera ex102 to which only a band that is not sufficient to transfer the video data may be allocated may transmit only the position information and the shooting direction information detected by the camera ex102. When estimating the position information and the shooting direction information of the camera ex102 in the video information processing apparatus ex101, the camera ex102 converts the shot video data into a resolution necessary for estimating the position information and the shooting direction information. Alternatively, the converted video data may be transmitted to the video information processing device ex101. According to this configuration, a service of superimposed display of additional information or a video distribution service can be provided even for the camera ex102 to which only a small communication band is allocated. Further, since the video information processing apparatus ex101 can acquire the information of the shooting area from more cameras ex102, even when the information of the shooting area is used for the purpose of, for example, detecting the area of interest, It is valid.

  The switching of the video data transfer process according to the allocated communication band described above may be performed by the camera ex102 based on the notified communication band, or the video information processing apparatus ex101 may control the operation of each camera ex102. The camera ex102 may be notified of a control signal indicating the determined operation and the determined operation. Thus, the processing can be appropriately shared in accordance with the amount of calculation required to determine the switching of the operation, the processing capability of the camera ex102, the required communication band, and the like.

(Modification 9)
The analysis unit ex112 may determine the importance of the video data based on the visual field information (and / or the viewpoint information) transmitted from the video receiving device ex103. For example, the analysis unit ex112 sets the importance of the captured video data including many regions indicated by the visual field information (and / or the viewpoint information) to be high. The analysis unit ex112 may determine the importance of the video data in consideration of the number of people included in the video or the number of moving objects. The method of determining the importance is not limited to this.

  The communication control method described in the present embodiment does not necessarily need to be used in a system that reconstructs a three-dimensional shape from a plurality of video data. For example, in the environment where a plurality of cameras ex102 are present, if the video data is transmitted by wire communication and / or wireless communication selectively or with a different transmission speed, the communication control method described in the present embodiment It is valid.

(Modification 10)
In the video distribution system, the video information processing apparatus ex101 may transmit an overview video showing the entire shooting scene to the video receiving apparatus ex103.

  Specifically, when the video information processing apparatus ex101 receives the distribution request transmitted from the video receiving apparatus ex103, the video information processing apparatus ex101 reads the overview video of the entire facility or the stadium from the storage unit ex111, and reads the external appearance video into the video receiving apparatus. ex103. The overview video may have a long update interval (may have a low frame rate) and may have low image quality. The viewer touches a desired part in the overview video displayed on the screen of the video receiving device ex103. Thereby, the video receiving device ex103 transmits the visual field information (and / or the viewpoint information) corresponding to the touched portion to the video information processing device ex101.

  The video information processing device ex101 reads video data according to the view information (and / or viewpoint information) from the storage unit ex111, and transmits the video data to the video receiving device ex103.

  In addition, the analysis unit ex112 generates a free viewpoint image by preferentially performing a three-dimensional reconstruction (three-dimensional reconstruction) on an area indicated by the view information (and / or viewpoint information). The analysis unit ex112 restores the three-dimensional shape of the entire inside of the facility or the stadium with an accuracy that gives an overview. Thereby, the video information processing apparatus ex101 can efficiently restore the three-dimensional shape. As a result, it is possible to realize a high frame rate and a high image quality of a free viewpoint video in an area desired by the viewer.

(Modification 11)
Note that the video information processing apparatus ex101 may store in advance, for example, three-dimensional shape restoration data of a facility or a stadium generated in advance from a design drawing or the like as a preliminary video. The preliminary image is not limited to this, and may be virtual space data in which unevenness of a space obtained from a depth sensor and a picture derived from an image or video data in the past or at the time of calibration are mapped for each object.

  For example, when soccer is played in a stadium, the analysis unit ex112 performs restoration of a three-dimensional shape limited to only the player and the ball, and combines the obtained restored data with a preliminary image to obtain a free viewpoint image. May be generated. Alternatively, the analysis unit ex112 may preferentially restore the three-dimensional shape of the player and the ball. Thereby, the video information processing apparatus ex101 can efficiently restore the three-dimensional shape. As a result, it is possible to realize a higher frame rate and a higher image quality of the free viewpoint video relating to the player and the ball that the viewer pays attention to. In the monitoring system, the analysis unit ex112 may restore the three-dimensional shape by limiting only to the person and the moving object or giving priority to them.

(Modification 12)
The time of each device may be calibrated at the start of shooting based on the reference time of the server or the like. The analysis unit ex112 uses a plurality of pieces of video data shot at times belonging to a preset time range in accordance with the accuracy of the time setting, among a plurality of pieces of video data shot by the plurality of cameras ex102. , To restore the three-dimensional shape. For detecting the time, for example, the time at which the captured video data is stored in the storage unit ex111 is used. The method of detecting the time is not limited to this. Accordingly, the video information processing apparatus ex101 can efficiently restore the three-dimensional shape, and thus can achieve a higher frame rate and a higher image quality of the free viewpoint video.

  Alternatively, the analysis unit ex112 may restore the three-dimensional shape by using only the high-quality data or preferentially using the high-quality data among the plurality of video data stored in the storage unit ex111. .

(Modification 13)
The analysis unit ex112 may restore the three-dimensional shape using the camera attribute information. In this case, the camera ex102 transmits the captured video data and the camera attribute information to the video information processing device ex101. The camera attribute information is, for example, a shooting position, a shooting angle, a shooting time, a zoom magnification, and the like.

  Accordingly, the video information processing apparatus ex101 can efficiently restore the three-dimensional shape, and thus can achieve a higher frame rate and a higher image quality of the free viewpoint video.

  Specifically, the camera ex102 defines three-dimensional coordinates in a facility or a stadium, and information on which coordinates the camera ex102 took, from what angle, how much zoom, and at what time along with the video. It is transmitted to the video information processing apparatus ex101 as camera attribute information. When the camera ex102 is activated, a clock on the communication network in the facility or the stadium is synchronized with a clock in the camera, and time information is generated.

  In addition, by pointing the camera ex102 to a specific point in the facility or stadium at the time of starting the camera ex102 or at an arbitrary timing, the position and angle information of the camera ex102 is obtained. FIG. 21 is a diagram illustrating an example of a notification displayed on the screen of the camera ex102 when the camera ex102 is activated. According to this notification, when the user touches the display of the camera ex102 with “+” displayed at the center of the screen and “+” at the center of the soccer ball in the advertisement on the north side of the stadium, and the camera ex102 is The vector information up to the advertisement is acquired and the reference of the camera position and angle is specified. Thereafter, the camera coordinates and the angle at that time are specified from the motion information of the camera ex102. Of course, the present invention is not limited to this display, and a display for indicating the coordinates, the angle, the moving speed of the shooting area, or the like during the shooting period using an arrow or the like may be used.

  The coordinates of the camera ex102 may be specified using radio waves of GPS, WiFi (registered trademark), 3G, LTE (Long Term Evolution), and 5G (wireless LAN), or a beacon (Bluetooth (registered trademark)). , Ultrasonic waves, etc., may be used. Further, information indicating which base station in the facility or stadium the captured video data has reached may be used.

(Modification 14)
The system may be provided as an application operating on a mobile terminal such as a smartphone.

  Accounts such as various SNSs may be used to log in to the system. Note that an account dedicated to the application or a guest account with limited functions may be used. By using the account in this way, a favorite video or a favorite account can be evaluated. Also, by preferentially allocating a band to video data similar to the video data being shot or viewed, video data having a viewpoint similar to the viewpoint of the video data being shot or viewed, the resolution of these video data can be improved. Can be increased. As a result, the three-dimensional shape can be more accurately restored from these viewpoints.

  In addition, the user can select a favorite image and video in the application and follow the other party to view the selected image in preference to other users, or perform text chat or the like on the condition that the other party approves. Can be connected. Thus, a new community can be created.

  By connecting the users in the community as described above, the photographing itself, sharing of the photographed image, and the like are activated, and it is possible to promote the restoration of the three-dimensional shape with higher accuracy.

  Also, according to the setting of the connection in the community, the user can edit an image or a video taken by another person, or create a new image or video by collaging an image of the other person with his / her own image. This allows sharing of a new video work, such as sharing a new image or video only with people in the community. Also, by inserting a CG character in this editing, the video work can be used in an augmented reality (Augmented Reality) game or the like.

  Further, according to the system, three-dimensional model data can be sequentially output, so that a 3D printer or the like of a facility can output a three-dimensional object based on the three-dimensional model data in a characteristic scene such as a goal scene. . Thus, after the game, it is possible to sell an object based on the scene during the game as a souvenir such as a key ring, or to distribute it to participating users. Of course, it is also possible to print an image from the best viewpoint as a normal photograph.

(Modification 15)
Using the above system, for example, a rough state of the entire area can be managed by a center connected to the system from images of a police car-mounted camera and a police officer's wearable camera.

  At the time of general patrol, transmission and reception of still images are performed, for example, every few minutes. In addition, the center identifies areas with a high probability of crime occurrence based on a crime map based on the results of analysis using past crime data, etc., or areas related to the crime occurrence probability identified in this way. Holds data. In the identified area where the crime probability is high, the frequency of transmitting and receiving images may be increased, or the images may be changed to moving images. In addition, when an incident occurs, moving images or three-dimensional reconstructed data using SfM or the like may be used. In addition, since the center or each terminal simultaneously corrects an image or a virtual space using information from another sensor such as a depth sensor or a thermosensor, a police officer can more accurately grasp the situation.

  In addition, the center can feed back information of the object to a plurality of terminals by using the three-dimensional reconstruction data. This allows an individual with each terminal to track the object.

  Also, recently, for the purpose of investigating a building or environment, or photographing with a sense of reality such as sports, photographing from the air using a flightable device such as a quadcopter or a drone is performed. In such shooting by the autonomous mobile device, blurring of the image is likely to cause a problem, but SfM can perform three-dimensionalization while correcting the blurring based on the position and inclination. As a result, it is possible to improve the image quality and the space restoration accuracy.

  In addition, installation of an on-board camera for photographing the outside of the vehicle is required in some countries. Even in such a vehicle-mounted camera, by using three-dimensional data modeled from a plurality of images, it is possible to more accurately grasp the weather in the direction of the destination, the state of the road surface, the degree of congestion, and the like.

(Embodiment 3)
By recording a program for realizing the configuration of the image processing method described in each of the above embodiments on a storage medium, it is possible to easily execute the processing described in each of the above embodiments on an independent computer system. It becomes. The storage medium may be a magnetic disk, an optical disk, a magneto-optical disk, an IC card, a semiconductor memory, or any other medium capable of recording a program.

  Further, an application example of the image processing method described in each of the above embodiments and a system using the same will be described. The system includes an apparatus using an image processing method. Other configurations in the system can be appropriately changed as necessary.

  FIG. 22 is a diagram illustrating an overall configuration of a content supply system ex200 that realizes a content distribution service. A communication service providing area is divided into desired sizes, and base stations ex206, ex207, ex208, ex209, and ex210, which are fixed wireless stations, are installed in each cell.

  The content supply system ex200 includes a computer ex211, a PDA (Personal Digital Assistant) ex212, a camera ex213, a smartphone ex214, a game machine ex215, and the like, via the Internet ex201, the Internet service provider ex202 and the communication network ex204, and the base stations ex206 to ex210 via the ex210. Devices are connected.

  However, the content supply system ex200 is not limited to the configuration shown in FIG. 22, and may be connected by combining any of the elements. In addition, each device may be directly connected to a communication network ex204 such as a telephone line, a cable television, or an optical communication without passing through base stations ex206 to ex210 which are fixed wireless stations. Further, the devices may be directly connected to each other via short-range wireless communication or the like.

  The camera ex213 is a device capable of shooting moving images such as a digital video camera, and the camera ex216 is a device capable of shooting still images and moving images such as a digital camera. In addition, the smartphone ex214 has a GSM (registered trademark) (Global System for Mobile Communications) method, a CDMA (Code Division Multiple Access) method, a W-CDMA (Wideband-Code Division, or a Multimedia Multi-Let- mation Multi-Template Multi-Let- sion Multi-Let- , HSPA (High Speed Packet Access), or a smartphone corresponding to a communication method using a high-frequency band, or a PHS (Personal Handyphone System), and the like.

  In the content supply system ex200, the camera ex213 and the like are connected to the streaming server ex203 via the base station ex209 and the communication network ex204, thereby enabling live distribution and the like. In the live distribution, the encoding process is performed on the content (for example, a live music video) shot by the user using the camera ex213, and the encoded content is transmitted to the streaming server ex203. On the other hand, the streaming server ex203 stream-distributes the transmitted content data to the requested client. Examples of the client include a computer ex211, a PDA ex212, a camera ex213, a smartphone ex214, a game machine ex215, and the like, which are capable of decoding the encoded data. Each device that has received the distributed data decodes and reproduces the received data.

  The encoding process of the photographed data may be performed by the camera ex213, the streaming server ex203 that performs the data transmission process, or may be performed by sharing with each other. Similarly, the decoding processing of the distributed data may be performed by the client, the streaming server ex203, or may be performed by sharing with each other. In addition to the camera ex213, still image data and / or moving image data captured by the camera ex216 may be transmitted to the streaming server ex203 via the computer ex211. The encoding process in this case may be performed by any of the camera ex216, the computer ex211 and the streaming server ex203, or may be performed by sharing with each other. Further, regarding the display of the decoded image, a plurality of devices connected to the system may display the same image in conjunction with each other, or the entire image may be displayed on a device having a large display unit, and the image may be displayed on a smartphone ex214 or the like. May be enlarged and displayed.

  These encoding / decoding processes are generally performed by the computer ex211 or the LSI ex500 included in each device. The LSI ex500 may be a single chip or a configuration including a plurality of chips. It should be noted that the video encoding / decoding software is incorporated in any recording medium (CD-ROM, flexible disk, hard disk, etc.) readable by the computer ex211 or the like, and the encoding / decoding processing is performed using the software. May be. Further, when the smartphone ex214 has a camera, moving image data acquired by the camera may be transmitted. The moving image data at this time is data that has been encoded by the LSI ex500 included in the smartphone ex214.

  The streaming server ex203 may be a plurality of servers or a plurality of computers, and may process, record, or distribute data in a distributed manner.

  As described above, in the content providing system ex200, the client can receive and reproduce the encoded data. As described above, in the content providing system ex200, the client can receive, decode, and reproduce the information transmitted by the user in real time, and a user without special rights or facilities can realize personal broadcasting.

  Note that each embodiment may be applied not only to the example of the content supply system ex200 but also to a digital broadcasting system ex300 as shown in FIG. Specifically, at the broadcasting station ex301, multiplexed data in which music data and the like are multiplexed with video data is transmitted to a communication or a satellite ex302 via radio waves. This video data is data encoded by the moving picture encoding method described in each of the above embodiments. The broadcast satellite ex302 receiving this transmits a broadcast radio wave, and this radio wave is received by a home antenna ex304 capable of receiving satellite broadcasts. The received multiplexed data is decoded and reproduced by a device such as a television (receiver) ex400 or a set-top box (STB) ex317.

  Also, multiplexed data recorded on a recording medium ex315 such as a DVD or a BD or a memory ex316 such as an SD is read and decoded, or a video signal is encoded on the recording medium ex315 or the memory ex316, and in some cases, a music signal and The moving picture decoding apparatus or the moving picture coding apparatus described in each of the above embodiments can also be mounted on the reader / recorder ex318 that multiplexes and writes. In this case, the reproduced video signal is displayed on the monitor ex319, and the video signal can be reproduced in another device or system by the recording medium ex315 or the memory ex316 on which the multiplexed data is recorded. Further, a moving picture decoding apparatus may be mounted in a set-top box ex317 connected to a cable ex303 for cable television or an antenna ex304 for satellite / terrestrial broadcasting, and this may be displayed on a monitor ex319 of the television. At this time, the video decoding device may be incorporated in the television instead of the set-top box.

  FIG. 24 is a diagram illustrating the smartphone ex214. FIG. 25 is a diagram illustrating a configuration example of the smartphone ex214. The smartphone ex214 includes an antenna ex450 for transmitting and receiving radio waves to and from the base station ex210, a camera unit ex465 capable of taking a video and a still image, a video captured by the camera unit ex465, a video received by the antenna ex450, and the like. A display unit ex458 such as a liquid crystal display for displaying the decrypted data is provided. The smartphone ex214 further includes an operation unit ex466 such as a touch panel, a sound output unit ex457 such as a speaker for outputting sound, a sound input unit ex456 such as a microphone for inputting sound, captured images, and still images. 23, a memory unit ex467 capable of storing encoded data or decoded data such as a recorded voice or a received video, still image, mail, or the like, or a memory ex316 illustrated in FIG. And a slot unit ex464 as an interface unit with the SIMex468 for authenticating access to various data including a network.

  The smartphone ex214 has a power supply circuit unit ex461, an operation input control unit ex462, a video signal processing unit ex455, a camera interface unit ex463, an LCD ( A liquid crystal display control unit ex459, a modulation / demodulation unit ex452, a multiplexing / demultiplexing unit ex453, an audio signal processing unit ex454, a slot unit ex464, and a memory unit ex467 are connected to each other via a bus ex470.

  When the end of the call and the power key are turned on by the user's operation, the power supply circuit ex461 starts the smartphone ex214 in an operable state by supplying power to each unit from the battery pack.

  The smartphone ex214 converts a voice signal collected by the voice input unit ex456 into a digital voice signal in the voice signal processing unit ex454 in the voice communication mode based on the control of the main control unit ex460 having a CPU, a ROM, a RAM, and the like. This is subjected to spread spectrum processing in a modulation / demodulation unit ex452, digital-analog conversion processing and frequency conversion processing in a transmission / reception unit ex451, and then transmitted via an antenna ex450. In addition, the smartphone ex214 amplifies the received data received via the antenna ex450 in the voice call mode, performs frequency conversion processing and analog-to-digital conversion processing, performs a spectrum despreading process in the modulation / demodulation unit ex452, and performs a voice signal processing unit ex454. After converting into an analog audio signal, this is output from the audio output unit ex457.

  Further, when an e-mail is transmitted in the data communication mode, text data of the e-mail input by operating the operation unit ex466 or the like of the main body is sent to the main control unit ex460 via the operation input control unit ex462. The main control unit ex460 performs a spread spectrum process on the text data in the modulation / demodulation unit ex452, performs a digital-to-analog conversion process and a frequency conversion process on the transmission / reception unit ex451, and transmits the text data to the base station ex210 via the antenna ex450. . When an e-mail is received, the process is substantially the reverse of the received data and is output to the display unit ex458.

  When transmitting a video, a still image, or video and audio in the data communication mode, the video signal processing unit ex455 compresses the video signal supplied from the camera unit ex465 by the moving picture coding method described in each of the embodiments. The encoded video data is transmitted to the multiplexing / demultiplexing unit ex453. The audio signal processing unit ex454 encodes an audio signal collected by the audio input unit ex456 while capturing a video, a still image, or the like with the camera unit ex465, and transmits the encoded audio data to the multiplexing / demultiplexing unit ex453. I do.

  The multiplexing / demultiplexing unit ex453 multiplexes the encoded video data supplied from the video signal processing unit ex455 and the encoded audio data supplied from the audio signal processing unit ex454 in a predetermined format, and obtains the result. The multiplexed data is subjected to spread spectrum processing by a modulation / demodulation unit (modulation / demodulation circuit unit) ex452, subjected to digital / analog conversion processing and frequency conversion processing by a transmission / reception unit ex451, and then transmitted via an antenna ex450.

  When receiving data of a moving image file linked to a homepage or the like in the data communication mode, or receiving an e-mail attached with video and / or audio, the multiplexed data received via the antenna ex450 is decoded. In order to perform this, the multiplexing / demultiplexing unit ex453 separates the multiplexed data into a bit stream of video data and a bit stream of audio data, and processes the video data encoded via the synchronization bus ex470 into video signal processing. While supplying the audio data to the unit ex455, the encoded audio data is supplied to the audio signal processing unit ex454. The video signal processing unit ex455 decodes the video signal by decoding using the video decoding method corresponding to the video encoding method described in each of the above embodiments, and displays the display unit ex458 via the LCD control unit ex459. Thus, for example, video and still images included in the moving image file linked to the homepage are displayed. The audio signal processing unit ex454 decodes the audio signal, and the audio is output from the audio output unit ex457.

  Similarly to the television ex400, the terminal such as the smartphone ex214 is a transmission / reception type terminal having both an encoder and a decoder, as well as a transmission terminal including only an encoder and a reception terminal including only a decoder. There are several possible implementation formats. Furthermore, in the digital broadcasting system ex300, it has been described that multiplexed data in which music data and the like are multiplexed with video data is received and transmitted, but data in which character data and the like related to video are multiplexed in addition to audio data. Or video data itself instead of multiplexed data.

  Further, the present invention is not limited to the above embodiment, and various changes or modifications can be made without departing from the scope of the present invention.

  INDUSTRIAL APPLICABILITY The present invention can be applied to a video distribution system that distributes video captured by a plurality of cameras.

Reference Signs List 100 video distribution system 101 camera 102 terminal device 103 server 104A, 104B network 105 monitor 106 user 111, 121 receiving unit 112, 122 storage unit 113, 123 image processing unit 114 output unit 115, 124 transmission unit 116, 126 control unit 300A, 300B, 300C output screen 301 viewpoint video display screen 302, 313 viewpoint selection screen 303A, 303B, 303C, 304A, 304B operation buttons 305, 306, 308 information display screen 307A, 307B, 307C operation menu 311 main screen 312A first sub Screen 312B 2nd sub screen

Claims (8)

A video distribution method by a server that distributes a first video signal based on a plurality of videos obtained by capturing the same scene from a plurality of different viewpoints to a terminal device in a plurality of continuous periods,
A first selecting step of selecting a first selected video from the plurality of videos for each of the plurality of periods based on evaluations of the plurality of videos by a plurality of users;
A first transmission step of transmitting the first video signal to the terminal device for continuously reproducing the plurality of first selection videos selected for the plurality of periods,
The video distribution method, wherein the evaluation indicates whether the plurality of videos are good or bad. The video distribution method according to claim 1, wherein the evaluation includes a bad degree for the plurality of videos. The video distribution method according to claim 1, wherein the evaluation is input as a comment from a user at the terminal device. A receiving step of receiving evaluation values input to the plurality of terminal devices, the evaluation values for the displayed plurality of subjects,
Generating a second video including a plurality of first videos connected in time series by dividing each of the plurality of first videos having the same shooting period and a plurality of viewpoints by time and connecting the plurality of first videos; ,
Transmitting the second video to the plurality of terminal devices,
The second video is generated so as to continuously display a first subject included in the plurality of subjects,
The video distribution method, wherein a first evaluation value of the first subject is higher than a second evaluation value of a second subject included in the plurality of subjects. The video distribution method according to claim 4, wherein the second video is generated such that a first display area of the first subject is larger than a second display area of the second subject. The video distribution method according to claim 4, wherein the first evaluation value is a highest evaluation value among evaluation values of the plurality of subjects. A server that delivers a first video signal based on a plurality of videos obtained by capturing the same scene from a plurality of different viewpoints in a plurality of continuous periods to a terminal device,
A selection unit that selects a first selected video from the plurality of videos for each of the plurality of periods based on evaluations of the plurality of videos by a plurality of users;
A transmission unit that transmits to the terminal device the first video signal for continuously reproducing the plurality of first selection videos selected for the plurality of periods,
The evaluation is a server that indicates the quality of the plurality of videos. A receiving unit that receives the evaluation values for the plurality of displayed subjects, the evaluation values being input in the plurality of terminal devices,
A generating unit configured to generate a second video including the plurality of first videos connected in time series by dividing each of the plurality of first videos having the same shooting period and a plurality of viewpoints by time and connecting the plurality of first videos; ,
A transmitting unit that transmits the second video to the plurality of terminal devices,
The generation unit generates the second video so as to continuously display a first subject included in the plurality of subjects,
A server in which a first evaluation value for the first subject is higher than a second evaluation value for a second subject included in the plurality of subjects.

Images