JP2010258894A - Video receiving apparatus, method of receiving video, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To compatibly satisfy immediacy and reliability regardless of kinds of protocols. <P>SOLUTION: A video receiving apparatus includes a first video distribution mode, where no frame drops exist although delay exists, and a second video distribution mode, where frame drops exist although the amount of delay is small. The video receiving apparatus includes a moving body detection means for performing frame difference and that for background difference in the first and second video distribution modes, respectively. Also, the video receiving apparatus includes a third video distribution mode having a retransmission function, small delay, and no frame drops. The video receiving apparatus includes a means of detecting the moving body also in occurrence of retransmission to issue a temporary event, and a means of performing moving body detection processing again in the completion of the retransmission to issue a regular event or a corrected event. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a video reception device, a video reception method, and a program, and more particularly to a technique suitable for use in receiving a live video.

  In recent years, live video distribution using a network camera has been put into practical use. Also, moving object detection and non-moving object detection are performed on the received video data by image processing, and event notification processing such as control of external devices and recording in conjunction with detection is performed.

  For example, Patent Document 1 discloses a video communication method that uses a communication protocol with high reliability for a partial area of a motion-detected image and uses a protocol with low reliability for other partial areas. Patent Document 2 discloses a method of changing the distribution frame rate of live video depending on the state of motion detection.

JP 2000-298661 A JP 2006-279464 A

  In the live video distribution method, different protocols are used depending on the system. For example, network cameras often use HTTP (Hypertext Transfer Protocol) as a protocol for video transmission. On the other hand, in a video conference system, RTP (Real Time Transfer Protocol) is often used as a protocol for video transmission. Here, TCP uses TCP, which is a virtual circuit communication, and data arrival is guaranteed, but a delay may occur. On the other hand, RTP uses UDP, which is a datagram type communication, and although there is little delay, the arrival of data is not guaranteed. Conventionally, the same moving object detection method is used even in the case of delivery using different protocols as described above, and as a result, there has been a problem in either the promptness or reliability of the detection result.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to make it possible to achieve both promptness and reliability regardless of the type of protocol.

  The video receiving apparatus of the present invention is a video receiving apparatus having communication means and image processing means, wherein a plurality of video receiving means corresponding to a plurality of video distribution means having different communication protocols, and the plurality of video receiving means And a plurality of image processing means corresponding to each.

  ADVANTAGE OF THE INVENTION According to this invention, the highly reliable moving body detection function which does not depend on the kind of protocol can be provided by switching the moving body detection method used according to the kind of protocol. In addition, it is possible to provide an event notification function that provides both promptness and reliability in a protocol that guarantees retransmission and low delay.

1 is a block diagram illustrating an example of the overall configuration of a system according to a first embodiment of this invention. It is a figure which shows an example of the video delivery mode in the 1st Embodiment of this invention. It is a flowchart which shows the change process of the video delivery mode of a video transmission apparatus. It is a flowchart which shows the change process of the video delivery mode of a video receiver. It is a figure which shows the collaboration between a video transmitter and a video receiver. It is a flowchart which shows an example of the process sequence of the video delivery thread | sled of a video transmission apparatus. It is a flowchart which shows an example of the event issuing process procedure of a video receiver. It is a flowchart which shows an example of the event issuing process procedure of a video receiver.

(First embodiment)
In this embodiment, a method of changing the moving body detection process on the receiving side according to a plurality of protocols in the live video distribution system will be described. Here, the transmission device is assumed to be a network camera, and the reception device is assumed to be a personal computer (PC) that executes a viewer application or a surveillance camera application. There are many systems that perform motion detection on the network camera side, but there are many existing cameras and network cameras that do not have a motion detection function. Therefore, it may be required to perform moving object detection on the PC side.

FIG. 1 is a block diagram showing an example of the overall configuration of the system of the present embodiment. The system configuration of this embodiment will be described with reference to FIG.
The system in FIG. 1 includes a video transmission device 100 and a video reception device 200 connected via a network. The video transmission device 100 has a function of capturing and distributing video, and can be realized by using, for example, a network camera. Here, imaging and distribution of video may be the same device, or may be a separated device configuration. In other words, an analog video signal can be input to a video distribution server having a video capture function. This embodiment does not depend on the mounting method of the photographing apparatus.

  The video receiving apparatus 200 includes a video receiving unit 211, a video distribution control unit 212, a video RAM 221, a display control unit 222, a moving object detection unit 241, and a moving object detection control unit 242. Further, it has an input / output I / F 243, an external storage control unit 260, an external storage device 261, a button / switch control unit 270, and a button / switch 271. Note that the video receiving apparatus 200 can be realized using a commercially available PC, but can also be implemented by an embedded device and does not depend on the OS or the device form. The display 230 is not essential in the present embodiment.

  A flow of processing in the video receiving apparatus will be described. A video reception unit 211 having a communication interface communicates with the video transmission device 100 to acquire video data. The communication protocol used for transmission of the video data is determined by the video distribution control unit 212 and transmitted to the video reception unit 211 and the video transmission device 100. The received video data is displayed on the display 230 via the video RAM 221.

  The moving object detection unit 241 performs a moving object detection process, and the moving object detection control unit 242 determines the result and issues an event notification. The event notification is a change in the moving object detection state, and the result is stored, an external device such as an alarm device is controlled via the input / output I / F 243, or notified to other devices via the network. Further, the video data can be stored in an external storage device 261 such as an external HDD (Hard Disk Drive) or a memory card via the external storage control unit 260.

  FIG. 2 is a diagram illustrating an example of a video distribution mode in the present embodiment. In FIG. 2, there are two different modes, for example, photographing interval, delivery delay, frame loss, protocol, and moving object detection.

  Mode 1 is “constant shooting interval, delay, no frame loss”, and the protocol uses, for example, HTTP. Since HTTP is a connection-oriented protocol using TCP, instead of guaranteeing the arrival of data at the protocol level, there is an overhead and a delay occurs because communication processing is performed while checking the packet transmission / reception state. Mode 1 is suitable for video recording in the video receiving apparatus 200 because there is no frame drop.

  Further, for the moving object detection in mode 1, inter-frame differences are used as image processing. The interframe difference is, for example, a moving object detection method that uses a sum of pixel value differences between two adjacent frame images, and is characterized by light processing. For this reason, the processing on the viewer side can be lightened. However, missing arrival frames occur, or accurate detection results cannot be obtained if the shooting interval between frames is not constant.

  On the other hand, mode 2 is “shooting interval is best effort, no delay, frame missing”, and the protocol uses, for example, RTP. Since RTP is a connectionless oriented protocol using UDP, the arrival of data is not guaranteed while the delay is small. For this reason, a frame may be lost. Mode 2 is suitable for applications such as monitoring and video phone because there is little delay in images.

  In addition, the background difference is used as the image processing in the moving object detection processing in mode 2. Background difference is a method of determining whether a background or foreground is applied by applying the current image to a background model created based on the statistical properties of the image. For this reason, even if a part of the frame is missing, the influence on the creation of the background model can be ignored. However, the processing load is heavy compared to the inter-frame difference, and memory resources are consumed when the background model is created.

  The distribution mode is set when the video reception device 200 requests the video transmission device 100 for video distribution connection. The video receiver 200 makes a connection request with mode information to the video transmitter 100. In response to this, the video transmitting apparatus 100 returns permission or rejection. A protocol for performing such connection control can be implemented by, for example, HTTP. In this case, for example, if transmission is performed by describing VideoTransferMode = 1 or the like in the HTTP BODY, a reply such as Accept or Denied is returned from the transmission device, so that permission or denial can be determined. Similarly, event notification to other devices can be realized by HTTP or UPnP (Universal Plug and Play).

  The operation procedure of the video transmission device 100 and the video reception device 200 in the present embodiment will be described with reference to FIGS. FIG. 3 is a flowchart illustrating an example of a processing procedure related to the change of the video distribution mode in the operation procedure of the video transmission apparatus 100, and FIG. 4 relates to the change of the video distribution mode in the operation procedure of the video reception apparatus 200. It is a flowchart which shows an example of a process sequence.

  In FIG. 3, the video transmission apparatus 100 waits for the occurrence of an event in step S301 after the processing is started. When the occurrence of an event is detected, it is determined in step S302 whether the event is a connection request event with a distribution mode. Here, the connection request with distribution mode is issued from the video receiving apparatus 200 at the start of connection and when the distribution mode is changed.

  If the request is not the result of the determination, other event processing is performed in step S309, and the process returns to step S301 to wait for the next event. Other events include, for example, a disconnect event, but detailed description thereof is omitted. On the other hand, if the result of determination in step S302 is a connection request with distribution mode, it is determined in step S303 whether connection is possible.

  Here, whether or not the distribution mode can be changed is determined by the presence / absence of a connected video receiving apparatus and the connection mode of another video receiving apparatus. For example, there may be a case where distribution in a different mode for each video reception device is impossible due to resource restrictions of the video transmission device. In such a case, when another video receiving apparatus is connected in mode 1, it may not be possible to shift to mode 2. This is because when a mode 1 is used for recording in a certain video receiving apparatus, that is prioritized.

  If the connection is impossible as a result of the determination in step S303, a connection rejection notification is returned to the video reception device 200 in step S308, and the process returns to step S301 to wait for the next event. On the other hand, if the connection is possible as a result of the determination in step S303, the process proceeds to step S304, and a connection permission notification is returned to the video reception device 200. Next, in step S305, a video shooting interval is set. In step S306, a video distribution protocol is set. In step S307, video distribution is started. Then, after completion, the process returns to step S301 to wait for the next event.

  On the other hand, in FIG. 4, the video receiving apparatus 200 waits for the occurrence of an event in step S401 after starting the processing. Events include connection request events and disconnection events. These events occur, for example, when an operator of the video receiving apparatus 200 operates an application GUI (Graphical User Interface) using an input device such as a mouse or a button. When the occurrence of an event is detected, it is determined whether or not the event that occurred in step S402 is a connection request with a delivery mode. If the result of this determination is not a connection request with distribution mode, other event processing is performed in step S408. As other event processing, there is a disconnection event, but the description is omitted. Then, after the process is completed, the process returns to step S401 to continue the process.

  On the other hand, if the result of determination in step S402 is a connection request with distribution mode, a connection request is made to the video transmitting apparatus 100 with distribution mode information attached in step S403. The video transmitting apparatus 100 receives this request in step S301 as already described. Next, in step S404, the video reception device 200 waits for a reply from the video transmission device 100, and determines whether or not connection permission has been performed. As a result of the determination, if the connection is rejected, the process returns to step S401 and waits for the next event.

  On the other hand, if the connection is permitted as a result of the determination in step S404, the video reception protocol is set in step S405, the moving object detection method is set in step S406, and then video reception is started in step S407. Then, after finishing the process, the process returns to step S401 to wait for the next event.

  As is clear from the above description, according to the present embodiment, an effective motion detection result can be provided regardless of the protocol by changing the motion detection method according to the video distribution protocol.

(Second Embodiment)
As another embodiment relating to the present invention, a new distribution mode without missing frames is provided in mode 2 in the first embodiment. This mode is referred to as mode 3. In mode 3, as in mode 2, RTP is used as a video transmission protocol, but when a frame drop occurs, retransmission is requested from the application. For this reason, video distribution with low delay and no frame dropping is realized. In the present embodiment, an example of an event notification function that achieves both promptness and reliability in the third distribution mode will be described. Note that the video transmission device and the video reception device of this embodiment are the same as those in FIG.

The video distribution and moving object detection method in this embodiment will be described with reference to FIG. FIG. 5 is a diagram illustrating collaboration between the video transmission device 100 and the video reception device 200. In addition, there is a moving object detection thread 201 executed by the video reception device 200. A frame is transmitted from the video transmission device 100 to the video reception device 200.
In FIG. 5, sequence numbers are assigned to video frames in the order of shooting. The captured frame 1 (601) is stored in the buffer. Next, it is transmitted by RTP and received as frame 1 (603) by the video receiver 200. The same frame is also stored in the buffer on the video receiving apparatus 200 side.

  Here, it is assumed that the next frame 2 is lost without reaching the video receiving apparatus 200. When the frame 3 is further transmitted, the video receiving apparatus 200 knows the disappearance of the frame 2 when the frame 3 is received, and therefore makes a retransmission request. Such a mechanism for a retransmission request in RTP can be implemented by a known technique using RTCP (Real Time Control Protocol).

  Further, consider a case where a difference between adjacent frames is used as a moving object detection method. When the frame 2 is missing, the difference cannot be performed between the frame 2 and the frame 1, so that the difference is performed between the frame 3 and the frame 1 instead. The difference result is notified as an event. This event is referred to as a “provisional event”. The event notification mechanism is the same as in the first embodiment, and a description thereof will be omitted. Next, when frame 2 is received by retransmission, inter-frame difference is performed again with frame 1 stored in the buffer. The result is notified again as an event. This event is referred to as a “normal event”. The difference between a provisional event and a primary event can be determined by describing the type of event in the BODY of the HTTP message when HTTP is used for event notification.

  In the above description, for simplification of description, it is assumed that a loss occurs in units of frames, but in reality, a loss occurs in units of packets. In some cases, the arrival order of packets may be mixed. Therefore, there may be a case where a missing frame is known before the frame is completely reproduced. In such a case, a retransmission request can be issued without waiting for reception of the next frame.

  Next, processing procedures of the video transmission device 100 and the video reception device 200 in the present embodiment will be described with reference to FIGS. 6 and 7. FIG. 6 is a flowchart illustrating an example of a processing procedure of a video distribution thread of the video transmission device 100. The connection processing procedure in this embodiment is the same as that in the first embodiment described with reference to FIG. Hereinafter, description will be limited to event notification processing at the time of a retransmission request specific to the present embodiment.

  After the start, it is determined in step S601 whether it is a normal transmission request or a retransmission request event. If the result of this determination is a normal transmission request, an image is acquired from the photographing unit in step S602. In step S603, the image is stored in a buffer. This is to store a certain frame in a buffer in advance assuming that a frame is lost. Next, an image is transmitted at step S604. After transmission, the process returns to step S601 to continue the processing.

  On the other hand, if it is determined in step S601 that there is a retransmission request from the video receiving apparatus 200, it is checked in step S605 whether there is a buffer for the frame. As a result of this determination, if it exists, the frame or a part of the frame is retransmitted. After the retransmission, the process returns to step S601 and continues. On the other hand, as a result of the determination in step S605, if the frame does not remain in the buffer, a notification that retransmission is not possible is performed to the video reception device 200 in step S607.

  Next, the processing procedure of the video receiving apparatus 200 in this embodiment will be described with reference to FIG. FIG. 7 is a flowchart illustrating an example of a processing procedure of the image acquisition / moving object detection thread of the video reception device 200. Similar to the video transmission apparatus 100, the connection processing procedure is the same as that in the first embodiment, and a description thereof will be omitted.

  After the start, image data is acquired in step S701. In step S702, it is confirmed whether the frame is the latest frame. If it is determined that the frame is the latest frame, it is checked in step S703 if there is no frame drop. If there is a frame drop as a result of this confirmation, a retransmission is requested in step S709. Then, it returns to step S701 and continues a process.

  On the other hand, if there is no frame drop as a result of the confirmation in step S703, the process proceeds to step S705, and a moving object detection process is performed. This moving object detection process is an inter-frame difference process using the latest frame and the immediately preceding usable frame. In step S706, the storage buffer is erased. The buffer is erased after the inter-frame difference is performed between the frame stored in the buffer and the next frame. For example, if a difference between adjacent frames is performed between the frame 11 and the frame 10 when the frame 10 is buffered, the buffer of the frame 10 is deleted. If an inter-frame difference is made between the frame 12 and the frame 10 without the arrival of the frame 11, the frame 10 is stored in the buffer until the frame 11 arrives.

  Next, in step S707, it is determined whether or not the detection state has changed as a result of the moving object detection process. If there is a change as a result of this determination, a moving object detection event is issued in step S708. On the other hand, if the result of determination in step S707 is that there is no change, processing returns to step S701 and processing continues.

  Here, the detection event includes a provisional event and a normal event as described above. When there is no frame drop and a difference between adjacent frames is performed, it is issued as a “normal event”, and when there is a frame drop and a difference between adjacent frames is not performed, it is issued as a “temporary event”.

  On the other hand, if the result of determination in step S702 is that the arrived frame is not the latest frame, the frame immediately before the arrival frame is read from the buffer in step S710. Next, a moving body detection process is performed at step S711. The moving object detection process here is the same process as step S705, and is an inter-frame difference using two images. In step S712, the read buffer image is erased. Next, a primary event is issued in step S713. And it returns to step S701 and continues a process.

  In the present embodiment, recording can also be performed. Generally, events that occur during recording are also recorded at the same time. Therefore, the event recording process when recording is to record only the normal event. Temporary events are always notified later, so recording is not necessary. The normal event is issued when it is an adjacent frame in step S708 and when a difference between adjacent frames is performed using a retransmission frame in step S713.

  As described above, according to the present embodiment, when a frame is dropped, a temporary event is issued by moving object detection using the previous and subsequent frames. Then, when the frame is retransmitted, the motion detection process is performed again and a normal event is issued. As a result, in the low-delay delivery protocol with a retransmission function, it is possible to achieve both the promptness and reliability of events.

(Third embodiment)
In the present embodiment, it is not always necessary to issue a primary event, and it is also possible to issue an event to be corrected only when it is necessary to correct the contents of a provisional event. Hereinafter, this is referred to as a “correction event”. Therefore, in this embodiment, a temporary event is also issued as a normal event. The operation procedure of the video receiving apparatus according to this embodiment will be described with reference to FIG. In FIG. 8, the same processes as those in FIG. 7 are given the same numbers. Therefore, only differences from FIG. 7 will be described. Note that the video transmission device and the video reception device of this embodiment are the same as those in FIG.

  The timing for issuing the correction event is when a delay frame arrives by retransmission. This corresponds to the case of branching to NO in step S702. Thereafter, after the moving object detection process in step S712, it is determined in step S813 whether a correction event is necessary. Here, whether or not it is necessary is determined based on whether or not the result of the motion detection performed previously and the result of the motion detection performed this time are the same. If the result of this determination is that the results are different, a correction event is necessary, so a correction event is issued in step S814. On the other hand, if the result of determination in step S813 is that a correction event is not necessary, processing returns to step S701.

  Here, there are a plurality of types of correction events. One is a new motion detection event, and the other is a motion detection cancellation event. In addition to this, an event for correcting the contents of the moving object detection result may be provided. This updates the object area information when the detection result of the moving object area is different.

  In the present embodiment, it is also possible to perform recording as in the second embodiment. The event in this embodiment may be corrected. Therefore, basically all events are recorded, but when a correction event is issued, the corresponding old event is deleted. In the present embodiment, the event is issued in steps S708 and S814. Among them, the corresponding old event record is deleted simultaneously with the issuance in step S814 where the correction event is issued.

  As described above, according to the present embodiment, when a frame is dropped, a temporary event is issued by moving object detection using the previous and subsequent frames. When the frame is retransmitted, the motion detection process is performed again. If the result is different from the temporary event, a correction event is issued. As a result, in the low-delay delivery protocol with a retransmission function, it is possible to achieve both the promptness and reliability of events.

(Other embodiments according to the present invention)
Each means constituting the video receiving apparatus and each step of the video receiving method in the embodiment of the present invention described above can be realized by operating a program stored in a RAM or ROM of a computer. This program and a computer-readable recording medium (storage medium) recording (storing) the program are included in the present invention.

  In addition, the present invention can be implemented as a system, apparatus, method, program, recording medium (storage medium), or the like, and can be applied to a system including a plurality of devices. It may also be applied to an apparatus consisting of a single device.

  In the present invention, a software program that realizes the functions of the above-described embodiments (in the embodiment, programs corresponding to the flowcharts shown in FIGS. 3, 4, and 6 to 8) can be directly applied to the system or apparatus, or Including the case of supplying remotely. This includes the case where the system or the computer of the apparatus is also achieved by reading and executing the supplied program code.

  Accordingly, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention.

  In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, and the like.

  Examples of the recording medium (storage medium) for supplying the program include a flexible disk, a hard disk, an optical disk, and a magneto-optical disk. Further, there are MO, CD-ROM, CD-R, CD-RW, magnetic tape, nonvolatile memory card, ROM, DVD (DVD-ROM, DVD-R) and the like.

  As another program supply method, there is a method of connecting to a homepage on the Internet using a browser of a client computer. Further, the computer program itself of the present invention or a compressed file including an automatic installation function can be downloaded from the homepage by downloading it to a recording medium (storage medium) such as a hard disk.

  It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, the present invention includes a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer.

  As another method, the program of the present invention is encrypted, stored in a recording medium (storage medium) such as a CD-ROM, distributed to users, and a homepage is established via the Internet for users who have cleared predetermined conditions. Download key information to decrypt from. It is also possible to execute the encrypted program by using the key information and install the program on a computer.

  Further, the functions of the above-described embodiments are realized by the computer executing the read program. Furthermore, based on the instructions of the program, an OS or the like running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments can be realized by the processing.

  Furthermore, as another method, a program read from a recording medium (storage medium) is first written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Then, based on the instructions of the program, the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are also realized by the processing.

  211 Video reception unit, 241 moving object detection unit, 242 moving object detection control unit

Claims (4)

A video receiving device having communication means and image processing means,
A plurality of video receiving means corresponding to a plurality of video distribution means having different communication protocols;
A video receiving apparatus comprising a plurality of image processing means corresponding to each of the plurality of video receiving means.   The plurality of video receiving means include one corresponding to video distribution using a protocol that guarantees retransmission of a video frame and one corresponding to other video distribution where retransmission of the video frame is not guaranteed. The video receiving device according to claim 1. A video receiving method of a video receiving apparatus having a communication means and an image processing means,
A plurality of video receiving steps corresponding to a plurality of video distribution means having different communication protocols;
And a plurality of image processing steps corresponding to each of the plurality of image receiving steps.   A program for causing a computer to execute each step of the video receiving method according to claim 3.

Images