JP2024074327A - Communication device, transmission system, control method for communication device, and program - Google Patents

Abstract

To achieve efficient packet transmission on a plurality of communication devices.SOLUTION: A communication device connected to a portion between a first other communication device and a second other communication device is configured to generate a packet, transmit the generated packet to the second other communication device when a predetermined packet is received from the first other communication device, or when a predetermined period of time elapses without the predetermined packet being received, and change the predetermined period to a shorter period on the basis of the number of times the predetermined period of time elapses without the predetermined packet being received.SELECTED DRAWING: Figure 4

Description

The present invention relates to technology for transmitting data between multiple communication devices.

Recently, virtual viewpoint image generation systems have become widespread, in which multiple cameras are installed in different positions to synchronously capture images from multiple viewpoints, and a virtual viewpoint image is generated using image data from the multiple viewpoints obtained by the capture. Patent Document 1 discloses a system in which a server device aggregates and uses image data transmitted from multiple communication devices connected in a daisy chain to generate virtual viewpoint content. The image data is image data obtained by capturing images using cameras connected to each of the multiple communication devices.

JP 2020-145559 A

In a system such as that described above, each of the multiple communication devices connected in a daisy chain needs to transmit not only packets containing image data (image packets) received from other adjacent communication devices, but also image packets acquired by each device itself to the other adjacent communication device.
Here, assume that one communication device (communication device B) among a plurality of communication devices connected in a daisy chain receives an image packet from another adjacent communication device (communication device A), triggering the transmission of the image packet acquired by communication device B to the other adjacent communication device (communication device C). In such a case, if communication device B does not receive the image packet transmitted from communication device A due to some kind of failure, the image packet acquired by communication device B cannot be transmitted to communication device C forever. In order to avoid such a problem, a timeout time is set in communication device B for a hold state of transmission of the image packet acquired by communication device B, and when a timeout occurs, communication device B becomes able to transmit the acquired image packet to communication device C.

However, if the timing for transmitting the image packets acquired by communication device B is always after a timeout occurs, the communication bandwidth is not used efficiently during the time until the timeout occurs, and efficient packet transmission cannot be continued.

The present invention was made in consideration of the above problems, and aims to achieve efficient packet transmission among multiple communication devices.

As one means for achieving the above object, the communication device of the present invention has the following configuration. That is, the communication device is connected between a first other communication device and a second other communication device, and has a generation means for generating packets, a reception means for receiving a predetermined packet from the first other communication device, a transmission means for transmitting the packet generated by the generation means to the second other communication device when the reception means receives the predetermined packet or when a predetermined period has elapsed without the reception means receiving the predetermined packet, and a change means for changing the predetermined period to a shorter period based on the number of times the reception means has not received the predetermined packet and the predetermined period has elapsed.

The present invention makes it possible to achieve efficient packet transmission among multiple communication devices.

1 illustrates an example of the configuration of a virtual viewpoint image generating system according to an embodiment. 1 is a block diagram showing an example of a configuration of a camera adapter according to an embodiment. FIG. 2 is a block diagram showing an example of a configuration of an image transmission unit according to the embodiment. 1 is a flowchart of a process performed by a sensor system according to an embodiment. FIG. 11 is a diagram showing a transmission sequence of an image packet according to an embodiment.

The following describes in detail an embodiment for carrying out the present invention with reference to the attached drawings. Note that the embodiment described below is one example of a means for realizing the present invention, and should be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions, and the present invention is not limited to the following embodiment. Furthermore, not all of the combinations of features described in this embodiment are necessarily essential to the solution of the present invention.

[Configuration of the virtual viewpoint image generation system]
FIG. 1 shows a configuration example of a virtual viewpoint image generating system 100 according to the present embodiment. The virtual viewpoint image generating system 100 shown in FIG. 1 is a system that generates virtual viewpoint content such as a virtual viewpoint image using data of multiple viewpoint images. The virtual viewpoint image generating system 100 includes a transmission system including sensor systems 110a to 110z, a hub 101, a control device 102, a time server 103, an image processing device 104, and a user device 105. The sensor systems 110a to 110z each include a camera adapter 111a to 111z and a camera 112a to 112z. The camera adapters 111a to 111z and the cameras 112a to 112z are connected by a first connection 113a to 113z for transmitting captured image data and a second connection 114a to 114z for transmitting camera control information.

Unless otherwise specified, the multiple sensor systems from sensor system 110a to sensor system 110z are collectively referred to as sensor system 110 without distinguishing between the 26 sets from sensor system 110a to sensor system 110z. Similarly, camera adapter 111a to camera adapter 111z are collectively referred to as camera adapter 111, and cameras 112a to 112z are collectively referred to as cameras 112. Also, first connection 113a to first connection 113z are collectively referred to as first connection 113, and second connection 114a to second connection 114z are collectively referred to as second connection 114. Sensor system 110a to sensor system 110z are connected by transmission paths 115b to 115z, respectively, and sensor system 110a and hub 101 are connected by transmission path 115a. The transmission path is, for example, a cable such as an Ethernet (registered trademark) cable.

In the virtual viewpoint image generation system 100, the camera 112 is controlled to capture images in a time-synchronized manner. In this embodiment, the Precision Time Protocol (PTP) is used as the protocol for time synchronization. The virtual viewpoint image generation system 100 is configured to execute a time synchronization sequence using packets conforming to the PTP (hereinafter, PTP packets).

The time server 103 functions as a time source device and has an accurate clock inside. The time server 103 executes time synchronization with each sensor system 110 using PTP packets including time information according to the clock.
The hub 101 is a switching hub with a transparent clock (TC) function. The TC function is a function that measures the residence time of a PTP packet in a device that receives the PTP packet, adds the residence time to the PTP packet, and bridges (transfers) the packet. The hub 101 is also capable of sending and receiving TCP/IP packets and other packets.

The sensor system 110 is a communication device composed of a camera 112 and a camera adapter 111. In the virtual viewpoint image generation system 100, the sensor systems 110a to 110z are connected by a daisy chain. The daisy chain connection has the effect of reducing the number of connection cables and saving the labor of wiring work when the volume of image data increases with the increase in resolution of the captured image to 4K or 8K and the increase in frame rate. In this embodiment, the word "image" is described as including the concepts of moving images and still images unless otherwise specified. In other words, the virtual viewpoint image generation system 100 according to this embodiment is capable of processing both still images and moving images. The 26 sets of image data acquired (generated) by the sensor systems 110a to 110z are transmitted to the image processing device 104.

The camera adapter 111 receives the PTP packet, and Genlocks the camera 112 based on the time information contained in the PTP packet, and performs image frame synchronization. The camera 112 is controlled via a second connection 114 for transmitting control information. In this embodiment, the camera adapter 111 has a TC function and an OC (Ordinary Clock) function in PTP. That is, when the camera adapter 111 transfers a received PTP packet, it calculates the residence time within the camera adapter 111, adds this to the PTP packet, and then transfers it to another subsequent camera adapter 111 (TC function). In addition, the camera adapter 111 itself synchronizes with the time of the time server 103 based on the received PTP packet (OC function).

All camera adapters 111 in the virtual viewpoint image generation system 100 have these two functions (TC function and OC function), which enables highly accurate time synchronization with the time server 103. The camera adapters 111 that are time-synchronized based on the time of the time server 103 can then synchronize the shooting timing of the cameras 112.

The control device 102 performs control over the operation status and parameter settings for each block constituting the virtual viewpoint image generation system 100 through a network. The control device 102 also specifies the viewpoint for the image processing device 104. The control information for performing these controls is mainly transmitted and received via a network (transmission path) using communication packets (hereinafter, control packets) defined by TCP/IP. The network is, for example, Ethernet, such as GbE (Gigabit Ethernet), 10 GbE, or 100 GbE that conforms to the IEEE standard. The network may also be configured by combining interconnect Infiniband, industrial Ethernet, and the like. The network is not limited to these, and may be other types of networks.

Next, the operation of transmitting 26 sets of image data acquired by sensor system 110z from sensor system 110a to image processing device 104 will be described. The imaging data captured by camera 112z is input to camera adapter 111z through first connection 113z for transmitting imaging data. The imaging data is then image-processed and packetized in camera adapter 111z. In this embodiment, the image-processed data from the imaging data is referred to as image data, and a packet containing image data (a unit in which image data is packetized) is referred to as an image packet. The image packet is transmitted to camera adapter 111y of sensor system 110y. Similarly, sensor system 110y transmits the image packet acquired from sensor system 110z to adjacent sensor system 110x, and then transmits the image packet generated from the imaging data captured by camera 112y. By continuing this operation, image packets generated by sensor systems 110a to 110z are transmitted from sensor system 110a via transmission path 115a to hub 101, and then transmitted to image processing device 104.

The image processing device 104 processes image data contained in the image packet acquired from the sensor system 110. First, the image data acquired from the sensor system 110 is reconstructed and the data format is converted, and then the data is stored according to the camera identifier, data type, and frame number. Then, the image processing device 104 receives a designation of a viewpoint from the control device 102, reads out corresponding image data from the stored information based on the viewpoint, and performs rendering processing to generate a virtual viewpoint image. Note that at least a part of the functions of the image processing device 104 may be possessed by the control device 102, the sensor system 110, or the user device 105. The image data that has been rendered is transmitted from the image processing device 104 to the user device 108, and the user operating the user device 108 can view the image of the viewpoint according to the designation. That is, the image processing device 104 generates virtual viewpoint content such as a virtual viewpoint image based on the captured images (multiple viewpoint images) captured by the multiple cameras 112 and viewpoint information. Note that, in this embodiment, the virtual viewpoint content is generated by the image processing device 104, but is not limited to this. That is, the virtual viewpoint content may be generated by the control device 102 or the user device 108.

In this embodiment, as shown in FIG. 1, the number of sensor systems is 26 sets, but this is merely an example, and the number may be any number. Furthermore, the multiple sensor systems 110 do not need to be of the same configuration, and for example, each may be composed of a different model of device. Note that the sensor system 110 may include an audio device such as a microphone, or a camera head that controls the orientation of the camera 112. The sensor system 110 may also be composed of one camera 112 and multiple camera adapters 111. The camera 112 and the camera adapter 111 may also be configured as an integrated unit. Furthermore, at least some of the functions of the camera adapter 111 may be provided by the image processing device 104.

Although FIG. 1 shows a configuration in which all of the camera adapters 111 are daisy-chained, this is not limiting. For example, multiple sensor systems 110 may be divided into several groups, and the sensor systems 110 may be daisy-chained for each group. This configuration is effective in stadiums. For example, a stadium may be made up of multiple floors, with a sensor system 110 installed on each floor. In this case, input to the image processing device 104 can be performed for each floor or for each half circumference of the stadium, simplifying installation and making the system more flexible even in places where it is difficult to wire all of the sensor systems 110 in a single daisy chain.

Note that since a malfunction of the time server 103 may cause the virtual viewpoint image generation system 100 to fail, multiple time servers 103 may be provided for redundancy. In that case, it is necessary to synchronize the times of the multiple time servers 103 using a global positioning system (GPS) or the like so that the times are aligned.

The transmission path 106 between the image processing device 104 and the hub 101 may be configured to enable larger capacity data transmission than the daisy-chained transmission paths 115a to 115z. For example, the transmission path 106 may be configured with a cable with a wider bandwidth than the cables used in the transmission paths 115a to 115z. This allows the image processing device 104 to receive all image data even if it has a sensor system group with multiple lanes. An example of a wideband cable is an optical cable.

[Camera adapter configuration]
Next, the configuration of the camera adapter 111 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the configuration of the camera adapter 111. Note that, although the camera adapter 111b in Fig. 1 will be described as an example here, the other camera adapters 111 may have a similar configuration.

The camera adapter 111b includes a CPU 201, a memory unit 202, a first communication I/F unit 203, a second communication I/F unit 204, a time synchronization control unit 205, an image processing unit 206, an imaging control unit 207, an image transmission unit 208, and a system bus 209.

The CPU (Central Processing Unit) 201 controls the entire camera adapter 111b, as well as controlling the exchange of PTP packets with the time server 103 and control packets with the control device 102. This control can be performed by the CPU 201 executing a control program stored in the storage unit 202.

The storage unit 202 is a memory that holds the control program executed by the CPU 201, and PTP packets and control packets to be transmitted and received. In addition, the image processing unit 206 holds, as image data, image processing data that has been subjected to image processing on the imaging data received from the camera 112b via the first connection 113b. Note that while only one storage unit 202 is shown in FIG. 2, the storage unit 202 may have multiple storage units depending on the application. In addition, the type of memory that constitutes the storage unit 202 is not limited to a specific one.

The first communication I/F (interface) unit 203 and the second communication I/F unit 204 transmit and receive communication packets (PTP packets, TCP/IP packets, and image packets) between the camera adapter 111b and an external device. DMA (Direct Memory Access) transfer instructions for transmitting and receiving packets are issued by the CPU 201 and the image transmission unit 208. In addition, the I/F unit 203 and the second communication I/F unit 204 each have a clock.

The operations of the first communication I/F unit 203 and the second communication I/F unit 204 in response to a DMA transfer instruction will be described.
First, a general operation of the first communication I/F unit 203 and the second communication I/F unit 204 will be described.
The DMA transfer instruction is accompanied by information specifying an area to read from or an area to write to in the storage unit 202. When the first communication I/F unit 203 transmits a packet, it reads the packet from the area in the storage unit 202 specified by the DMA transfer instruction and transmits it to the external camera adapter 111. On the other hand, when the first communication I/F unit 203 receives a packet from the external camera adapter 111, it writes the received packet to the area in the storage unit 202 specified by the DMA transfer instruction. The second communication I/F unit 204 operates in a similar manner.

Next, the transmission and reception process of the PTP packet will be described.
The PTP packet transmitted by the time server 103 is received by the second communication I/F unit 204 via the camera adapter 111a, and is transferred to a designated area of the storage unit 202 in response to a DMA transfer instruction from the CPU 201. Then, after protocol processing is performed on the PTP packet inside the camera adapter 111b, the packet is read from the designated area of the storage unit 202 in response to a DMA transfer instruction from the CPU 201, and is transmitted to the camera adapter 111c via the first communication I/F unit 203.
Similarly, a PTP packet transmitted to the time server 103 by the camera adapters 111c to 111z is received by the first communication I/F unit 203 and transferred once to the storage unit 202. Then, after protocol processing is performed on the PTP packet, the PTP packet is transferred from a specified area of the storage unit 202 to the second communication I/F unit 204 in response to a DMA transfer instruction from the CPU 201, and finally transferred to the time server 103.
The transfer of image packets will be described later.
Since it is not known when a packet will be received, the first communication I/F unit 203 and the second communication I/F unit 204 can be configured to buffer a plurality of transfer instructions in advance.

The first communication I/F unit 203 and the second communication I/F unit 204 have a function (time stamp function) of outputting the time of a clock inside each communication I/F unit at the timing of transmitting or receiving a PTP packet. The CPU 201 uses the time stamp function for calculating time synchronization.
In this embodiment, it is assumed that the clocks of the first communication I/F unit 203 and the second communication I/F unit 204 are synchronized. This makes it possible to accurately calculate the time that a PTP packet stays in the camera adapter 111b (stay time). In other words, the stay time can be calculated by subtracting the timestamp value output from the communication I/F unit to which the PTP packet is input from the timestamp value output when the PTP packet is output.

The imaging control unit 207 controls the camera 112b. When the imaging control unit 207 receives an instruction from the control device 102, it outputs an imaging instruction to the camera 112b. The imaging instruction includes a Genlock signal and a timecode, and is transmitted (output) to the camera 112b using the second connection 114. The Genlock signal and the timecode are generated based on a timing signal 210 received from the time synchronization control unit 205. The camera 112b captures an image in synchronization with the received Genlock signal, and outputs the imaging data together with the received timecode to the image processing unit 206 via the first connection 113.

The image processing unit 206 performs image processing (such as cutting out background and foreground parts) required to generate a virtual viewpoint video from the imaging data received from the camera 112b, and transfers the processing result (image data) to the storage unit 202. Furthermore, when the image processing unit 206 completes the image processing, it transmits image information to the image transmission unit 208 at the timing when it receives the timecode. In other words, the image processing unit 206 transmits image information to the image transmission unit 208 at the same intervals as the imaging frame rate (at periodic timing). The image information includes address information for each image type, such as foreground image and background image, as well as data size, frame number, etc., and is transmitted using the image information signal 211.

The image transmission unit 208 generates image packets by packetizing image data based on the image information signal 211 received from the image processing unit 206, and issues a transmission instruction (transfer instruction) to the second communication I/F unit 204. The image transmission unit 208 also has a function of transferring image packets received from the camera adapter 111c via the second communication I/F unit 204 to the camera adapter 111a. The image transmission unit 208 receives the image information signal 211 generated based on the time synchronized with the time server 103 at the same interval as the imaging frame rate, and therefore the transmission instruction for the image packets is also issued periodically.

The time synchronization control unit 205 has a clock that is synchronized with the clocks of the first communication I/F unit 203 and the second communication I/F unit 204. Furthermore, the time synchronization control unit 205 generates a pulse signal (e.g., 1 Hz) synchronized with its own clock, and outputs the pulse signal as a timing signal 210 to the imaging control unit 207 and the image transmission unit 208.

[Configuration of image transmission section]
Next, the configuration of the image transmission unit 208 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example of the configuration of the image transmission unit 208. Note that, although the image transmission unit 208 of the camera adaptor 111b in Fig. 2 will be described here, the image transmission units 208 of the other camera adaptors 111 may have a similar configuration.

The image transmission unit 208 is composed of a setting unit 301, an image transmission control unit 302, a receiving buffer unit 303, a forwarding unit 304, an arbitration unit 305, a sending buffer unit 306, an image packet generation unit 307, and a start timer unit 308.

The setting unit 301 holds setting information of the image transmission unit 208 set by the CPU 201. Each functional block of the image transmission unit 208 is configured to be able to acquire the setting information.
The receiving buffer unit 303 temporarily holds image packets received by the first communication I/F unit 203 from the camera adapter 111 c of the sensor system 110 c , and transfers them to the forwarding unit 304 .

The forwarding unit 304 analyzes the image packet received from the receiving buffer unit 303. Specifically, the forwarding unit 304 analyzes whether the received image packet is the last packet or not. This analysis includes obtaining information indicating which camera adapter the image packet was sent from and which image packet it is. If the forwarding unit 304 determines that the image packet is the last packet, it notifies the image transmission control unit 302 of the detection of the last packet. Furthermore, the forwarding unit 304 edits the header information of the received image packet. Specifically, if the destination MAC address of the received image packet is addressed to the camera adapter 111b (i.e., addressed to the device itself), the forwarding unit 304 modifies (edits) the header information so that the destination of the packet is the camera adapter 111a. The forwarding unit 304 then transmits the image packet with the modified header information to the arbitration unit 305.

When the image transmission control unit 302 is notified by the forwarding unit 304 that the last packet has been detected, it outputs a packet generation instruction to the image packet generating unit 307. The image transmission control unit 302 also acquires information on the period of the timing signal 210, the number of sensor systems 110 in the virtual viewpoint image generating system 100, and the position of the sensor system 110b in the sensor system 110, for example, from setting information. The image transmission control unit 302 then determines a default timer value (predetermined period) for the start timer unit 308 based on the setting information, and sets it in the start timer unit 308. A method for determining the default timer value will be described later. The image transmission control unit 302 is also configured to change the timer value based on the number of timeouts of the start timer unit 308.

The image packet generating unit 307 generates image packets. The image packet generating unit 307 starts generating image packets when triggered by a packet generation instruction from the image transmission control unit 302. The image packet generating unit 307, which has received a packet generation instruction from the image transmission control unit 302, acquires image data recorded in the storage unit 202 via the system bus 209 using the image information contained in the image information signal 211 received from the image processing unit 206. The image packet generating unit 307 then generates a header for the image packet required to transmit the acquired image data, and generates an image packet from the image data and the header. The image packet generating unit 307 transmits the generated image packet to the arbitration unit 305. When the image packet generating unit 307 has completed the generation and transmission of one frame of image data, it notifies the image transmission control unit 302 of the completion of transmission.

The arbitration unit 305 transmits (transfers) the image packets received from the forward unit 304 and the image packet generation unit 307 to the transmission buffer unit 306. Furthermore, the adjustment unit 305 adjusts the transmission timing when the transmission timings of the image packets received from the forward unit 304 and the image packet generation unit 307 to the transmission buffer unit 306 overlap. In this embodiment, the adjustment unit 305 is configured to give priority to transmitting the image packets received from the forward unit 304 to the transmission buffer unit 306 when the transmission timings overlap.
Upon receiving the image packet from the arbitration unit 305 , the transmission buffer unit 306 transfers the image packet to the second communication I/F unit 204 .

The start timer unit 308 has a timer function and is controlled by the image transmission control unit 302. A default timer value is set in the start timer unit 308 in advance by the image transmission control unit 302. When the timing signal 210 is input from the time synchronization control unit 205, the start timer unit 308 starts timer counting. After the timer count starts, when the timer value set by the image transmission control unit 302 has elapsed, the start timer unit 308 notifies the image transmission unit control unit 302 of a timeout (performs a timeout notification). The start timer unit 308 also includes a counter for counting (incrementing) and holding the number of timeouts (the number of times a timeout has occurred). The number of timeouts can be controlled by the image transmission control unit 302. For example, the start timer unit 308 can clear and/or increment the number of timeouts by the image transmission control unit 302. As will be described later, the default timer value is changed (updated) by the image transmission control unit 302 based on the number of timeouts.

[Operation of the sensor system]
Next, the operation of the sensor system 110 will be described with reference to the flowchart of Fig. 4. Fig. 4 is a flowchart of processing executed by the sensor system 110 according to this embodiment. The processing of Fig. 4 will be described below with reference to the configurations of the sensor system 110b and the camera adaptor 111b shown in Fig. 2 and Fig. 3. The processing of Fig. 4 can be started after the sensor system 110b receives an instruction to start generating a virtual viewpoint image from the user device 105 or the control device 102.

When the time synchronization control unit 205 generates the timing signal 210 (Yes in S401), it outputs the timing signal 210 to the imaging control unit 207 and the image transmission unit 208, and the process proceeds to S402 and S404. While the timing signal 210 is not generated (No in S401), the time synchronization control unit 205 waits.

In S402, the imaging control unit 207 receives the timing signal 210 and outputs an imaging instruction generated based on the timing signal 210 to the camera 112b. The imaging instruction includes a Genlock signal and a Timecode. The camera 112b captures an image in accordance with the imaging instruction and outputs the captured image data to the image processing unit 206. The image processing unit 206 stores the generated image data by performing image processing on the captured image data in the storage unit 202 (S403).

In S404, the start timer unit 308 receives the timing signal 210 and starts the timer count. In this embodiment, a default timer value is set in the start timer unit 308 by the image transmission control unit 302. This timer value is the timer value set for the sensor system 110b.

When the first communication I/F unit 203 receives an image packet from the sensor system 110c (i.e., the preceding sensor system) (Yes in S405), the receiving buffer unit 303 stores the received image packet (S406). Note that in FIG. 4, the image packet received from the sensor system 110c is shown as a received packet to distinguish it from the image packet generated from the image data (S403) generated by the sensor system 110b. The receiving buffer unit 303 then transmits (transfers) the received image packet to the forwarding unit 304 (S407). The forwarding unit 304 analyzes whether the image packet received from the receiving buffer unit 303 is the last packet (S408). The forwarding unit 304 also edits the header information of the image packet (S408).

In this embodiment, the last packet refers to a packet that satisfies two conditions.
The first condition is that the source of the received image packet is a sensor system in the previous stage (connected to one side). For example, in the case of the sensor system 110b, the sensor system in the previous stage is the sensor system 110c. The forwarding unit 304 can detect the source sensor system by using, for example, source information (such as a MAC address) included in the received image packet.
The second condition is that the received image packet is the last packet among a plurality of packets that make up one frame of an image. For example, the forwarding unit 304 can be configured to detect whether the received image packet is the last packet based on a sequence number included in the image packet.

When the forward unit 304 determines as a result of the analysis that the received image packet is the last packet (Yes in S409), it notifies the image transmission control unit 302 that the last packet has been detected (S410). The image transmission control unit 302, which has been notified of the last packet detection by the forward unit 304, sets (sets) a last packet detection flag (S411). Note that the last packet detection flag can be replaced by other means as long as the image transmission control unit 302 recognizes that it has been notified of the last packet detection by the forward unit 304.

In S412, the second communication I/F unit 204 transmits the image packet to the downstream sensor system (connected to the other side). Specifically, the forward unit 304 transmits the received image packet to the arbitration unit 305, and the arbitration unit 305 outputs the image packet to the transmission buffer unit 306. When the transmission buffer unit 306 receives the image packet from the arbitration unit 305, it transfers it to the second communication I/F unit 204. The second communication I/F unit 204 transmits the image packet to the downstream sensor system (sensor system 110a in this example). After that, the process proceeds to S405. In the sensor system 110b, if the image transmission control unit 302 does not confirm that the last packet notification flag is set (No in S413) and the start timer unit 308 has not timed out (No in S421), the process continues from S405 to S412.

When the image transmission control unit 302 confirms that the last packet notification flag is set (Yes in S413), it outputs a packet generation instruction to the image packet generating unit 307 (S414). Then, the image packet generating unit 307 performs the following process as long as unsent image data of one frame's worth of image data is stored in the storage unit 202 (Yes in S415).
First, the image packet generating unit 307 generates an image packet (S416). Specifically, the image packet generating unit 307 extracts data of a size required for a packet from the untransmitted image data stored in the storage unit 202, generates a header of an image packet required for transmitting the data, and generates an image packet from the data and the header. The image packet generating unit 307 then transmits the generated image packet to the arbitration unit 305, and the arbitration unit 305 outputs the image packet to the transmission buffer unit 306. When the transmission buffer unit 306 receives the image packet from the arbitration unit 305, it transfers it to the second communication I/F unit 204. The second communication I/F unit 204 transmits the image packet to the downstream sensor system (S417).

When the transmission of all image data constituting one frame of image is completed (No in S415), the image packet generation unit 307 notifies the image transmission control unit 302 that the image data transmission is completed (S418). When the image transmission control unit 302 is notified that the image data transmission is completed, it clears the number of timeouts (number of timeout occurrences) in the start timer unit 308 to 0 (S419). This is because in S413, the image transmission control unit 302 received a last packet detection (the last packet detection flag was set) before the start timer unit 308 timed out. Then, the image transmission control unit 302 sets the timer value of the start timer unit 308 to the default (S420). The process then proceeds to S430.

If the image transmission control unit 302 cannot confirm that the last packet notification flag is set (No in S413) and the start timer unit 308 times out (Yes in S421), the process proceeds to S422. Then, the image packet generation unit 307 generates an image packet (S423) and transmits it (S424) as long as unsent image data of one frame's worth of image data is stored in the storage unit 202 (Yes in S422). The processes of S423 and S424 are the same as those of S416 and S417.

When the transmission of all image data constituting one frame of image is completed (No in S422), the image packet generation unit 307 notifies the image transmission control unit 302 of the completion of the image data transmission (S425). When the image transmission control unit 302 is notified of the completion of the image data transmission, it increments the timeout count for the start timer unit 308 (S426). In S427, the start timer unit 308 may be configured to automatically increment the timeout count. Thereafter, when the timeout count reaches a predetermined number (Yes in S427), the image transmission control unit 302 changes the default timer value (i.e., the time until a timeout occurs) of the start timer unit 308 (S428). In addition, the image transmission control unit 302 clears the timeout count to 0 (S429). After the process of S429, and when the timeout count has not reached the predetermined number when the timeout count is incremented (S426) (No in S427), the process proceeds to S430.

In S430, if the last packet detection flag is set, the image transmission control unit 302 lowers the flag. Thereafter, if the sensor system 110b detects an instruction to end the generation of the virtual viewpoint image from the user device 105, the control device 102, etc. (Yes in S431), the process shown in FIG. 4 ends. If the instruction is not detected (No in S431), the process proceeds to S401.

[Image packet transmission example]
Next, an example of image packet transmission between the multiple sensor systems 110 in the virtual viewpoint image generating system 100 will be described. Fig. 5 shows a transmission sequence of image packets according to this embodiment. For the sake of simplicity, Fig. 5 shows a transmission sequence of image packets captured and generated in the four sensor systems 110z, 110y, 110x, and 110w through four transmission paths 115z, 115y, 115x, and 115w.

In FIG. 5, the horizontal axis represents time, and the vertical axis represents the state of each transmission path. Times T1, T2, ... represent the generation timing of the timing signal 210 in the sensor system 110. Also in FIG. 5, time ty1 represents the default timer value set for the sensor system 110y. Similarly, time tx1 represents the default timer value set for the sensor system 110x, and time tw1 represents the default timer value set for the sensor system 110w. Also, in the example of FIG. 5, the number of timeouts that trigger a change (update) of the default timer value (corresponding to the predetermined number of times in S427 in FIG. 4) is assumed to be three.

When the timing signal T1 is generated, each of the sensor systems 110z, 110y, 110x, and 110w immediately captures an image and generates an image packet. Then, a group of image packets constituting one frame of an image are sequentially transmitted from the sensor system 110z.
First, the sensor system 110z transmits an image packet group 501 generated in the sensor system 110z through the transmission path 115z. The image packet group 501 is made up of a plurality of image packets, and the header of each image packet includes information on the sender and information (e.g., a sequence number) that can detect whether the image packet is the last packet or not.

The sensor system 110y receives the image packet group 501 via the transmission path 115z, and transmits (transfers) an image packet group 502 in which the header information of each image packet in the image packet group 501 has been updated to the sensor system 110x via the transmission path 115y. The sensor system 110y also analyzes each image packet in the image packet group 501. When the sensor system 110y detects that the image packet included in the image packet group 501 is the last packet (Yes in S409 in FIG. 4), after transmitting the image packet group 502, it transmits the image packet group 503 generated by the sensor system 110y to the sensor system 110x via the transmission path 115y (S417 in FIG. 4).
Thereafter, the sensor system 110x repeats the same process, and the image packets generated in the sensor systems 110z, 110y, and 110x are sequentially transmitted to the transmission path 115x. Subsequently, the sensor system 110w repeats the same process, and the image packets generated in the sensor systems 110z, 110y, 110x, and 110w are sequentially transmitted to the transmission path 115w.

Here, assume that the camera 112y of the sensor system 110y breaks down at time t1, which is before the timing signal 210 is generated at time T2. In this case, after time T2, the sensor system 110z transmits the image packet group 504 generated in the sensor system 110z to the sensor system 110y. Next, the sensor system 110y transmits the image packet group 505, in which the header information of each image packet of the image packet group 504 is updated, to the sensor system 110x. However, the sensor system 110y is unable to generate or transmit the image packet group 506 in the sensor system 110y.
When the sensor system 110x receives the image packet group 505, it transmits to the sensor system 110w an image packet group 507 in which the header information of each image packet in the image packet group 505 is updated. However, since the sensor system 110x cannot receive the image packet group 506 from the sensor system 110y, it cannot detect that the image packet included in the image packet group 506 is the last packet.

Because the sensor system 110x cannot detect the last packet, a period occurs during which the image packet group 508 generated by the sensor system 110x cannot be transmitted. After that, the start timer unit 308 of the sensor system 110x times out, allowing the image packet group 508 to be transmitted to the sensor system 110y. That is, the sensor system 110x can transmit the image packet group 508 to the sensor system 110y after the timer value tx1 has elapsed. When the start timer unit 308 of the sensor system 110x times out three times in a row (occurring between times T2 and T3, T3 and T4, and T4 and T5 in the example of FIG. 5), the default timer value tx1 set in the sensor system 110x is changed (S428 in FIG. 4).

5, at time t3, the image transmission control unit 302 of the sensor system 110x changes the default timer value for the sensor system 110x from timer value tx1 to timer value tx2. The change to timer value tx2 may occur at any time before the generation time T5 of the subsequent timing signal. When the timing signal occurs at time T5, the sensor system 110x transmits image packets 509 including image data generated in the sensor system 110z, and at this point, the start timer unit 308 of the sensor system 110z has already timed out. In other words, after the sensor system 110x transmits the image packets 509, the timer value tx2 has already elapsed. Therefore, the sensor system 110x can promptly transmit the image packets 510 generated in the sensor system 110x to the sensor system 110w.

[How the default timer value is determined]
A method for determining a default timer value of the start timer unit 308 to be set in each sensor system 110 will be described with reference to Fig. 1. In this embodiment, the timer value is determined by the image transmission control unit 302 of each sensor system 110, but is not limited to this. In this embodiment, the image transmission control unit 302 of each sensor system 110 acquires information on the period of the timing signal 210, the number of sensor systems 110 in the virtual viewpoint image generation system 100, and the position of the sensor system itself in the sensor system 110 in advance.

The default timer value of the first sensor system 110z among the sensor systems 110a to 110z is set to 0 in order to immediately start transmitting image packets. On the other hand, the default timer value of the other sensor systems 110a to 110y is determined as follows. If the number of sensor systems in the virtual viewpoint image generation system 100 is n and the period of the timing signal is T (seconds), then T/n (seconds) is the approximate time that is allowable for transmitting a group of image packets per sensor system. Therefore, the sensor systems 110a to 110y determine the timer value as a value obtained by multiplying T/n (seconds) by a coefficient according to the position (head/end) of each sensor system. In addition, when the number of timeouts reaches a predetermined number, the default timer value is determined (changed) so that the value (period) is gradually decreased by T/n (seconds) from the previously determined default timer value.

A specific example of the default timer value for the sensor system 110x in Fig. 1 will be described. For example, in a 60 FPS video image, the period of the timing signal is 16.7 (milliseconds), and in a 26-unit system (n=26) as shown in Fig. 1, T/n ≈ 0.64 (milliseconds). Since the sensor system 110x is the third unit, 0.64 x (3-1) = 1.28 (milliseconds) can be determined as the default timer value. Then, when a timeout occurs consecutively in the start timer unit 308 of the sensor system 110x and the number of timeouts reaches a predetermined number (Yes in S427 in Fig. 4), the value is decreased by T/n (seconds) from the previously determined default timer value.
Note that the method of determining the default timer value described here is just one example, and it is sufficient if the default timer value is changed so that it is smaller than the previously determined default timer value each time the number of timeouts reaches a predetermined number.

In this way, according to this embodiment, the timer value of the start timer unit 308 can be appropriately set according to the image packet transmission status in each sensor system 110. This makes it possible to continue efficient image packet transmission without wasting communication bandwidth even if a failure occurs in a sensor system 110 or in a transmission path between sensor systems 110 in the virtual viewpoint image generation system 100. In addition, it becomes possible to transmit image data generated at each specified timing promptly (for example, by a specified time) to the image processing device 104, making it possible to continuously generate high-quality virtual viewpoint content in real time.

In the above embodiment, the timer value is based on the time when the timing signal 210 is generated, but other timings may be used as the reference. For example, the reference may be the timing when the image information signal 211 reaches (is generated by) the image transmission unit 208. Furthermore, the reference may be the timing when the sensor system 110 receives a specified packet. For example, the detection time of a PTP packet (a packet that distributes synchronization time) periodically output by the time server 103 may be used as the reference.

In the above embodiment, the camera adapter 111 is configured to start transmitting the image packets acquired by the camera adapter 111 when the camera adapter 111 receives the last packet as a trigger, but the trigger is not limited to the reception of the last packet. It may be a specific packet constituting an image packet group or a specially set packet.
In the above embodiment, the packets generated and transmitted by the camera adapter 111 are image packets generated based on image data captured by a camera, but other types of packets may be used. For example, the camera adapter 111 may be configured to generate and transmit all types of packets to other camera adapters 111.

[Other embodiments]
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

The disclosure of this embodiment includes the following configurations and methods.
(Configuration 1) A communication device connected between a first other communication device and a second other communication device, comprising: a generating means for generating a packet; a receiving means for receiving a specified packet from the first other communication device; a transmitting means for transmitting the packet generated by the generating means to the second other communication device when the receiving means receives the specified packet or when a specified period has elapsed without the receiving means receiving the specified packet; and a changing means for changing the specified period to a shorter period based on the number of times the specified period has elapsed without the receiving means receiving the specified packet.

(Configuration 2) The communication device described in configuration 1, in which the generating means generates image data by performing image processing on imaging data obtained by imaging with an imaging device, and generates a packet including the image data as the packet.

(Configuration 3) The communication device according to configuration 2, wherein the imaging device is included in the communication device.

(Configuration 4) The communication device according to configuration 2, wherein the imaging device is connected to the communication device.

(Configuration 5) The communication device according to any one of configurations 1 to 4, wherein the change means changes the predetermined period to a shorter period when the number of times reaches a predetermined number.

(Configuration 6) The communication device described in any one of configurations 1 to 5, wherein the specified packet is the last packet of multiple packets constituting one packet group from the first other communication device.

(Configuration 7) A communication device according to any one of configurations 1 to 6, in which the generating means generates the packets based on a periodic timing, and the predetermined period is set based on the period of the timing.

(Configuration 8) A communication device according to any one of configurations 1 to 7, in which, after the change means changes the specified period, if the receiving means receives the specified packet before the changed specified period has elapsed, the change means returns the changed specified period to the specified period.

(Configuration 9) The communication device according to any one of configurations 1 to 8, wherein the change means changes the specified period to a shorter period based on the number of times the changed specified period has elapsed without the receiving means receiving the specified packet after the change.

(Configuration 10) A transmission system having a plurality of communication devices connected in a daisy chain, each of the plurality of communication devices having a generating means for generating a packet, a receiving means for receiving a predetermined packet from a first other communication device connected to one of the communication devices, a transmitting means for transmitting the packet generated by the generating means to a second other communication device connected to the other communication device when the receiving means receives the predetermined packet or when a predetermined period has elapsed without the receiving means receiving the predetermined packet, and a changing means for changing the predetermined period to a shorter period based on the number of times the receiving means has not received the predetermined packet and the predetermined period has elapsed.

(Configuration 11) The transmission system described in configuration 11, in which the predetermined period is set differently for the multiple communication devices.

(Configuration 12) The transmission system described in configuration 11, in which the predetermined period is set to become gradually longer in the order in which packets generated by the generating means of each of the plurality of communication devices are transmitted.

(Configuration 13) The transmission system described in any one of configurations 10 to 12, wherein the generating means generates the packets based on a periodic timing, and the predetermined period is set based on the period of the timing.

(Method 1) A control method for a communication device connected between a first other communication device and a second other communication device, the control method comprising: a generation step of generating a packet; a reception step of receiving a predetermined packet from the first other communication device; a transmission step of transmitting the generated packet to the second other communication device when the predetermined packet is received or when a predetermined period has elapsed without the predetermined packet being received; and a modification step of modifying the predetermined period to a shorter period based on the number of times the predetermined period has elapsed without the predetermined packet being received.

100: Virtual viewpoint image generation system, 101: Hub, 102: Control device, 103: Time server, 104: Image processing device, 105: User device, 106: Transmission path, 110a to 110z: Sensor system, 111a to 111z: Camera adapter, 112a to 112z: Camera, 113a to 113z: First connection, 114a to 114z: Second connection, 115a to 115z: Transmission path, 201: CPU, 202: Storage unit, 203: First communication I/F unit, 204: Second communication I/F unit, 205: Time synchronization control unit, 206: Image processing unit, 207: Imaging control unit, 208: Image transmission unit, 209: System bus, 210: Timing signal, 211: Image information signal, 301: Setting unit, 302: Image transmission control unit, 303: Reception buffer unit, 304: Forward unit, 305: Adjustment unit, 306: Transmission buffer unit, 307: Image packet generation unit, 308: Start timer unit

Claims (15)

A communication device connected between a first other communication device and a second other communication device,
generating means for generating packets;
receiving means for receiving a predetermined packet from the first other communication device;
a transmitting means for transmitting the packet generated by the generating means to the second other communication device when the receiving means receives the predetermined packet or when a predetermined period of time has elapsed without the receiving means receiving the predetermined packet;
a change means for changing the predetermined period to a shorter period based on the number of times the predetermined period has elapsed without the receiving means receiving the predetermined packet;
A communication device comprising: the generating means generates image data by performing image processing on imaging data obtained by imaging using an imaging device, and generates a packet including the image data as the packet;
2. The communication device according to claim 1 . The communication device according to claim 2, characterized in that the imaging device is included in the communication device. The communication device according to claim 2, characterized in that the imaging device is connected to the communication device. The change means changes the predetermined period to a shorter period when the number of times reaches a predetermined number.
5. The communication device according to claim 1, wherein the first and second inputs are connected to the first and second inputs. the predetermined packet is a last packet of a plurality of packets constituting one packet group from the first other communication device;
2. The communication device according to claim 1 . The generating means generates the packets based on a periodic timing;
The predetermined period is set based on a cycle of the timing.
2. The communication device according to claim 1 . If, after the change means has changed the predetermined period, the receiving means receives the predetermined packet before the changed predetermined period has elapsed,
The change means returns the changed predetermined period to the predetermined period.
2. The communication device according to claim 1 . the changing means changes the changed predetermined period to a shorter period based on the number of times the changed predetermined period has elapsed without the receiving means receiving the predetermined packet after the change of the predetermined period.
2. The communication device according to claim 1 . A transmission system having a plurality of communication devices connected in a daisy chain,
Each of the plurality of communication devices
generating means for generating packets;
a receiving means for receiving a predetermined packet from a first other communication device connected to one of the first communication devices;
a transmitting means for transmitting the packet generated by the generating means to a second other communication device connected to the other side when the receiving means receives the predetermined packet or when a predetermined period of time has elapsed without the receiving means receiving the predetermined packet;
a change means for changing the predetermined period to a shorter period based on the number of times the predetermined period has elapsed without the receiving means receiving the predetermined packet;
A transmission system comprising: the predetermined period is set to be different for each of the plurality of communication devices;
11. The transmission system according to claim 10. the predetermined period is set to be gradually longer in the order in which the packets generated by the generation means of each of the plurality of communication devices are transmitted.
12. The transmission system according to claim 11 . The generating means generates the packets based on a periodic timing;
The predetermined period is set based on a cycle of the timing.
13. A transmission system according to claim 10, wherein the first and second inputs are connected to the first and second inputs. A method for controlling a communication device connected between a first other communication device and a second other communication device, comprising the steps of:
a generating step of generating packets;
a receiving step of receiving a predetermined packet from the first other communication device;
a transmitting step of transmitting the generated packet to the second other communication device when the predetermined packet is received or when a predetermined period of time has elapsed without the predetermined packet being received;
changing the predetermined period to a shorter period based on the number of times the predetermined period has elapsed without the predetermined packet being received;
A control method comprising the steps of: A program for causing a computer to function as each of the means of the communication device described in claim 1.

Images