JP2024147867A - COMMUNICATION DEVICE, COMMUNICATION SYSTEM, COMMUNICATION METHOD, AND PROGRAM - Google Patents

Abstract

[Problem] To avoid remaining timestamps. [Solution] A communication device includes a receiving means for receiving a packet and outputting a timestamp indicating the time of reception, a transferring means for transferring the received packet to a predetermined memory, a storage means for storing the outputted timestamp, and a deleting means for deleting the timestamp corresponding to the discarded packet from the storage means when the transferring means discards the packet. [Selected Figure] Figure 2

Description

The present invention relates to a communication device, a communication system, a communication method, and a program.

In recent years, technology has been attracting attention for capturing images from multiple viewpoints by synchronously shooting with multiple cameras installed in different locations, and generating virtual viewpoint content from the captured images from multiple viewpoints. In addition, technology is known that realizes the synchronization control, which is essential for synchronous shooting, through network communication.

One method of synchronization control through network communication is, for example, PTP (Precision Time Protocol) in IEEE 1588-2008. PTP is a technology in which a master device with accurate time transmits time information in PTP packets, and slave devices receive the PTP packets to perform time synchronization.

Time stamps are used in PTP time synchronization, and the time stamps are obtained, for example, at the PHY layer (Physical Layer) or MAC layer (Media Access Control) when a PTP packet is sent or received by a communication device. The obtained time stamps are then stored in a memory unit accessible by the CPU (Central Processing Unit) and used for time synchronization processing.

For example, Patent Document 1 discloses a technology in which a communication device receives a PTP packet and stores a timestamp corresponding to the PTP packet in a memory unit. In the technology of Patent Document 1, a timestamp is assigned to all received packets, including PTP packets used for time synchronization, after processing by the PHY processing unit is completed. Then, after processing by the MAC processing unit is completed, a PTP packet is identified from the received packet. If it is a PTP packet, a timestamp is stored in the memory unit. The received packet is then forwarded to a higher layer.

JP 2021-256965 A

However, with the technology described in Patent Document 1, if a received packet is discarded during transfer, there is a possibility that the timestamp corresponding to the received packet will remain in the storage unit. If the timestamp remains in the storage unit, the correspondence between the received packet and the timestamp will become misaligned, and time synchronization processing will not be performed correctly.

The problem this invention aims to solve is to avoid extra timestamps.

To solve the above problem, a communication device according to one aspect of the present invention includes a receiving means for receiving a packet and outputting a timestamp indicating the time of reception, a forwarding means for forwarding the received packet to a predetermined memory, a storage means for storing the outputted timestamp, and a deleting means for deleting the timestamp corresponding to the discarded packet from the storage means when the forwarding means discards the packet.

The present invention makes it possible to avoid excess timestamps.

1 is a system configuration diagram of a synchronous imaging system to which the present invention is applied. FIG. 4 is a block diagram showing a network control function in the camera adapter. FIG. 4 is a diagram showing a sequence of a time synchronization process. 11 is a flowchart showing a process when a packet is received. 11 is a flowchart showing a process of allocating received time stamps. 11 is a flowchart showing a transfer process of a received packet. 13 is a flowchart showing a process of deleting a received time stamp. 13 is a flowchart showing a time synchronization process.

Below, the embodiments of the present invention will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the present invention, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the present invention. The configuration of the embodiments may be modified or changed as appropriate depending on the specifications of the system and device to which the present invention is applied and various conditions (conditions of use, environment of use, etc.). The technical scope of the present invention is defined by the claims, not by the individual embodiments below.

In the following description, components shown in earlier referenced drawings will be referred to as appropriate in the description of later drawings.
<Configuration of Synchronized Shooting System>

FIG. 1 is a system configuration diagram of a synchronous imaging system to which the present invention is applied.
The synchronous imaging system 100 is a system that captures images by installing multiple cameras in a facility such as a sports stadium or concert hall.
The synchronized photography system 100 includes a time server 101, a hub 102, a control terminal 103, an image computing server 104, a user terminal 105, and sensor systems 110a to 110z. A network is formed by these elements in the synchronized photography system 100, and the hub 102 has a routing function for communication packets. The hub 102 and the sensor systems 110a to 110z are daisy-chained together by network cables 106a to 106z.

In the following description, when elements of the same type that are distinguished by alphabetical subscripts of the reference numerals are collectively referred to, the reference numerals may be used without the subscripts. For example, 26 sets of sensor systems 110a to 110z may be collectively referred to as sensor system 110.
In a network in which multiple sensor systems 110a to 110z are connected, the side closer to the hub 102 is referred to as the root side, and the side farther from the hub 102 is referred to as the leaf side. Also, in the internal configuration of the sensor system 110, the side closer to the hub 102 is referred to as the root side, and the side farther from the hub 102 is referred to as the leaf side.

As an example, the synchronized photography system 100 includes 26 sets of sensor systems 110a-110z, but the number of sets is not limited to this. Furthermore, the multiple sensor systems 110 do not need to have the same configuration, and for example, each may be configured with a different model of device. In this embodiment, unless otherwise specified, the term image will be described as including the concepts of video and still images. In other words, the synchronized photography system 100 of this embodiment is capable of processing both still images and video.

The time server 101 has a function of distributing time information, and distributes time information synchronized with the Global Navigation Satellite System (GNSS) 107 to the sensor system 110. Alternatively, the time server 101 may distribute the local time held by the time server 101 as the master time to the sensor system 110.

The hub 102 is responsible for distributing, to each destination, a PTP (Precision Time Protocol) packet carrying time information distributed by the time server 101 and image data transmitted by the sensor system 110 to the image computing server 104 .
The control terminal 103 manages the operating state and controls parameter setting for each element of the synchronous imaging system 100 through a network. Here, the network may be GbE (Gigabit Ethernet), 10 GbE, or 100 GbE conforming to the IEEE standard, which is Ethernet (registered trademark). The network may also be configured by combining Infiniband, which is an interconnect, industrial Ethernet, and the like. Furthermore, the network is not limited to these, and may be other types of networks.

In FIG. 1, all 26 sets of sensor systems 110a to 110z are shown connected in a daisy chain, but the connection form in the synchronized imaging system 100 is not limited to this. For example, multiple sensor systems 110 may be divided into several groups, and the sensor systems 110 may be daisy chain connected in units of the divided groups.

This type of configuration is particularly effective in stadiums and the like. For example, a stadium may be configured with multiple floors, with a daisy chain of sensor systems 110 installed on each floor. In this case, input to the image computing server 104 is possible for each floor or for each half circumference of the stadium. This simplifies installation and makes the system more flexible, even in places where it is difficult to wire all of the sensor systems 110 together in a single daisy chain.

By connecting the sensor systems 110 to each other in a daisy chain, it is possible to reduce the number of connecting cables and save on wiring work when the volume of image data increases due to the increase in resolution of captured images to 4K, 8K, etc. and the increase in frame rate.
Each sensor system 110 has a camera 111 and a camera adapter 112. That is, the synchronous photography system 100 includes a plurality of cameras 111 (e.g., 26 cameras) for photographing a subject from a plurality of directions. Note that the cameras 111 included in the synchronous photography system 100 may have different performance or models.

Furthermore, each sensor system 110 is not limited to a configuration having a camera 111 and a camera adapter 112, and may include, for example, an audio device such as a microphone, a camera head for controlling the orientation of the camera, etc. Also, for example, the sensor system 110 may be composed of one camera adapter 112 and multiple cameras 111, or one camera 111 and multiple camera adapters 112. In other words, the multiple cameras 111 and multiple camera adapters 112 in the synchronized shooting system 100 correspond to each other in an N to M ratio (N and M are both integers greater than or equal to 1).

1 shows only one connection between the camera 111 and the camera adapter 112, the number of connections is not limited to this. There are cases where multiple connections are provided, such as a signal line for controlling the camera 111 and a signal line for transmitting data captured by the camera 111.
The camera 111 and the camera adapter 112 may be integrated. Furthermore, at least a part of the functions of the camera adapter 112 may be included in the image computing server 104. In the following, an example will be described in which each sensor system 110 includes one camera 111 and one camera adapter 112 of the same type.

<Generation of virtual viewpoint images>
The multiple cameras 111 included in the synchronous imaging system 100 capture images of a subject from multiple directions. The camera adapter 112 of each sensor system 110 is connected to the camera 111 and controls the camera 111, acquires captured images, provides a synchronization signal, sets the time, etc. The camera adapter 112 also performs image processing on the image data captured by the camera 111.
Specifically, the camera adaptor 112 outputs a foreground image and a background image from the captured image data. In the synchronous shooting system 100, a virtual viewpoint image is generated based on foreground images and background images captured from a plurality of viewpoints. Note that there may be a camera adaptor 112 that outputs a foreground image separated from a captured image but does not output a background image.

The image processed data in each sensor system 110 is transmitted to the image computing server 104 via the hub 102 daisy-chained to the sensor systems 110 .
The image computing server 104 processes images captured by each sensor system 110 and transmitted. First, the image computing server 104 reconstructs an image from the transmitted image packet. To reconstruct the image, the camera adapter ID, image type, frame number, and the like contained in the header information of the image packet are used. The image computing server 104 also stores the image in association with this information.

The image computing server 104 receives a viewpoint specification from the user terminal 105, reads out a corresponding image from stored information based on the specified viewpoint, and performs rendering processing to generate a virtual viewpoint image. Note that at least some of the functions of the image computing server 104 may be possessed by the control terminal 103, the sensor system 110, or the user terminal 105.

The virtual viewpoint image generated by the rendering process is transmitted from the image computing server 104 to the user terminal 105 and displayed on the display screen of the user terminal 105. A user operating the user terminal 105 can view an image from a viewpoint according to a specification.
That is, the image computing server 104 generates virtual viewpoint content based on images (multiple viewpoint images) captured by the multiple cameras 111 and viewpoint information.

In this embodiment, the virtual viewpoint content is generated by the image computing server 104 , but the virtual viewpoint content may be generated by the control terminal 103 or the user terminal 105 .
When taking pictures using multiple cameras 111, synchronized shooting is required. For this reason, each camera adapter 112 operates as a PTP slave and synchronizes with the time server 101, which is a GM (Grand Master) device. The camera adapter 112 also operates as a PTP master and as a BC (Boundary Clock) device that synchronizes with other sensor systems 110. Alternatively, the camera adapter 112 may operate as a TC (Transparent Clock) device that performs time correction on PTP packets from the GM device and transfers them to other sensor systems 110.

Alternatively, the camera adapter 112 may operate as an OC (Ordinary Clock) device that performs only time synchronization using PTP packets from a GM device or PTP packets transferred from another sensor system 110 .
Alternatively, the camera adapter 112 may perform time synchronization using a PTP packet from a GM device or a PTP packet transferred from another sensor system 110, and may perform time correction on the PTP packet and transfer it to the other sensor system 110. In other words, the camera adapter 112 may operate as both a TC device and an OC device.

The camera adapter 112 uses a time and a reference signal synchronized with the GM device or another sensor system 110 to provide the camera 111 with shooting timing (control clock). As a result, synchronized shooting by multiple cameras 111 becomes possible.

<Network control of camera adapter>
Next, network control by the camera adapter 112 will be described.
FIG. 2 is a block diagram showing the network control function in the camera adapter 112. As shown in FIG.

2 is an example, and multiple functional blocks may be combined into one functional block, or any functional block may be divided into blocks performing multiple functions. Furthermore, at least one of the functional blocks may be implemented as hardware.
When implemented by hardware, for example, a specific compiler may be used to automatically generate a dedicated circuit on an FPGA from a program for implementing each step. FPGA is an abbreviation for Field Programmable Gate Array. Also, a gate array circuit may be formed in the same manner as an FPGA and implemented as hardware, or may be implemented by an ASIC (Application Specific Integrated Circuit).

The camera adapter 112 has the following functional blocks: a CPU 200, a video packet processing unit 201, a first memory unit 202, a second memory unit 203, a DMA control unit 204, a PHY/MAC unit 205, a system bus 206, and a packet analysis processing unit 210. The functional blocks 200-204 and 210 within the camera adapter 112 are interconnected via the system bus 206. The camera adapter 112 is also connected to the camera 110 by an image transmission cable 209.

The CPU 200 executes application software at the application layer defined by the OSI (Open Systems Interconnection) reference model, and also controls the entire camera adapter 112. The CPU 200 then stores data, programs, and the like required to execute the application software in the first memory unit 202.

If necessary, the CPU 200 performs reading and writing via a system bus 206. Specifically, the CPU 200 performs PTP stack processing, and also generates a descriptor for DMA transfer control using a DMA control unit 204.
Video packet processing unit 201 receives image data captured by camera 110 via image transmission cable 209 and packetizes the image data. Video packet processing unit 201 also generates descriptors for DMA transfer control using DMA control unit 204. Although video packet processing unit 201 is described here as dedicated hardware, the functions of video packet processing unit 201 may be processed by a CPU different from CPU 200 that has similar functions.

The first memory unit 202 is a memory area used by the CPU 200. The first memory unit 202 is composed of semiconductor memory such as DRAM and SRAM. DRAM stands for Dynamic Random Access Memory, and SRAM stands for Static Random Access Memory. The internal area of the first memory unit 202 is divided into an area for storing programs and an application buffer area used by application software. The internal area of the first memory unit 202 stores, for example, descriptors for transferring PTP packets.

The descriptor here is information used by the DMA control unit 204 (described later) for DMA transfer of packets, and includes address information on the source side where the transfer is performed, address information on the destination side, information on the transfer size, and the like.
The second storage unit 203 is a storage area used by the video packet processing unit 201. The second storage unit 203 is also composed of a semiconductor memory such as a DRAM or an SRAM. For example, a descriptor for transferring an image packet is stored in the internal area of the second storage unit 203.

2, the first memory unit 202 and the second memory unit 203 are respectively occupied by the CPU 200 and the video packet processing unit 201, but the configuration of the first memory unit 202 and the second memory unit 203 is not limited to this. For example, the CPU 200 and the video packet processing unit 201 may share and use a memory unit in which the first memory unit 202 and the second memory unit 203 are combined into one.

The PHY/MAC unit 205 is an interface with a network 208, typically a LAN (Local Area Network), and is responsible for controlling communication at the PHY layer (physical layer) and MAC layer (data link layer). The PHY/MAC unit 205 transmits and receives communication packets.

The PHY/MAC unit 205 outputs a timestamp indicating the reception time for every communication packet it receives. Here, the timestamp refers to time information indicating the time when the PHY/MAC unit 205 transmitted or received a packet. In the following, a timestamp indicating the reception time is referred to as a reception timestamp, and a timestamp indicating the transmission time is referred to as a transmission timestamp. The timestamp is used for calculations such as time correction in PTP protocol processing.

The DMA control unit 204 executes DMA transfers in response to instructions from the CPU 200 and the video packet processing unit 201. That is, the DMA control unit 204 transfers received packets received by the PHY/MAC unit 205 to a specified storage unit, and transfers transmitted packets sent from the PHY/MAC unit 205 from the specified storage unit to the PHY/MAC unit 205.

The DMA transfer is performed according to the contents of the descriptors generated by the CPU 200 and the video packet processing unit 201, respectively. The DMA control unit 204 then selects the descriptor to be used for the DMA transfer in response to instructions from an analysis unit 212 within the packet analysis processing unit 210, which will be described later. After the DMA transfer is completed, the DMA control unit 204 notifies the CPU 200 or the video packet processing unit 201 of a completion interrupt 207 corresponding to the descriptor.

The DMA control unit 204 discards a received packet if a descriptor for DMA transfer is not set, for example, due to an increase in the processing load in the CPU 200 or the video packet processing unit 201. The DMA control unit 204 also discards a received packet if the packet length of the received packet is smaller than 64 bytes, which is the minimum length specified in IEEE 802.3. The DMA control unit 204 then outputs a discard notification including the type of the discarded packet and a packet identifier that can uniquely identify the discarded packet to the timestamp deletion processing unit 217, which will be described later.

The packet identifier may be information typically included in a packet that can identify the packet, such as a sequence number stored in the header area of the packet. Alternatively, the packet identifier may be a unique identifier defined by a user of the synchronized imaging system 100 to uniquely associate a received packet with a reception timestamp. The unique identifier may be written, for example, in the header area of the packet, or may be added to the payload area or end. Alternatively, the unique identifier may be information that is used temporarily only within one sensor system 110.

When discarding a received packet, the DMA control unit 204 notifies the timestamp deletion processing unit 217 of the type and packet identifier of the packet received from the analysis unit 212, which will be described later. Alternatively, the DMA control unit 204 itself may perform packet analysis and notify the result.
Next, the packet analysis processing unit 210 will be described.
The packet analysis processing unit 210 includes a timestamp temporary storage unit 211 , an analysis unit 212 , an analysis result storage unit 213 , a timestamp allocation processing unit 214 , a timestamp storage unit 215 , a register unit 216 , and a timestamp deletion processing unit 217 .

The time stamp temporary storage unit 211 is a storage unit that temporarily stores the reception time stamp output by the PHY/MAC unit 205, and is made up of a FIFO (First Input First Output) memory.
The analysis unit 212 analyzes the MAC header of the received packet received by the PHY/MAC unit 205, and determines the type of the packet from the EtherType of the received packet. The analysis unit 212 then analyzes the header area, payload area, or tail of the received packet to determine the packet identifier.

The analysis unit 212 notifies the DMA control unit 204 and the analysis result storage unit 213 of the type of packet and the packet identifier, which are the analysis results. Note that the analysis results by the analysis unit 212 may be notified directly to the time stamp allocation processing unit 214 without being stored in the analysis result storage unit 213.
The analysis result storage unit 213 is made up of a FIFO memory, and stores the results of the analysis of the received packet by the analysis unit 212 .

The timestamp allocation processing unit 214 allocates some of the reception timestamps stored in the timestamp temporary storage unit 211 according to the type of packet of the analysis result stored in the analysis result storage unit 213 to the timestamp storage unit 215. The timestamp allocation processing unit 214 allocates the reception timestamps to the timestamp storage unit 215 at least for PTP packets.

The CPU 200 executes time synchronization processing using the PTP packet transferred to the first storage unit 202 and a reception timestamp corresponding to the PTP packet. The CPU 200 also executes time synchronization processing upon receiving a completion interrupt 207 corresponding to a notification of transfer completion from the DMA control unit 204.
Among the analysis results stored in the analysis result storage unit 213, the packet identifiers are also sorted together with the reception timestamps into the timestamp storage unit 215. Then, the timestamp sorting processor 214 discards the reception timestamps and the analysis results for received packets whose analysis result type is not subject to sorting.

The timestamp storage unit 215 is made up of multiple FIFO memories, and stores the reception timestamps and packet identifiers assigned by the timestamp allocation processing unit 214. The FIFO memories are capable of pre-reading data without the need to execute a read process. The reception timestamps and packet identifiers are stored in one of the multiple FIFO memories in the timestamp storage unit 215 that corresponds to the type of packet that is the analysis result. In other words, the FIFO memories store the reception timestamps and packet identifiers separately for each type of packet.

The register unit 216 stores the correspondence between the packet type and the descriptor of the analysis result, and the type that is not subject to distribution by the received timestamp distribution processing unit 214. The register unit 216 also stores the correspondence between the packet type of the analysis result and the FIFO memory of the timestamp storage unit 215 in which it is stored.

The information stored in the register unit 216 is information for determining which descriptor the DMA control unit 204 uses for the received packet and where the timestamp allocation processing unit 214 allocates the received timestamp. The information stored in the register unit 216 is set by the CPU 200 before the start of the DMA transfer.

The timestamp deletion processing unit 217 receives and holds the packet type and packet identifier output from the DMA control unit 204 when a received packet is discarded. The timestamp deletion processing unit 217 also monitors the FIFO memory of the timestamp storage unit 215 that corresponds to the packet type. Furthermore, the timestamp deletion processing unit 217 compares the packet identifier stored in the timestamp storage unit 215 with the packet identifier held by the timestamp deletion processing unit 217.

If the packet identifiers match, the timestamp deletion processing unit 217 deletes the reception timestamp and packet identifier stored in the timestamp storage unit 215. In other words, if a received packet is discarded, the timestamp deletion processing unit 217 deletes from the timestamp storage unit 215 the reception timestamp that was stored in association with the packet identifier of the discarded received packet.

The deletion process by the timestamp deletion processing unit 217 is executed before the CPU 200 or the video packet processing unit 217 accesses the timestamp storage unit 215. Therefore, remaining received timestamps are avoided during time synchronization processing, enabling normal time synchronization processing.

<Time synchronization processing sequence>
Next, the sequence of the time synchronization process will be described.
FIG. 3 is a diagram showing a sequence of the time synchronization process.
3 shows a sequence of transmission and reception of PTP packets between the time server 101, which corresponds to a GM device, and the sensor system 110, which corresponds to a PTP slave device. In the sequence described below, a Sync packet, a Follow_Up packet, a Delay_Request packet, and a Delay_Response packet are used as PTP packets.

In S301, the time server 101 transmits a Sync packet to the sensor system 110. The sensor system 110 holds a reception timestamp t2 indicating the time when the Sync packet was received. The time server 101 repeatedly transmits the Sync packet at regular intervals. Specifically, the reception timestamp t2 is held in the sensor system 110 by the packet analysis processing unit 210 storing the reception timestamp t2 in a specified timestamp storage unit 215 when the Sync packet is received. The held reception timestamp t2 is then used by the CPU 200 for time synchronization processing.

In S302, the time server 101 transmits a transmission timestamp t1, which indicates the transmission time of the Sync packet, to the sensor system 110 in a Follow_Up packet. The sensor system 110 retains the transmission timestamp t1 by receiving the Follow_Up packet.

In the example of FIG. 3, the transmission timestamp of the Sync packet is carried on the Follow_Up packet and transmitted, but the transmission timestamp of the Sync packet may be carried on the Sync packet itself and transmitted instead of on the Follow_Up packet.
In S303, the sensor system 110 transmits a Delay_Request packet to the time server 101. At this time, the sensor system 110 holds a transmission timestamp t3 of the Delay_Request packet.

In S304, the time server 101 transmits a Delay_Response packet to the sensor system 110. The time server 101 also transmits the reception timestamp t4 of the Delay_Request packet in a Delay_Response packet to the sensor system 110. By receiving the Delay_Response packet, the sensor system 110 holds the reception timestamp t4 indicating the time when the time server 101 received the Delay_Request packet.

The sensor system 110 calculates the average transmission delay time from the exchange of PTP packets in S301 to S304, and corrects the time in the sensor system 110. The average transmission delay time is calculated by the following formula.
Average transmission delay time = {(t2 - t1) + (t4 - t3)}/2

<Processing when receiving packets>
Next, the process when a packet is received will be described.
FIG. 4 is a flowchart showing the process when a packet is received.

The process in FIG. 4 starts when the sensor system 110 is started (S401).
In S402, the CPU 200 sets the settings in the register unit 216. Specifically, the CPU 200 sets the correspondence between the packet type and the descriptor, and the packet type that is not subject to distribution by the reception timestamp distribution processing unit 214. Furthermore, the CPU 200 also sets the correspondence between the packet type and the FIFO memory of the timestamp storage unit 215 in which the reception timestamp is stored. Note that, in the example of Fig. 4, the CPU 200 sets the settings in the register unit 216 after the sensor system 110 is started up, but this setting may be set as an initial setting when the sensor system 110 is started up.

When the setting of the register unit 216 is completed, the process proceeds to S403, and the PHY/MAC unit 205 enters a standby state until it receives a packet (S403: No). Then, if the PHY/MAC unit 205 receives a packet (S403: Yes), the process proceeds to S404 and S405.

In S 404 , the reception timestamp output by the PHY/MAC unit 205 is temporarily stored in the timestamp temporary storage unit 211 .
In parallel with S404, in S405, the analysis unit 212 analyzes the received packet. The analysis unit 212 reads the type field of the MAC header and determines the type of the packet from the EtherType of the received packet. For example, if the EtherType is 0x0800, it is determined to be an IPv4 packet, and if it is 0x88F7, it is determined to be a PTP packet. Furthermore, the analysis unit 212 determines the packet identifier by analyzing the header area, payload area, and tail of the received packet.

Specifically, the analysis unit 212 analyzes a sequence ID included in the header area of a PTP packet, a sequence number included in the TCP header area of a TCP/IP packet, etc. Alternatively, the analysis unit 212 may analyze a sequence number in a unique format created by the user that is added to the header area, payload area, or end of a packet.

When the analysis unit 212 completes the analysis of the received packet, the process proceeds to S406 and S407.
In S406, the type of packet and the packet identifier obtained as the analysis result by the analysis unit 212 are stored in the analysis result storage unit 213. Then, when both the processing of S404 and the processing of S406 are completed, the process proceeds to S408.

In S408, the reception timestamps temporarily stored in the timestamp temporary storage unit 211 in S404 are distributed by the timestamp distribution processing unit 214 according to the type of packet stored in the analysis result storage unit 213 in S406. The details of the distribution process performed by the timestamp distribution processing unit 214 will be described later.

In parallel with S406, in S407, the analysis unit 212 notifies the DMA control unit 204 of the packet type and packet identifier determined in S405. The DMA control unit 204 can know the descriptor to be used for the transfer from this notification. That is, the correspondence between the determined packet type and the descriptor to be used is set in advance in the register unit 216.

When notification to the DMA control unit 204 is complete, the process proceeds to S409, where it is determined whether the DMA control unit 204 discarded the packet received from the PHY/MAC unit 205. If a descriptor for DMA transfer to the first memory unit 202 or the second memory unit 203 has not been set, the DMA control unit 204 discards the received packet because it does not know the transfer destination of the packet. Alternatively, if the packet length of the received packet is shorter than 64 bytes, which is the minimum length specified by IEEE, the DMA control unit 204 discards the received packet as an invalid packet.

If the DMA control unit 204 does not discard the received packet (S409: No), in S410 the DMA control unit 204 selects a descriptor in accordance with the information notified by the analysis unit 212 in S407, and transfers the received packet to a predetermined storage unit.
On the other hand, if the DMA control unit 204 discards the received packet (S409: Yes), in S411 the timestamp deletion processing unit 217 receives a discard notification from the DMA control unit 204 and deletes the received timestamp from the timestamp storage unit 215. The deleted received timestamp is the received timestamp stored in association with the packet identifier of the discarded received packet.

After the sorting process in S408, the DMA transfer in S410, and the deletion of the reception timestamp in S411, the process proceeds to S412, where it is determined whether or not to continue the series of processing flows from S403 to S411. If a factor that ends communication with the external device occurs (S412: No), the process in FIG. 4 ends in S413. If no such factor occurs (S412: Yes), the process returns to S403.

<Flow of sorting process>
Next, the allocation process executed by the time stamp allocation processor 214 in S408 of FIG. 4 will be described.

FIG. 5 is a flowchart showing the allocation process of received time stamps.
The sorting process in FIG. 5 starts when the reception timestamp is stored in the timestamp temporary storage unit 211 in S404 in FIG. 4, and the packet type and packet identifier are stored in the analysis result storage unit 213 in S406 (S501).

In S502, it is determined whether the type of packet stored in the analysis result storage unit 213 is a video packet. If it is a video packet (S502: Yes), the process proceeds to S503, where it is determined whether to discard the reception timestamp of the video packet. The timestamp allocation processing unit 214 determines whether to discard or store the reception timestamp of the video packet in the timestamp storage unit 215, according to the settings that the CPU 200 has set in advance in the register unit 216.

In S503, if it is determined that the reception timestamp of the video packet is to be discarded (S503: Yes), the process proceeds to S504, where the reception timestamp of the video packet is discarded. After the discarding is completed, the process proceeds to S513, where the process of Fig. 5 ends, and the process proceeds to S412 of Fig. 4.
As a specific method of discarding the received timestamp, for example, the timestamp allocation processor 214 may write the received timestamp in a timestamp storage section for discarding in the timestamp storage section 215. Alternatively, the timestamp allocation processor 214 may write the received timestamp in a discard area separate from the timestamp storage section 215, or may read and discard the temporarily stored received timestamp.

If it is determined in S503 that the reception timestamp of the video packet is to be stored in the timestamp storage unit 215 (S503: No), the process proceeds to S505, where the reception timestamp and packet identifier of the video packet are stored in the timestamp storage unit 215. After storage is complete, the process proceeds to S513, where the process in FIG. 5 ends.

Here, a specific example of storing the reception timestamp in the timestamp storage unit 215 will be described.
For example, if three FIFO memories are provided in the time stamp storage unit 215, the reception time stamp and packet identifier of, for example, a PTP packet are allocated to the first FIFO memory, and the reception time stamp and packet identifier of, for example, a video packet are allocated to the second FIFO memory, and the reception time stamp and packet identifier of a packet other than the PTP packet and the video packet are allocated to the third FIFO memory.
The correspondence between the packet type and the storage destination for storing the reception timestamp is not limited to this. The correspondence between the packet type and the storage destination for storing the reception timestamp is set in the register unit 216 by the CPU 200 in advance.

In addition, in the above example, the reception timestamp and the packet identifier are allocated to the same FIFO memory in the timestamp storage unit 215, but as long as the reception timestamp and the packet identifier are linked, they may be allocated and stored in different areas.
If it is determined in S502 that the type of the packet stored in the analysis result storage unit 213 is not a video packet (S502: No), the process proceeds to S506, where it is determined whether or not the type of the packet is a PTP packet.

If it is a PTP packet (S506: Yes), the process proceeds to S507, where the timestamp allocation processing unit 214 determines whether to discard the reception timestamp of the PTP packet. That is, according to the setting that the CPU 200 has set in advance in the register unit 216, it is determined whether the reception timestamp of the PTP packet is to be discarded or stored in the timestamp storage unit 215. Here, since it is assumed that the PTP packet is used for time synchronization processing, the setting set in the register unit 216 indicates storage in the timestamp storage unit 215.

In S507, if it is determined that the reception timestamp of the PTP packet is to be discarded (S507: Yes), the process proceeds to S508, where the reception timestamp of the PTP packet is discarded. After the discarding is completed, the process proceeds to S513, where the process of FIG. 5 ends, and the process proceeds to S412 of FIG.
In S507, if it is determined that the reception timestamp of the PTP packet is to be stored in the timestamp storage unit 215 (S507: No), the process proceeds to S509, where the reception timestamp and packet identifier of the PTP packet are stored in the timestamp storage unit 215. After the storage is completed, the process proceeds to S513, where the process in FIG. 5 ends.

In S506, if it is determined that the type of the packet stored in the analysis result storage unit 213 is not a PTP packet (S506: No), the process proceeds to S510.
In S510, the time stamp allocation processing unit 214 determines whether or not to discard the reception time stamps of other received packets other than the video packets and the PTP packets. That is, according to the setting that the CPU 200 has set in advance in the register unit 216, it is determined whether to discard the reception time stamps of the other received packets or to store them in the time stamp storage unit 215.

If it is determined in S510 that the reception timestamp of a received packet other than the video packet and the PTP packet is to be discarded (S510: Yes), the process proceeds to S511, where the reception timestamp is discarded. After the discarding is complete, the process proceeds to S513, where the process in FIG. 5 ends, and the process proceeds to S412 in FIG. 4.

If it is determined in S510 that the reception timestamp of a received packet other than the video packet and the PTP packet is to be stored in the timestamp storage unit 215 (S510: No), the process proceeds to S512. In S512, the reception timestamp and the packet identifier are stored in the timestamp storage unit 215. After the storage is completed, the process proceeds to S513, and the process in FIG. 5 ends.

<Transfer process flow>
Next, the transfer process executed by the DMA control unit 204 in S410 of FIG. 4 will be described.
FIG. 6 is a flowchart showing a transfer process of a received packet.
The transfer process shown in FIG. 6 is started when it is determined in S409 in FIG. 4 that the received packet is not to be discarded (S601).
In S602, it is determined whether or not the type of the packet is a video packet based on the analysis result of the analysis unit 212. If the type of the packet is a video packet (S602: Yes), this corresponds to the DMA control unit 204 receiving a notification from the analysis unit 212 that a video packet descriptor is to be used, and the process proceeds to S603.

In S603, a video packet descriptor is selected, and the packet is DMA-transferred to the second memory unit 203 that corresponds to the video packet descriptor, out of the first memory unit 202 and the second memory unit 203. When the DMA transfer starts, the process proceeds to S607, the process in FIG. 6 ends, and the process proceeds to S412 in FIG. 4.

If it is determined in S602 that the type of the packet is not a video packet (S602: No), the process proceeds to S604, where it is determined whether or not the type of the packet is a PTP packet.
If the type of packet is a PTP packet (S604: Yes), this corresponds to the DMA control unit 204 receiving a notification from the analysis unit 212 that a PTP packet descriptor is to be used, and the process proceeds to S605.

In S605, a PTP packet descriptor is selected, and the packet is DMA-transferred to the first memory unit 202 corresponding to the PTP packet descriptor out of the first memory unit 202 and the second memory unit 203. When the DMA transfer is started, the process proceeds to S607, where the process in FIG. 6 ends.
If it is determined in S604 that the type of packet is not a PTP packet (S604: No), this corresponds to the DMA control unit 204 receiving a notification from the analysis unit 212 to use a descriptor for a received packet other than a video packet and a PTP packet.

In this case, the process proceeds to S606, where another received packet descriptor is selected, and the packet is DMA-transferred to the memory unit corresponding to the other received packet descriptor among the first memory unit 202 and the second memory unit 203. When the DMA transfer is started, the process proceeds to S607, where the process in FIG. 6 ends.

<Deletion process flow>
Next, the deletion process executed by the time stamp deletion processing unit 217 in S411 of FIG. 4 will be described.

FIG. 7 is a flowchart showing the process of deleting a received time stamp.
The deletion process in FIG. 7 is started when it is determined in S409 in FIG. 4 that the DMA control unit 204 has discarded the received packet (S701).
In S702, the process waits (S702: No) until a discard notification including the type and packet identifier of the discarded received packet is received from the DMA control unit 204. Then, when the discard notification is received (S702: Yes), the process proceeds to S703, where the type and packet identifier of the received packet are held inside the timestamp deletion processing unit 217.
As an example, the time stamp deletion processing unit 217 has a FIFO memory for each type of packet, and stores the packet identifier in the corresponding FIFO memory. Note that the time stamp deletion processing unit 217 may not have a FIFO memory corresponding to the packet type, and may store both the packet type and the packet identifier in association with each other.

In S704, it is determined whether the packet identifier stored in the time stamp storage unit 215 matches the packet identifier held in the time stamp deletion processing unit 217.
Here, a specific example will be given of a match check between the packet identifier stored in the time stamp storage unit 215 and the packet identifier held in the time stamp deletion processing unit 217. For example, it is assumed that the time stamp storage unit 215 has FIFO memories prepared for each type of received packet, for example, three types of packets: PTP packets, video packets, and packets other than PTP packets and video packets.

The timestamp deletion processing unit 217 checks the timestamp storage unit 215 for each type of packet. The timestamp storage unit 215 can pre-read the stored packet identifier without the need to execute a read process. The timestamp deletion processing unit 217 then performs a match check between the pre-read packet identifier and the held packet identifier, and if there is a mismatch (S704: No), performs a match check for the next type.

If the two packet identifiers match in S704 (S704: Yes), the process proceeds to S705, where the reception timestamp and packet identifier stored in the timestamp storage unit 215 are deleted. After the deletion, the process proceeds to S706, where the deletion process in FIG. 7 ends, and the process proceeds to S412 in FIG. 4.

<Time synchronization process flow>
Next, a time synchronization process in which the CPU 200 synchronizes the time in the sensor system 110 with the time of the time server 101 will be described.

FIG. 8 is a flowchart showing the time synchronization process.
The time synchronization process in FIG. 8 is started after the time server 101 is determined as the PTP master and the sensor system 110 is determined as the PTP slave in the synchronous photography system 100 (S801).

In S802, the sensor system 110 waits (S802: No) until it receives a Sync packet and a Follow_Up packet from the time server 101. If the Sync packet and the Follow_Up packet are received (S802: Yes), the process proceeds to S803 and S804.

In S803, the timestamps of the received Sync packet and Follow_Up packet are obtained.
Specifically, for the Sync packet received by the PHY/MAC 205 in S802, the packet analysis processing unit 210 stores a reception timestamp t2 in the timestamp storage unit 215. Furthermore, when the DMA transfer of the Sync packet by the DMA control unit 204 is completed, a completion interrupt 207 is notified. Then, upon receiving the completion interrupt 207, the CPU 200 acquires the reception timestamp t2 stored in the timestamp storage unit 215 in S803.

When a Sync packet is discarded by the DMA control unit 204, the reception timestamp t2 is stored in the timestamp storage unit 215, but the completion interrupt 207 is not notified to the CPU 200. Therefore, in the deletion process of FIG. 7, the timestamp deletion processing unit 217 receives the discard notification and deletes the reception timestamp t2 stored in the timestamp storage unit 215, thereby avoiding the remaining reception timestamp t2.

As shown in FIG. 3, the time server 101 transmits a Follow_Up packet containing the transmission timestamp t1 of the Sync packet. When the PHY/MAC 205 receives the Follow_Up packet in S802, the Follow_Up packet is DMA-transferred to the first memory unit 202 by the DMA control unit 204. In addition, the CPU 200, which has received the completion interrupt 207, obtains the transmission timestamp t1 from the received Follow_Up packet in S803. When the acquisition of the timestamp information t1 and t2 is completed, the process proceeds to S808.

In parallel with S803, in S804, the CPU 200 generates a Delay_Request packet and transmits it to the time server 101. During transmission, the CPU 200 acquires transmission timestamp information t3 indicating the transmission time of the Delay_Request packet. When transmission of the Delay_Request packet is completed, the process proceeds to S805, where it is determined whether the CPU 200 has received a Delay_Response packet from the time server 101.

If the Delay_Response packet is not received (S805: No), the process proceeds to S806, where reception is awaited until a certain period of time has elapsed (S806: No).If the certain period of time has elapsed without reception (S806: Yes), the process returns to S802.
If a Delay_Response packet has been received in S805 (S805: Yes), the process proceeds to S807, where the CPU 200 acquires a reception timestamp t4 from the Delay_Response packet.

Specifically, the Delay_Response packet received by the PHY/MAC 205 is DMA-transferred to the first memory unit 202 by the DMA control unit 204. The Delay_Response packet includes a reception timestamp t4 indicating the time when the Delay_Request packet was received by the time server 101. The CPU 200 then obtains the reception timestamp t4 of the Delay_Request packet from the Delay_Response packet.

When the CPU 200 acquires the reception timestamp t4, the process proceeds to S808, where the acquired timestamps t1 to t4 are used to calculate the time difference between the time of the time server 101 and the time of the sensor system 110, and the amount of correction is determined. Once the amount of correction has been determined, the process proceeds to S809, where the time of the sensor system 110 is corrected according to the determined amount of correction.
When the time correction in S809 is completed, the time of the sensor system 110 is synchronized with the time of the time server 101. Since the remaining received timestamp t2 in the timestamp storage unit 215 is avoided by executing the deletion process in Fig. 7, an accurate correction amount is determined in S808, and the time is correctly corrected in S809.

Note that if the amount of time correction performed at one time is large, there is a risk of it destabilizing the operation of the entire sensor system 110. Therefore, the time may be corrected gradually over a certain period of time, or the amount of time correction performed at one time may be limited.
After the time correction, the process proceeds to S810, where it is determined whether the sensor system 110 continues to operate. For example, when the control terminal 103 stops the synchronization process for all of the sensor systems 110a to 110z, or when the control terminal 103 stops the synchronization process for a specified one of the sensor systems 110a to 110z, the operation is stopped (S810: No). When the operation is stopped (S810: No), the synchronization process in FIG. 8 ends (S811), and when the operation is to continue (S810: Yes), the process returns to S802.

<Other embodiments>
The present invention can be embodied as, for example, a system, an apparatus, a method, a program, or a recording medium (storage medium), etc. Specifically, the present invention may be applied to a system composed of multiple devices (for example, a host computer, an interface device, a Web application, etc.), or may be applied to an apparatus composed of a single device.

The present invention can also be realized by supplying a program (computer program) that realizes one or more of the functions of the above-mentioned embodiments to a system or device via a network or recording medium (storage medium). One or more processors in the computer of the system or device read and execute the program. In this case, the program (program code) read from the recording medium itself realizes the functions of the embodiment. Furthermore, the recording medium on which the program is recorded can constitute the present invention.

In addition, not only can the functions of the embodiments be realized by the computer executing a program it has read, but the functions of the above-described embodiments can also be realized by an operating system (OS) running on the computer performing some or all of the actual processing based on instructions from the program.

Furthermore, the program read from the recording medium may be written into a memory provided on a function expansion card inserted into a computer or a function expansion unit connected to the computer, and then a CPU provided on the function expansion card or function expansion unit may perform some or all of the actual processing based on the instructions of the program, thereby realizing the functions of the above-mentioned embodiments.

When the present invention is applied to the above-mentioned recording medium, the recording medium stores a program corresponding to the flowchart described above.
The disclosure of the above embodiment includes the following configurations and methods.

(Configuration 1)
a receiving means for receiving a packet and outputting a timestamp indicating a time of reception;
A transfer means for transferring the received packet to a predetermined memory;
A storage means for storing the output time stamp;
a deletion means for deleting, when the packet is discarded, the time stamp corresponding to the discarded packet from the storage means;
A communication device comprising:

(Configuration 2)
2. The communication device according to claim 1, further comprising: a synchronization unit that executes a time synchronization process using the packet transferred to the memory and the time stamp corresponding to the packet.

(Configuration 3)
3. The communication device according to claim 2, wherein the synchronization means executes the time synchronization process upon receiving a notification of completion of transfer by the transfer means.

(Configuration 4)
4. The communication device according to claim 2, wherein the deletion means deletes the time stamp before the synchronization means executes the time synchronization process if the packet is discarded.

(Configuration 5)
the storage means stores the time stamp in association with an identifier for identifying the packet;
5. The communication device according to claim 1, wherein the deletion means deletes the timestamp stored in association with the identifier of the discarded packet.

(Configuration 6)
When the forwarding means discards the packet, the forwarding means outputs an identifier for identifying the discarded packet;
6. The communication device according to claim 5, wherein the deletion means deletes the time stamp stored in the storage means in association with an identifier that matches the identifier output by the transfer means.

(Configuration 7)
7. The communication device according to claim 6, wherein the deletion means monitors the identifier corresponding to the time stamp stored in the storage means and compares the identifier with the identifier output by the transfer means.

(Configuration 8)
the identifier is information included in the packet,
8. The communication device according to any one of configurations 5 to 7, further comprising an analyzing means for analyzing the identifier from the received packet.

(Configuration 9)
9. The communication device according to claim 8, wherein the analyzing means analyzes the identifier in any one of a header area, a payload area, and an end area of the packet.

(Configuration 10)
The method further includes: determining a type of the packet;
10. The communication device according to any one of configurations 1 to 9, wherein the storage means stores a corresponding time stamp for at least a type of packet used in time synchronization processing among the packets.

(Configuration 11)
11. The communication device according to configuration 10, wherein the storage means stores the time stamps by classifying them according to their types.

(Configuration 12)
12. The communication device according to any one of configurations 1 to 11, wherein the storage means is a FIFO memory capable of pre-reading data without the need to execute a read process.

(Configuration 13)
13. The communication device according to any one of configurations 1 to 12, wherein the transfer means discards the received packet if a descriptor for transferring the packet to the memory is not set.

(Configuration 14)
14. The communication device according to any one of configurations 1 to 13, wherein the transfer means discards the received packet if the length of the received packet is shorter than a minimum length defined in IEEE 802.3.

(Configuration 15)
A plurality of communication devices according to any one of configurations 1 to 14, which are daisy-chained via a communication line;
a server device that communicates the packets with each of the plurality of communication devices;
A communication system comprising:

(Method 1)
a receiving step of receiving a packet and outputting a timestamp indicating a time of reception;
a transfer step of transferring the received packet to a memory;
a storage step of storing the output time stamp in a storage means;
a deletion step of deleting the timestamp corresponding to the discarded packet from the storage means when the packet is discarded in the transfer step;
A communication method comprising:

(Configuration 16)
A program for causing a computer to function as each of the means of the communication device according to any one of configurations 1 to 14.

100...Synchronized shooting system, 101...Time server, 102...Hub, 103...Control terminal, 104...Image computing server, 105...User terminal, 110a-110z...Sensor system, 107...GNSS, 111...Camera, 112...Camera adapter, 200...CPU, 201...Video packet processing unit, 202...First memory unit, 203...Second memory unit, 204...DMA control unit, 205...PHY/MAC unit, 206...System bus, 207...Completion interrupt, 208...Network, 209...Image transmission cable, 210...Packet analysis processing unit, 211...Temporary timestamp storage unit, 212...Analysis unit, 213...Analysis result storage unit, 214...Timestamp allocation processing unit, 215...Timestamp storage unit, 216...Register unit, 217...Timestamp deletion processing unit

Claims (17)

a receiving means for receiving a packet and outputting a timestamp indicating a time of reception;
A transfer means for transferring the received packet to a predetermined memory;
A storage means for storing the output time stamp;
a deletion means for deleting, when the packet is discarded, the time stamp corresponding to the discarded packet from the storage means;
A communication device comprising: The communication device according to claim 1, further comprising a synchronization means for performing time synchronization processing using the packet transferred to the memory and the timestamp corresponding to the packet. The communication device according to claim 2, characterized in that the synchronization means executes the time synchronization process upon receiving a notification of completion of transfer by the transfer means. The communication device according to claim 2, characterized in that, if the packet is discarded, the deletion means deletes the timestamp before the synchronization means executes the time synchronization process. the storage means stores the time stamp in association with an identifier for identifying the packet;
2. The communication device according to claim 1, wherein the deletion means deletes the timestamp stored in association with the identifier of the discarded packet. When the forwarding means discards the packet, the forwarding means outputs an identifier for identifying the discarded packet;
6. The communication device according to claim 5, wherein the deletion means deletes the time stamp stored in the storage means in association with an identifier that matches the identifier output by the transfer means. The communication device according to claim 6, characterized in that the deletion means monitors the identifier corresponding to the timestamp stored in the storage means and compares it with the identifier output by the transfer means. the identifier is information included in the packet,
6. The communication device according to claim 5, further comprising an analyzing unit for analyzing the identifier from the received packet. The communication device according to claim 8, characterized in that the analysis means analyzes the identifier in any one of a header area, a payload area, and an end area of the packet. The method further includes: determining a type of the packet;
2. The communication device according to claim 1, wherein the storage means stores a corresponding time stamp for at least a type of packet used in time synchronization processing among the packets. The communication device according to claim 10, characterized in that the storage means stores the timestamps by type. The communication device according to claim 1, characterized in that the storage means is a FIFO memory capable of pre-reading data without the need to execute a read process. The communication device according to claim 1, characterized in that the transfer means discards the received packet if a descriptor for transferring the packet to the memory is not set. The communication device according to claim 1, characterized in that the forwarding means discards the received packet if the length of the received packet is shorter than the minimum length specified in IEEE 802.3. A plurality of communication devices according to any one of claims 1 to 14, which are daisy-chained via a communication line;
a server device that communicates the packets with each of the plurality of communication devices;
A communication system comprising: a receiving step of receiving a packet and outputting a timestamp indicating a time of reception;
a transfer step of transferring the received packet to a memory;
a storage step of storing the output time stamp in a storage means;
a deletion step of deleting the timestamp corresponding to the discarded packet from the storage means when the packet is discarded in the transfer step;
A communication method comprising: A program for causing a computer to function as each of the means of a communication device according to any one of claims 1 to 14.

Images