WO2021132739A1

WO2021132739A1 - Multi-channel image transmission system and method

Info

Publication number: WO2021132739A1
Application number: PCT/KR2019/018317
Authority: WO
Inventors: 유재형; 김중구
Original assignee: (주)비전에스티
Priority date: 2019-12-23
Filing date: 2019-12-23
Publication date: 2021-07-01
Also published as: KR20210080753A

Abstract

The present invention relates to a multi-channel image transmission system and method, which receive a plurality of camera inputs so as to synchronize same and integrate same in real time so as to transmit a single image, thereby facilitating transmission of a large volume of images, and according to a multi-channel image transmission system and method of the present invention, if a plurality of cameras are synchronized and integrated in real time so as to be transmitted through USB 3.0 USB Video Class (UVC), a multi-channel camera can be easily applied to a PC or various embedded systems without a separate SDK or driver.

Description

Multi-channel video transmission system and method

The present invention relates to a multi-channel video transmission system and method, and more particularly, multi-channel video transmission capable of simplifying mass video transmission by receiving multiple camera inputs, synchronizing them, and integrating them in real time to transmit one video. It relates to systems and methods.

A camera is one of the core sensors of an autonomous driving platform, and the number of cameras required for autonomous driving is large and therefore a large amount of data must be transmitted in real time.

Connecting the camera to a PC usually requires the use of Gigabit ethernet or USB2.0/USB3.0 interfaces. There are various cameras on the market, but to connect and synchronize multiple cameras, there are several limitations due to the number of corresponding interface ports and synchronization methods, and each camera must be separately connected to the deep learning platform.

Connecting a camera to an embedded system is more difficult as it usually has fewer interfaces than a PC. Some systems, such as Nvidia jetson nano, Nvidia jetson TX2, Nvidia jetson xavier, Raspberry Pi 4 Model B, etc. support MIPI interface, but support few cameras, difficult to lengthen cable, and SDK (Software Development Kit) provided by the manufacturer. ) is subject to restrictions such as the use of

Therefore, the need for a multi-channel camera that can be easily connected to various embedded systems including PCs has increased.

In order to solve this problem, an object of the present invention is to provide a multi-channel image transmission system and method for applying a multi-channel camera to a PC or various embedded systems by synchronizing and transmitting a plurality of cameras in real time.

A multi-channel image transmission system according to the present invention includes: a camera unit including a plurality of cameras for photographing a subject; a camera interface for receiving and transmitting the image captured by the camera unit; an image synchronization integration and transmission unit for synchronizing and integrating images captured by the camera unit and transmitting; a video transmission interface for video transmission; and a host receiving the synchronized and integrated image.

In addition, the multi-channel image transmission method according to the present invention is characterized in that an image captured by a plurality of cameras is received through a camera interface, the images are synchronized and integrated, and the image is transmitted to a host through an image transmission interface.

According to the multi-channel video transmission system according to the present invention, if a plurality of cameras are synchronized and integrated in real time and transmitted as USB 3.0 UVC (USB Video Class), the multi-channel camera can be transmitted to a PC or various embedded systems without separate SDK or drivers It has the advantage of being easy to apply.

For UVC, image transmission is possible through the standard driver included in the OS without a separate driver, and various UVC support programs can be used. The basic example of image processing of a deep learning platform is an example using a UVC camera, so it can be easily applied to various platforms.

In particular, it can be used for multi-channel image analysis in low-cost deep learning systems equipped with GPUs.

1 is a block diagram for explaining the concept of a multi-channel video transmission system according to the present invention.

2 is a data flow diagram for explaining an image synchronization and integration process of a multi-channel image transmission system according to the present invention.

3A and 3B are photographs illustrating an embodiment in which an image synchronization and integration unit of a multi-channel image transmission system according to the present invention is implemented.

4A to 4C are diagrams illustrating an example of an operation result of an image synchronization and integration unit of a multi-channel image transmission system according to the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The terms used in the present specification and claims are not limited to a conventional or dictionary meaning, and should be interpreted in a meaning and concept consistent with the technical matters of the present invention.

The configuration shown in the embodiments and drawings described in this specification is a preferred embodiment of the present invention, and does not represent all of the technical spirit of the present invention, so various equivalents and modifications that can be substituted for them at the time of the present application are provided. there may be

Referring to FIG. 1 , a multi-channel image transmission system 100 according to the present invention includes a camera unit 110 , a camera interface 120 , an image synchronization integration and transmission unit 130 , an image transmission interface 140 , and a host ( 150).

The multi-channel image transmission system 100 according to the present invention receives images photographed from four cameras 111 , 112 , 113 and 114 as input through the camera interface 120 , and then receives the images from the image synchronization integration and transmission unit 130 . Synchronization and real-time image integration are performed, and the image is transmitted to the host 150, a PC or an embedded system, through a USB3.0 transceiver, which is the image transmission interface 140 .

A plurality of (1 to N) cameras 111 , 12 , 113 , and 114 for photographing a specific subject may be disposed in the camera unit 110 of the multi-channel image transmission system 100 according to the present invention. 1 shows that four cameras are used, but the number is not limited thereto, and at least two or more cameras may be freely used according to circumstances.

On the other hand, in the arrangement method of the plurality of cameras, N cameras may be arranged in a line on the same plane with respect to an arbitrary camera, and N cameras may be sequentially arranged on a circumference spaced apart from the subject by a predetermined distance. .

The camera interface 120 of the multi-channel video transmission system 100 according to the present invention may use a high-speed serial link of FPD LINK 3 or Gigabit Multimedia Serial Link (GMLS). FPD LINK 3 or Gigabit Multimedia Serial Link (GMLS) is a technology used to extend camera cables. In the case of DS90UB913/DS90UB914, the I2C control signal for video signal and camera control can be extended by about 10 meters using a coaxial cable or one LVDS signal, and through this, the distance between the synchronization board and the camera can be extended.

The image synchronization integration and transmission unit 130 receives images captured from a plurality of cameras of the camera unit 110 and transmitted through the camera interface 120 , synchronizes them, integrates them in real time, and hosts the transmission interface 140 through the transmission interface 140 . Send to 150.

Synchronizing images and integrating and transmitting the images in real time is performed by a Field Programmable Gate Array (FPGA). FPGA (Field Programmable Gate Array) is used for real-time image processing and data processing, and even if a low-cost FPGA is used, it has the advantage of high-speed and large-scale image processing through parallel processing.

Image data captured by multiple cameras is written in line units using dual port memory in the FPGA, and then read sequentially in the order in which the cameras are connected. As a result, the synchronized and merged image becomes like a side-by-side overlay of the input image.

Meanwhile, in order to use this method, the difference between the camera image signals must not exceed one line. Therefore, in order to satisfy this condition, the same trigger signal is applied to each camera.

In the case of the multi-channel video transmission system according to the present invention, by using line buffering instead of frame buffering used in the general method of synchronizing, integrating and transmitting images, the amount of time required to synchronize and integrate images in real time Time can be reduced from tens of milliseconds (ms) to tens of microseconds (us).

In a 1280x720 resolution camera with a frame rate of 30 fps, the time required for one frame buffering is 33.33 ms, and the time required for one line buffering under the same conditions is 47 (us), so the present invention In the case of a multi-channel image transmission system according to the present invention, it is possible to significantly reduce the time required for image synchronization and integration compared to the prior art.

The video transmission interface 140 uses a USB3.0 transceiver of the USB3.0 UVC (USB Video Class) standard. This is not dependent on the operating system and is an interface that can be widely used for the host 150 such as a PC and an embedded system.

The image synchronized by the image synchronization integration and transmission unit 130 becomes a single image by itself, and is an image attached in the horizontal direction. This image is output to the host 150 such as a PC and an embedded system through a USB3.0 transceiver that is an image transmission interface 140 .

2 is a data flow diagram for explaining a video synchronization integration and transmission process of the multi-channel video transmission system according to the present invention.

Referring to FIG. 2 , in the case of a four-channel camera, four images (Image of camera 1 ~ Image of camera 4) having a horizontal resolution X pixel and a vertical resolution Y line are stored in a dual port memory. is used to convert a single image with a horizontal resolution of X * 4 pixels and a vertical resolution of Y lines.

Figure 3a is a picture showing the hardware of the FPGA, which is the video synchronization integration and transmission unit of the multi-channel video transmission system according to the present invention, and Fig. 3b is a picture showing the FPD LINK 3 of the camera interface unit.

4A to 4C are diagrams illustrating an example of an operation result of the video synchronization integration and transmission unit of the multi-channel video transmission system according to the present invention.

FIG. 4A shows a transmission screen of an image photographed using a 4-channel camera, FIG. 4B shows camera image transmission information, and FIG. 4C shows an operation state of an image synchronization integration and transmission unit.

Referring to FIGS. 4A to 4C , it can be confirmed that the images synchronized and real-time integrated using a general UVC capture program are transferred to the PC through the USB 3.0 transceiver.

As shown in (a) and (b) of FIG. 4 , in the case of a four-channel camera, an image output result is an image with a resolution of 4 times in the horizontal direction. At this time, the resolution of each camera is 1280x720, the resolution of the synchronized and integrated image is 5120x720, and the frame rate is 30fps.

The data bandwidth of one 1280x720/30fps camera is 1280*720*16*30/1024/1024 = 422Mbps = 52.75MBps for the YUY2 format, which is 16 bits per pixel, and the data bandwidth of four cameras of 1280x720/30fps is pixel In the case of YUY2 format, which is 16 bits per one, 1280*720*16*30*4/1024/1024 = 1688Mbps = 211MBps.

The maximum bandwidth of video transmission data can be checked by creating a video with a frame rate of 78fps and resolution of 1920*1080. In this case, if the performance of the chip is improved, the bandwidth of the maximum image transmission data may be further increased.

According to the multi-channel image transmission system and method according to the present invention, if a code for receiving and storing or processing an image is developed in a programming language that is not OS-dependent (for example, python) and the camera is also configured with UVC, it is programmed differently depending on the camera or OS. The advantage is that there is no need to do

As described above, the present invention relates to a multi-channel video synchronization and real-time transmission method. For multi-channel video input, a high-speed serial link (FPD LINK 3, GMSL, etc.) is used to extend the cable distance, and the received video is transferred from the FPGA. It includes a way to synchronize and integrate in real time, using the USB3.0 UVC standard to selectively use multiple cameras with multiple resolutions and frame rates.

Through this, it is possible to secure high-resolution and low frame rate image data required for deep learning learning with one system, and to process low resolution and high frame rate required for deep learning in real time.

Claims

In a multi-channel video transmission system

a camera unit including a plurality of cameras for photographing a subject;

a camera interface for receiving and transmitting the image captured by the camera unit;

an image synchronization integration and transmission unit for synchronizing and integrating images captured by the camera unit and transmitting;

a video transmission interface for video transmission; and

and a host receiving the synchronized and integrated image.
The method of claim 1, wherein the camera interface is

A multi-channel video transmission system characterized by using a high-speed serial link of FPD LINK 3 or Gigabit Multimedia Serial Link (GMLS).
According to claim 1, wherein the video synchronization integration and transmission unit

A multi-channel image transmission system, characterized in that it is a field programmable gate array (FPGA).
The method of claim 3, wherein the video synchronization integration and transmission unit

A multi-channel video transmission system, characterized in that line buffering is performed using a dual-port memory.
5. The method of claim 4,

The multi-channel video transmission system, characterized in that the video synchronized by the video synchronization integration and transmission unit has a horizontal resolution multiplied by the number of cameras.
The method of claim 1, wherein the video transmission interface is

A multi-channel video transmission system, characterized in that it performs video transmission using USB 3.0 UVC (USB Video Class).
The method of claim 6, wherein the video transmission interface is

A multi-channel video transmission system, characterized in that it is a USB 3.0 transceiver.
According to claim 1, wherein the host,

A multi-channel video transmission system, characterized in that it is a PC or an embedded system.
In a multi-channel video transmission method

A multi-channel video transmission method, comprising: receiving images captured by a plurality of cameras through a camera interface, synchronizing and integrating the images, and transmitting the images to a host through an image transmission interface.
10. The method of claim 9,

A multi-channel video transmission method, characterized in that the video is synchronized and integrated through line buffering using a dual-port memory.
11. The method of claim 10,

The camera interface uses FPD LINK 3 or Gigabit Multimedia Serial Link (GMLS) and

The video transmission interface is a multi-channel video transmission method, characterized in that using USB 3.0 UVC (USB Video Class).
12. The method of claim 11,

A multi-channel video transmission method, characterized in that it is possible to implement various resolutions using a plurality of cameras and to change the camera, resolution and frame rate to be used within the maximum video data bandwidth.