KR20210149419A - The syncronizing method for the filming time and the apparatus - Google Patents

Abstract

The present invention provides a method which is operated at any time during shooting and accurately synchronizes shooting times of captured videos without shooting the same stopwatch and an apparatus thereof. According to the present invention, a shooting time synchronization method for shooting videos with a plurality of shooting units and synchronizing the shot videos shoots time information of time information display unit capable of being shot by the shooting unit among the plurality of time information display units; continues to shoot a video; maintain the same frame rate in a plurality of videos; recognizes the time information included in each video to find out shooting time with respect to each of the plurality of videos; extracts the frame number of the video including the time information to generate a data pair consisting of the time information and the frame number; removes data outliers outside an estimated statistical tolerance from the data pairs; estimates a time information-frame number function of each video on the basis of the data pair from which the data outlier is removed; finds a difference between the frame numbers belonging to the same time on the basis of the function with respect to each of the plurality of videos; and compensates the difference to the frame number of each video to synchronize the shooting time.

Description

Method and device for synchronizing shooting time

The present invention relates to a photographing time synchronization method and apparatus for synchronizing photographing times for a plurality of photographed images, and in particular, a plurality of photographed images in time through analysis of images included in photographed images and representing respective photographing times. It relates to a method and device for synchronizing with

Broadcasters, etc., edit still images or videos shot in the field using multiple cameras to create a single broadcast image. Therefore, it is very important to synchronize the time of multiple captured images.

Conventionally, for example, Patent Document 1 discloses synchronizing a camera to a standard time such as GPS time to match the operation time of a plurality of cameras equally, but by receiving a GPS satellite signal, the images of each camera are converted into GPS time In order to synchronize with , only a special camera with a built-in GPS receiver should be used.

On the other hand, Patent Document 2 provides a method and apparatus for synchronizing shooting time through image analysis that can accurately synchronize the shooting time of moving pictures taken by a plurality of general cameras, but each camera is operated before the start of work and common must shoot the only stopwatch of

In the invention of Patent Document 2, the limitation of shooting only one and the same stopwatch that is in operation before the start of shooting is a technology that is difficult to apply to broadcast shooting, which requires shooting at various places and at unspecified times. That is, the shooting location If the camera is wide or there is an obstacle between the camera and the stopwatch, it is impossible to shoot with only one stopwatch, or it is inconvenient to move the fixed camera to take a stopwatch and then move to the original position to continue shooting. In addition, in some cases, when images taken from a plurality of remote locations are used, the invention of Patent Document 2 is a technology that cannot be applied to the industry because it is physically impossible to shoot the same stopwatch.

Accordingly, there is a need for a method and apparatus capable of providing a synchronized time to a plurality of cameras from time to time during photographing and accurately synchronizing the times even when the plurality of cameras do not photograph the same stopwatch.

Patent Document 1: Korean Patent Publication No. 10-2015-0129406 Patent Document 2: Korean Patent Publication No. 10-2018-0027016

Accordingly, an object of the present invention is to provide a method and apparatus that can be operated at any time during shooting and can accurately synchronize shooting times of captured videos without shooting the same stopwatch.

In order to achieve such a technical problem, the present invention provides a shooting time synchronization method in which images are respectively captured using a plurality of photographing units, and synchronization between the captured images is achieved,

Among the plurality of visual information display units, the photographing unit captures visual information of the visual information display unit that can be photographed, respectively, and continuously shoots images;

In the plurality of images, the frame rate of the image is the same,

For each of the plurality of images, by recognizing the visual information included in each image to find out the shooting time, and extracting the frame number of the image including the visual information, a data pair consisting of visual information-frame number create,

Remove data outliers outside the estimated statistical tolerance among the data pairs,

Estimating a function of visual information-frame number of each image based on the data pair from which the data outlier is removed,

For each of the plurality of images, a difference between frame numbers belonging to the same time is obtained based on the function,

By compensating the difference to the frame number of each image

It is characterized by synchronizing the shooting time.

Also, to make the frame rate of the image the same, some frames are removed from the image when it is faster than the reference, and some frames are inserted before and after as a dummy in the image when it is slower than the reference.

Also, the present invention

a plurality of time information display units each displaying time information;

a plurality of photographing units each photographing an image including the visual information;

A synchronization unit that synchronizes multiple captured images.

A synchronization device comprising:

The synchronization unit,

In the plurality of images, the frame rate of the image is the same,

For each of the plurality of images, by recognizing the visual information included in each image to find out the shooting time, and extracting the frame number of the image including the visual information, a data pair consisting of visual information-frame number create,

Remove data outliers outside the estimated statistical tolerance among the data pairs,

Estimating a function of visual information-frame number of each image based on the data pair from which the data outlier is removed,

For each of the plurality of images, a difference between frame numbers belonging to the same time is obtained based on the function,

By compensating for the difference to the frame number of each image, there is provided a photographing time synchronization device, characterized in that the photographing time is synchronized.

According to the synchronization method and apparatus of the present invention, it is possible to accurately synchronize the shooting times of moving pictures captured by a plurality of general cameras.

BRIEF DESCRIPTION OF THE DRAWINGS It is a figure explaining the outline|summary of the apparatus of this invention.
2 is a view showing a state displayed by executing an application indicating a time in a smart terminal possessed by each photographer.
3 is a flowchart illustrating the synchronization process of the synchronization unit 130 according to the first embodiment of the present invention, and is a flow chart for synchronizing shooting time through image analysis.
4 is a diagram illustrating a method of calculating a frame number difference using a frame-time estimation function.
5 is a diagram illustrating a frame-time function estimation method.

Next, the present invention will be described by specifically describing preferred embodiments embodying the spirit of the present invention. The examples described below are specific examples of the present invention, not limiting the present invention, It should be construed in accordance with the appended claims.

First, an outline of the device of the present invention will be described based on the accompanying FIG. 1 .

The synchronization device 100 of the present invention includes a visual information display unit 110 , a photographing unit 120 , and a synchronization unit 130 . Hereinafter, each unit will be described.

Visual information display unit (110)

The time information display unit 110 is a device that displays the current time, and there is no limitation as long as it represents time information and can be recognized by photographing it. A wrist watch is good, and a wall clock hanging on the wall is fine. However, in recent years, with the development and spread of smart terminals such as wireless Internet and mobile phones, it is necessary to adopt a smart terminal (mobile phone) possessed by the photographer in terms of accuracy, etc. However, in an environment in which the Internet cannot be used, a wrist watch may be usefully used.

Here, the smart terminal includes all devices capable of displaying visual information, such as a mobile phone, a pad, and a laptop computer.

Therefore, in this embodiment, it is assumed that the smart terminals #1, #2, ..., #n possessed by each photographer constitute the visual information display unit 110 .

Each photographer executes an application installed on the smart terminal (time information display unit 110), sends and receives a time synchronization signal with the time synchronization server, and displays synchronized time information on the screen of the smart terminal. One specific example is shown in FIG. 2 is a diagram showing a state displayed by executing an application indicating the time on a smart terminal possessed by each photographer. In FIG. 2, the current time is 08:47:39:26 represents

On the other hand, it is preferable that the application related to the time executed in the smart terminal includes the following functions.

All terminals in which the application is installed are connected to the time synchronization server by wire/wireless connection to exchange time synchronization signals and display time information in synchronization with the time synchronization server.

The synchronized visual information includes year, month, day, hour, minute, second, millisecond, etc., and is updated at a cycle equal to or higher than the maximum refresh rate of the terminal screen.

The screen on which the time information is displayed may be configured as “year-month-day-hour-minute-second-millisecond” as shown in FIG. 2 .

filming unit (120)

The photographing unit 120 is composed of photographing equipment such as a camera, and the desired continuous or intermittent video and still images are taken, as well as the visual information display unit 110, which is composed of a smart terminal in which visual information is displayed. It is a device that records the visual information displayed on the screen of the camera so that it is included in the captured image at least once.

The photographing unit 120, in this embodiment, is composed of normal photographing equipment (in FIG. 1, photographing equipment #1, ..., photographing equipment N), but in some cases, a normal camera or a smart camera with a built-in camera function terminal can be used.

Here, when the smart terminal carried by the photographer is used as the photographing unit 120, the photographer's own wrist watch can be used as the time information display unit 110, and also the watch or smart terminal of a person around the photographer, such as an assistant. can be used as the visual information display unit 110 .

In addition, in some cases, it may be assumed that the photographer's own smart terminal serves as both the visual information display unit 110 and the photographing unit 120. In this case, first execute the application for displaying visual information on his/her smart terminal. to display the visual information on the smart terminal, capture the screen and use it as an image indicating the visual information, and include an image indicating the visual information in the image taken by the smart terminal.

As described above, the photographing unit 120 of the present invention is a device that captures the screen of the smart terminal on which the visual information is displayed at least once and continuously shoots videos using photographing equipment.

Synchronizer 130

It is a device for synchronizing the time of a plurality of captured images by analyzing images such as time information from the photographing unit 120. Such a photographing unit is a commercially available CPU or special-purpose CPU with a memory or input/output device added thereto. It is constituted by adding a program related to this method.

Hereinafter, the function of the synchronization device 100 of the present invention configured as described above will be described while explaining the specific configuration of the synchronization unit 130 based on a synchronization flowchart and the like.

3 is a flowchart illustrating a synchronization process of the synchronization unit 130 .

In step S100, the frame rates of the captured images are synchronized to the same value. The captured images may have different frame rates depending on the type of device of the capturing unit 120, such as a broadcasting shooting camera, a smart terminal with a built-in camera, etc. To calculate the frame difference, the frame rate of all captured images must be the same. Therefore, depending on the type of the photographing unit 110, frames are added by matching different frame rates to one rate. Alternatively, the frame rate of the entire video can be made the same by deleting it. That is, if the speed is faster than a certain standard, some frames are removed from the video, and if it is slow, frames in the video are inserted before and after as a dummy. .

In step S102, time information is acquired, and the frame number to which an image including the time information belongs is extracted. When the synchronization unit 130 receives the image from the photographing unit 120, the Visual information is extracted. This is to extract visual information from the captured image through recognition of the image and its analysis. As in the present embodiment, visual information displayed on the smart terminal is recognized and stored as a number, and time When the information display unit 110 is a general watch rather than a smart terminal, it recognizes the image and converts it into a number. Also, at the same time, the number of the frame to which the corresponding time information belongs is extracted.

Specifically, the received captured image is decoded and visual information is extracted for each frame through recognition technology. The extracted visual information is converted into milliseconds based on Coordinated Universal Time (UTC), and the corresponding time A pair (2-tuple) is created and stored with the frame number of the frame from which the information is extracted. In this case, the frame number indicates the relative order in which the decoded frames are reproduced.

으로 표시될 수 있다.For example, as in Fig. 5, in coordinates where the frame number n is the horizontal axis and the time information t is the vertical axis, the pair of visual information-frame number is one point

can be displayed as

The operation of extracting the time information and the corresponding frame number is performed for all the images from each photographing unit 120 , so that a pair of visual information-frame number of each photographing unit 120 is obtained and displayed on the coordinates.

들 중에서 격자 무늬의 점은,영상분석을 통한 시각정보 획득과정에서 오류가 발생하여 통계적 허용범위를 벗어난 값을 갖는 데이터로서,주로 시각정보에 대한 촬영오류 혹은 당해 시각정보에 대한 인식오류,혹은 양자의 결합으로부터 오는 것으로 생각된다.이러한 점들이 다음 단계인 시각정보-프레임 번호에 대한 함수 추정단계에 포함될 경우에,시각정보 동기화의 정확도를 낮추는 문제를 발생시킬 수 있다. 따라서,도5의 격자 무늬의 점 같은 데이터를 제거할 수 있는 데이터이상치 탐지기술을 적용한다.In step S104, linear analysis is performed on the relationship between visual information and frame number. Detecting and removing data outliers (data outliers) in order to increase the accuracy of function estimation for visual information-frame numbers may include. Points indicated in FIG. 5

Among them, the grid-patterned dots are data having a value outside the statistically acceptable range due to an error in the process of acquiring visual information through image analysis, mainly a shooting error for visual information or a recognition error for the visual information, or both. It is thought to come from the combination of . When these points are included in the next step, time information-frame number function estimation, a problem of lowering the accuracy of visual information synchronization may occur. Therefore, a data outlier detection technique that can remove data such as dots in the grid pattern of FIG. 5 is applied.

를 추정하는 단계이다.1차원함수를 추정하기 위해, 일반적인 다항식 계수 추정방법(polynomial curve fitting)을 사용할 수 있다.이렇게 얻은 함수가 도5에서 실선으로 나타내는 것으로서,이 함수

의 기울기는 단위 프레임당 증가하는 시간을 나타내는 값이 된다.따라서 동일한 시각정보를 동일한 프레임레이트로 촬영하였다면 같은 값을 가져야 한다.또한 함수

의 Y-절편(함수와 세로축이 만나는 지점)은 해당 영상을 촬영하기 시작한 UTC 시각을 나타낸다.In step S106, a function of time information-frame number is estimated using the time information and frame number data obtained above. function of number

In order to estimate a one-dimensional function, a general polynomial coefficient estimation method (polynomial curve fitting) can be used. The function obtained in this way is represented by a solid line in FIG.

The slope of is a value indicating increasing time per unit frame. Therefore, if the same visual information is captured at the same frame rate, it must have the same value. Also, the function

The Y-intercept (the point where the function and the vertical axis meet) represents the UTC time at which the image was taken.

, M은 촬영영상의 갯수)에 기반하여동일한 시각정보 값(T)에 대응하는 프레임 번호(

)들을 계산하고,계산된 프레임 번호들의 상대적인 프레임 차이값을 계산하는 단계를 포함한다.이 때, 계산된 프레임 차이값은 동일한 편집기의 타임라인에 같은 UTC 시각에 촬영된 프레임을 놓이게 하기 위해,각 촬영영상별로 보상되어야 하는 값을 의미한다. 한예로서,프레임 차이값은 도4의 시각정보-프레임 번호의 함수

과

를 이용하여 아래와 같은 수식으로 구할 수도 있다.In step S108, a frame difference between the linear estimation graphs is calculated using the derived linear estimation graphs. Time-synchronizing the captured images is to find a frame captured at the same UTC time, and the frame is displayed on the same timeline of the editor. ) means to arrange to be positioned in

, M is the number of captured images) based on the frame number (

) and calculating a relative frame difference value between the calculated frame numbers. At this time, the calculated frame difference value is used to place frames shot at the same UTC time on the timeline of the same editor, each It means a value that should be compensated for each captured image. As an example, the frame difference value is a function of the visual information-frame number of FIG.

class

It can also be obtained using the following formula:

In step S110, the shooting times of a plurality of moving pictures are synchronized based on the frame difference. Among the functions of the visual information-frame number, the function with the smallest Y-intercept means that the image was taken the earliest. have.

As described above, the present invention captures time information from a plurality of stopwatches, that is, each smart terminal in the embodiment, and also shoots a necessary image.

Then, time information is extracted from the captured image, and the number of frames including the time information is extracted and stored as a data pair together.

Among the data pairs, a data pair that is out of a statistical estimation range due to a recognition error, etc. is considered to be an error of photographing or recognition and removed.

A function of the frame-frame number is estimated based on the data pair of the visual information-frame number pair.

A frame difference is calculated based on the above function, and based on this, synchronization of each captured image is performed.

100: synchronizer
110: visual information display unit
120: shooting unit
130: synchronization unit

Claims (4)

A method of synchronizing a shooting time in which images are respectively taken by using a plurality of shooting units, and synchronization between the captured images is obtained, the visual information of the visual information display unit capable of being photographed by the shooting unit among the plurality of visual information display units is photographed, respectively; Continuing to shoot an image, make the frame rates of the images the same in the plurality of images, and for each of the plurality of images, find out the shooting time by recognizing the visual information included in each image, the time By extracting the frame number of the image containing information, a data pair consisting of visual information-frame number is generated, and data outliers outside the estimated statistical tolerance range are removed from the data pair, and the data anomaly Estimating a function of time information-frame number of each image based on the data pair from which the value has been removed, and obtaining a difference between frame numbers belonging to the same time based on the function for each of the plurality of images, The difference A shooting time synchronization method, characterized in that the shooting time is synchronized by compensating (補償) to the frame number of each image. According to claim 1, Equalizing the frame rate of the image, When it is faster than the standard, some frames are removed from the image, and when it is slower than the standard, some frames from the image are inserted before and after as a dummy. visual synchronizer. A synchronization device comprising: a plurality of time information display units each displaying time information; a plurality of photographing units each capturing an image including the time information; and a synchronizing unit synchronizing the plurality of captured images, wherein the synchronizer comprises In the plurality of images, the frame rate of the images is the same, and for each of the plurality of images, the visual information included in each image is recognized to find the shooting time, and the image including the visual information is A data pair composed of visual information-frame number is generated by extracting the frame number, data outliers outside the estimated statistical tolerance range are removed from among the data pairs, and each image based on the data pair from which the data outliers are removed Estimating a function of time information-frame number of, for each of the plurality of images, finding the difference between frame numbers belonging to the same time based on the function, and compensating for the difference to the frame number of each image. Shooting time synchronization device, characterized in that for synchronizing. The apparatus of claim 3, wherein the time information display unit is a mobile phone.

Images