KR102498870B1 - Camera system and method for processing image thereof - Google Patents

Abstract

An embodiment of the present invention discloses a camera system and an image processing method thereof.
An image processing method of a camera system according to an embodiment of the present invention includes receiving an event including an image processing request for generating a dewarped image from a fisheye image; and generating the dewarped image by applying a first dewarping algorithm or a second dewarping algorithm different from the first dewarping algorithm to the fisheye image in response to receiving the event.

Description

Camera system and its image processing method {CAMERA SYSTEM AND METHOD FOR PROCESSING IMAGE THEREOF}

Embodiments of the present invention relate to a camera system and an image processing method thereof, and specifically, to a camera system for acquiring a fisheye image and an image processing method thereof.

Today, a plurality of surveillance cameras are installed everywhere, and technologies for recording, storing, and transmitting images acquired by the surveillance cameras are being developed. In particular, as the number of installed surveillance cameras increases, the frequency of installing a system providing a plurality of views with a single fisheye camera is increasing instead of a system including a plurality of surveillance cameras.

Embodiments of the present invention are intended to provide a camera system capable of adaptively providing a dewarped image obtained by correcting distortion of a fisheye image according to circumstances and an image processing method thereof.

An image processing method of a camera system according to an embodiment of the present invention includes receiving an event including an image processing request for generating a dewarped image from a fisheye image; and generating the dewarped image by applying a first dewarping algorithm or a second dewarping algorithm different from the first dewarping algorithm to the fisheye image in response to receiving the event.

The generating of the dewarping image by applying the first dewarping algorithm may include: obtaining 2D coordinates of pixels of the fisheye image corresponding to pixels of the dewarped image in a 3D space in a pixel unit; and calculating pixel values of pixels of the dewarped image corresponding to pixel values of pixels of the fisheye image.

The generating of the dewarping image by applying the second dewarping algorithm may include: blocking the fisheye image into a plurality of blocks; estimating a transform function in units of blocks through feature point matching between a block of the fisheye image and a corresponding block of the dewarped image; acquiring 2D coordinates of pixels in a corresponding block of the dewarped image corresponding to pixels in a block of the fisheye image based on the transform function; and calculating pixel values of pixels of the dewarped image corresponding to pixel values of pixels of the fisheye image.

The method may further include determining a block size of the second dewarping algorithm based on an elapsed time from an end of a previous event until a next event is received.

The generating of the dewarping image may include generating a dewarping image by using the second dewarping algorithm when receiving an initial event; and generating a dewarping image using the first dewarping algorithm when an elapsed time from the end of the first event to reception of the next event is longer than a threshold time.

The generating of the dewarping image may include: generating a dewarping image using the second dewarping algorithm when an elapsed time from the end of a previous event to reception of a next event is longer than a second threshold time; and generating a dewarping image using the first dewarping algorithm when the elapsed time is longer than a first threshold time that is longer than a second threshold time.

Embodiments of the present invention may provide a camera system capable of providing a dewarped image in which distortion of a fisheye image is corrected by selectively considering image processing speed or image quality according to circumstances and an image processing method thereof.

1 and 2 are views schematically showing a camera system according to an embodiment of the present invention.
3 is a diagram schematically illustrating a process of generating a corrected image obtained by converting a fisheye image by applying a dewarping algorithm according to an embodiment of the present invention.
4 to 7 are views schematically illustrating a first dewarping algorithm according to an embodiment of the present invention.
8 and 9 are views schematically illustrating a second dewarping algorithm according to an embodiment of the present invention.
10 is a diagram showing the relationship between block size and distortion correction in the second dewarping algorithm.
11 is a flowchart schematically illustrating an adaptive dewarping method of an image capture device according to an embodiment of the present invention.
FIG. 12 is an example of an adaptive dewarping method of the image capture device according to the embodiment shown in FIG. 11 .
13 is a flowchart schematically illustrating an adaptive dewarping method of an image capture device according to another embodiment of the present invention.
14 is a flowchart schematically illustrating an adaptive dewarping method of an image capture device according to another embodiment of the present invention.
15 is an example of an adaptive dewarping method of an image capture device according to another embodiment of the present invention.

Since the present invention can apply various transformations and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. Effects and features of the present invention, and methods for achieving them will become clear with reference to the embodiments described later in detail together with the drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various forms.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, and when describing with reference to the drawings, the same or corresponding components are assigned the same reference numerals, and overlapping descriptions thereof will be omitted. .

In the following embodiments, terms such as first and second are used for the purpose of distinguishing one component from another component without limiting meaning. In the following examples, expressions in the singular number include plural expressions unless the context clearly dictates otherwise. In the following embodiments, terms such as include or have mean that features or components described in the specification exist, and do not preclude the possibility that one or more other features or components may be added. In the drawings, the size of components may be exaggerated or reduced for convenience of description. For example, since the size and shape of each component shown in the drawings are arbitrarily shown for convenience of description, the present invention is not necessarily limited to those shown.

1 and 2 are views schematically showing a camera system according to an embodiment of the present invention.

Referring to FIG. 1 , a camera system according to an embodiment may include an image acquisition device 100 and a monitoring device 200.

The image capture device 100 acquires an original image of the surveillance region by capturing the surveillance region. The image capture device 100 may capture a monitoring area in real time for the purpose of monitoring or security. The image acquisition device 100 may be a camera that provides an original image composed of RGB for a target space to be captured, in particular a fisheye image of the target space to be captured.

Hereinafter, the image capture device 100 is a fisheye camera having an angle of view of 180 degrees or more, and will be described on the premise that it provides a fisheye image of a space to be photographed, but the spirit of the present invention is not limited thereto. A 'fisheye image' described in the present invention may mean an original image acquired by a fisheye camera, or may mean a wide-angle image, a distorted image, and the like.

The image capture device 100 may generate a dewarped image (hereinafter, referred to as 'corrected image') obtained by correcting a fisheye image, which is an original image. Correcting the distortion of a fisheye image is called 'Distortion Calibration' or 'Dewarping'. The image capture device 100 may generate a corrected image for the entire original image or at least one region set as a region of interest in the original image according to settings.

The image acquisition device 100 may communicate with the monitoring device 200 through the network 300 .

The network 300 may include a wireless network, a wired network, a public network such as the Internet, a private network, a global system for mobile communications (GSM) network, a general packet radio network (GPRS), local area network (LAN), wide area network (WAN), metropolitan area network (MAN), cellular network, public switched telephone network (PSTN), private network ( personal area network), Bluetooth, Wi-Fi Direct, Near Field communication, Ultra Wide band, a combination thereof, or any other network. However, it is not limited thereto.

The monitoring device 200 may receive a fisheye image and/or a corrected image from the image capture device 100 . The monitoring device 200 may display at least one of a fisheye image and/or a corrected image. The monitoring device 200 may provide various view modes for displaying a fisheye image and/or a corrected image. A view mode refers to a method capable of displaying a plurality of images, and a user can view an original image or a corrected image by selecting one of several view modes.

The monitoring device 200 may provide a graphical user interface on the display. A user may set the functions of the image capture device 100 and/or the monitoring device 200 by inputting a view mode, a time interval, a region of interest (ROI), event conditions, etc. through a graphic user interface using an input device. . The monitoring device 200 may transmit a user input for setting functions of the image capture device 100 to the image capture device 100 .

The monitoring device 200 may be a terminal such as a personal computer, smart phone, tablet, or handheld device, or a server such as a cloud server, a recording server, an upgrade server, or an alarm server.

Referring to FIG. 2 , the image acquisition device 100 may include an image acquisition unit 102 , a control unit 104 , a storage unit 106 and a communication unit 108 .

The image acquisition unit 102 may include a fisheye lens and an image sensor. A fisheye lens can have a wide angle of view of 180° or more by overlapping various lens elements (convex lens, concave lens, spherical surface, aspherical lens, etc.) at a minimum of 6 sheets and at a maximum of 10 sheets or more. The image sensor may be a Complementary Metal Oxide Semiconductor (CMOS) device or a Charge-Coupled Device (CCD) device.

The controller 104 may include all types of devices capable of processing data, such as a processor. Here, a 'processor' may refer to a data processing device embedded in hardware having a physically structured circuit to perform functions expressed by codes or instructions included in a program, for example. As an example of such a data processing device built into hardware, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated (ASIC) circuit), field programmable gate array (FPGA), etc., but the scope of the present invention is not limited thereto.

The controller 104 may receive an event and generate a corrected image by applying a dewarping algorithm to a region where the event occurs in the fisheye image. Here, the event may be an image processing request for generating and/or regenerating a corrected image from a fisheye image, such as setting and changing a region of interest, changing a view mode, and changing a viewer's shape. In one embodiment, the event is an input for setting a region of interest, which is a partial region of the fisheye image, in a fisheye image, and the region where the event occurs may be the region of interest. Since the region of interest set in the fisheye image is converted to generate the corrected image, image data of the corrected image may be generated based on the image data of the region of interest.

The region of interest may be set as a figure in the fisheye image. The controller 104 may receive a setting value for setting a region of interest of the fisheye image input by the user, and display the region of interest corresponding thereto on the fisheye image. The setting value may be a value based on a pan/tilt/zoom value of the image acquisition unit 102 . The set region of interest may be displayed as a figure. Here, the figure may refer to a polygon composed of one or more points and lines connecting the one or more points to each other. For example, the figure may be a polygon such as a circle, an ellipse, a rectangle, a triangle, and a pentagon.

The controller 104 may receive a plurality of settings for the region of interest and generate a plurality of corrected images corresponding to the region of interest. The controller 104 may generate a corrected image by applying a dewarping algorithm to a fisheye image, which is an original image. The control unit 104 uses the first dewarping algorithm or the second dewarping algorithm in response to a subsequently received event according to the length of the time elapsed from the end of the previous event (hereinafter referred to as 'elapsed event time'). It is possible to generate a correction image adaptively by selectively applying to . The first dewarping algorithm and the second dewarping algorithm will be described later.

As the corrected image obtained by image processing of the original image is provided to the user, the user terminal or monitoring device 200 may support various view modes or generate events to satisfy the user's UI/UX interaction. there is. When an event occurs, it is necessary to consider the trade-off relationship between performance and speed.

The image capture device 100 according to an embodiment of the present invention applies a second dewarping algorithm to an original image in response to a situation with a high event frequency, thereby quickly providing a corrected image to a user and responding to a situation with a low event frequency. By applying the first dewarping algorithm to the original video, it is possible to provide a high-quality/high-definition corrected video to the user.

The storage unit 106 performs a function of temporarily or permanently storing data processed by the image capture device 100 . The storage unit 106 may include magnetic storage media or flash storage media, but the scope of the present invention is not limited thereto. The storage unit 106 may include a digital video recorder (DVR) or a network video recorder (NVR). The storage unit 106 may store an original image and/or a corrected image.

The communication unit 108 includes hardware necessary for the image capture device 100 to transmit/receive signals such as control signals or data signals to and from other network devices such as the monitoring device 200 and/or a server (not shown) through a wired or wireless connection. It may be a device containing software.

3 is a diagram schematically illustrating a process of generating a corrected image obtained by converting a fisheye image by applying a dewarping algorithm according to an embodiment of the present invention.

Referring to FIG. 3 , image data of a region of interest (ROI) in a fisheye image (FI), which is an original image, may be converted into image data of a corrected image (CI) by a dewarping algorithm. The dewarping algorithm may perform geometric transformations to change the position of the pixel Ps of the fisheye image FI to the position of the pixel P of the corrected image CI.

The dewarping algorithm may include a process of generating a dewarping map to generate a corrected image CI from the fisheye image FI. In order to obtain a pixel value of the pixel P of the corrected image CI, the coordinates of the pixel Ps of the fisheye image FI corresponding to the coordinates of the pixel P must be obtained. The dewarping map has information in which the coordinates of the pixels P of the corrected image and the coordinates of the pixels Ps of the fisheye image corresponding thereto are matched. That is, the size of the dewarping map may be proportional to the size of the corrected image.

4 to 7 are views schematically illustrating a first dewarping algorithm according to an embodiment of the present invention.

Referring to FIGS. 4 and 5 , the fisheye image FI is a picture of a subject imaged on a hemispherical surface of a virtual hemisphere S corresponding to a lens on a plane that is the bottom surface of the hemisphere S corresponding to an image sensor. It may be a developed video. For the hemisphere (S), 3 defined by the X-axis and Y-axis, which are orthogonal directions from the origin (O) of the fisheye image (FI), and the Z-axis, which is a direction extending from the origin (O) to the vertex of the hemisphere (S) A dimensional coordinate system may be set. R is the distance from the origin (O) to the surface of the hemisphere, i.e. the radius of the hemisphere.

Any position on the plane (A) perpendicular to the line of sight direction (B) as seen from the origin (O) and where the origin (O') is in contact with the hemisphere is the coordinate P (x, y) of the two-dimensional coordinate system of the x-axis and y-axis can be expressed as The plane A is a real world area that the user wants to see, and may correspond to the corrected image CI of FIG. 3 . The dewarping algorithm may be a process of reconstructing a region that a user wants to see from a fisheye image FI, which is an original image, into a corrected image CI.

In the user's Pan/Tilt/Zoom settings, zoom means change in size (a) of plane (A), that is, change in field of view (FOV), and tilt means change in plane ( It means that the origin (O') of A) rotates (b) by an angle (α) from the Z axis along the hemisphere at the apex of the hemisphere (S), and the fan is the origin (O') of the plane (A) It may mean rotational movement (c) by an angle (β) about the Z-axis along the hemisphere.

Referring to FIG. 6, the two-dimensional coordinates P (x, y) at any position in the plane A are X-axis rotation by the rotation angle ψ, Y-axis rotation by the rotation angle ϕ, and rotation angle ( It can be converted into three-dimensional coordinates P(X', Y', Z') by applying a transformation matrix by Z-axis rotation by θ).

The Z-axis is rotated by the rotation angle (φ), the Y-axis is rotated by the rotation angle (θ) while the Z-axis is rotated, and the X-axis is rotated by the rotation angle (ψ) while the Z-axis and Y-axis are rotated. In the case of rotation, the coordinate transformation matrix according to the rotation of the Z->Y->X axis, for example, the transformation matrix of the affine transformation is as shown in Equation (1) below. Since Affine Transformation is well known, a detailed description thereof will be omitted.

...(1)

...(One)

That is, the 3D coordinates P(X', Y', Z') of an arbitrary position in the plane A to which Affine Transformation is applied can be expressed as in Equation (2) below.

X' = M00 * x + M01 * y + M02 * R

Y' = M10 * x + M11 * y + M12 * R

Z' = M20 * x + M21 * y + M22 * R... (2)

Next, the three-dimensional coordinate P' (x", y", z") of a position obtained by projecting an arbitrary position in the plane A onto a hemispherical surface can be expressed as Equation (3) below.

PO = sqrt(X' ² + Y' ² + Z' ² )

P'O = R

x" = (P'O / PO) * X' = (sqrt(X' ² + Y' ² + Z' ² ) / R) * X'

y" = (P'O / PO) * Y' = (sqrt(X' ² + Y' ² + Z' ² ) / R) * Y'

z" = (P'O / PO) * Z' = (sqrt(X' ² + Y' ² + Z' ² ) / R) * Z'

Referring to FIG. 7 , the two-dimensional coordinates Ps(Sx, Sy) of the position projected onto the hemisphere S onto the bottom surface of the hemisphere S can be expressed as Equation (4) below. The position of Ps may vary depending on the lens.

α = arccos(z"/R)

temp1 = sqrt(R ² - z" ² )

Sx = pp * x"

Sy = pp * y" ... (4)

Here, α is an angle from the Z axis to P' (x", y", z") on the hemispherical surface, and pp may be a nonlinear function for projecting a position on the hemispherical surface to a position on the bottom surface. Nonlinear function pp may be different for each lens of the camera In one embodiment, the nonlinear function pp may use a stereographic projection formula.

The control unit 104 performs projection twice using the Affine Transformation of the first dewarping algorithm, so that the pixel Ps of the fisheye image FI corresponding to an arbitrary pixel P of the plane A is pixel-by-pixel. ), the coordinates of the XY coordinate system can be obtained. The pixel value of the pixel P of the plane A may be matched with the pixel value of the pixel Ps of the corresponding fisheye image FI.

The first dewarping algorithm is a nonlinear transformation process for displaying a corrected image in which distortion is corrected on a plane in an arbitrary viewing direction from a fisheye image. The 2D coordinates of pixels of the fisheye image corresponding to pixels of the corrected image in a 3D space are It may include a search process.

8 and 9 are views schematically illustrating a second dewarping algorithm according to an embodiment of the present invention.

Referring to FIGS. 8 and 9 , the controller 104 blocks the fisheye image FI into a plurality of blocks and uses a homography projective transformation to block each pixel of the fisheye image FI. A corresponding pixel of the corrected image CI corresponding to can be obtained.

The controller 104 may block the corrected image CI displayed on the viewer into a plurality of blocks corresponding to the plurality of blocks of the fisheye image FI. The controller 104 may obtain a conversion function H from corresponding feature points of corresponding blocks of the fisheye image FI and the corrected image CI, for example, 4 pairs of vertices. The transformation function H may be a matrix representing a geometric transformation relationship between corresponding blocks. For example, the transformation function H may be a 2D homography matrix expressing a position transformation relationship between corresponding pixels of two 2D planes existing in a 3D space as an n x m matrix.

Equation (5) below is a transformation matrix that is an example of a 3x3 transformation function (H) representing a geometric transformation relationship between a pair of corresponding pixels.

... (5)

... (5)

Each element (h11 to h33) of the transformation matrix includes rotation information indicating at which rotation angle to rotate, translation information indicating how much to move in the x, y, and z directions, and x, y, and z directions It includes scaling information indicating how much to change the size of .

The size and number of elements of the transformation matrix between the four pairs of corresponding pixels may be greater than the size and number of elements of the transformation matrix for the pair of corresponding pixels in Equation (5).

As shown in FIG. 9(a), the control unit 104 controls four pairs of vertices corresponding to the first block B1 of the fisheye image FI and the first corresponding block B1' of the corrected image CI. A first transform function H1 is obtained from the pairs, and positions of corresponding pixels of the first corresponding block B1' corresponding to pixels in the first block B1 are obtained using the first transform function H1. can do.

As shown in FIG. 9(b), the control unit 104 controls four pairs of vertices corresponding to the second block B2 of the fisheye image FI and the second corresponding block B2' of the corrected image CI. A second transform function H2 is obtained from the pairs, and positions of corresponding pixels of the second corresponding block B2' corresponding to pixels in the second block B2 are obtained using the second transform function H2. can do.

The control unit 104 obtains the transform function of the block of the fisheye image FI and the corresponding block of the corrected image CI in block units for each of the remaining blocks, and uses the transform function in block units to obtain the fisheye image FI. The positions of corresponding pixels of the corrected image CI corresponding to the pixels of ) may be obtained.

The control unit 104 obtains a transform function in block units using the Homography Projective Transformation by the second dewarping algorithm, and obtains a transform function corresponding to the pixel Ps of the fisheye image FI in block units using the block-specific transform function. The position of the pixel P of the corrected image CI may be obtained. A pixel value of a pixel of the fisheye image FI may be matched with a pixel value of a corresponding pixel of the corrected image CI.

The control unit 104 may pre-calculate the transform function of each block according to the block size and/or the number of blocks, convert them into a table, and store them. Accordingly, the dewarping operation speed may be increased.

10 is a diagram showing the relationship between block size and distortion correction in the second dewarping algorithm.

10(a) is an image of a subject (a) without distortion. The block size of the corrected image of FIG. 10(b) is larger than the block size of the corrected image of FIG. 10(c). It can be seen that the smaller the block size, that is, the greater the number of blocks, the better distortion correction of the subject. On the other hand, as the block size increases, the calculation amount and calculation time decrease, and the calculation speed increases.

The first dewarping algorithm uses Affine Transformation to perform primary projection of the position of the pixel Ps of the corrected image CI to a position on the hemispherical surface and to convert the position on the hemispherical surface to the position on the bottom surface of the hemisphere. Corresponding pixel by pixel by performing secondary projection, the corrected image is similar to the actual scene. That is, the corrected image to which the first dewarping algorithm is applied may be a high-quality image with minimized distortion. On the other hand, the first dewarping algorithm greatly affects the performance of the image capture device 100 because the amount of calculation increases due to vector calculation for affine transformation twice. The higher the resolution and the lower the hardware specifications, the bigger the performance issue.

The second dewarping algorithm acquires a conversion function only for feature points in block units and simply calculates the remaining pixels using this, thereby reducing the amount of calculation and calculation time, thereby increasing the calculation speed. On the other hand, the corrected image generated using the second dewarping algorithm may have greater distortion than the corrected image generated using the first dewarping algorithm.

An embodiment of the present invention can control the event response speed by adaptively applying different dewarping algorithms according to the event frequency.

11 is a flowchart schematically illustrating an adaptive dewarping method of an image capture device according to an embodiment of the present invention.

Referring to FIG. 11 , the controller 104 of the image capture device 100 receives a view mode setting from a user, starts image processing for generating a correction image from a fisheye image, and when an event is received (S11) , the elapsed time of the event may be compared with the first threshold time (S13). If the event elapsed time is longer than the first threshold time, the controller 104 may generate a corrected image by performing dewarping on the fisheye image using a first dewarping algorithm in response to the received event (S15). If the event elapsed time is shorter than the first threshold time, the controller 104 may generate a corrected image by performing dewarping on the fisheye image using the second dewarping algorithm (S17).

The controller 104 may use the second dewarping algorithm when receiving an initial event according to a user's setting. The control unit 104 counts the elapsed time of the event from the end of the first event, compares it with the first threshold time, and selectively uses the first dewarping algorithm or the second dewarping algorithm from the second or later received event according to the comparison result. there is.

FIG. 12 is an example of an adaptive dewarping method of the image capture device according to the embodiment shown in FIG. 11 .

Referring to FIG. 12 , the controller 104 may receive a view mode setting or the like from a user (S21), and start image processing for generating a correction image from a fisheye image. Upon receiving the first event (S23), the controller 104 may generate a corrected image by performing dewarping using the second dewarping algorithm (S25). The control unit 104 counts the elapsed time of the event from the end of the first event and compares it with the first threshold time (S27), and if no event is received for the first threshold time or longer, the first dewarping algorithm is performed in response to the event received thereafter. A corrected image may be generated by performing dewarping using the image (S29). When an event is received within the first threshold time, the control unit 104 may generate a corrected image by maintaining the second dewarping algorithm and performing dewarping using the second dewarping algorithm (S25).

In this embodiment, after quickly generating a dewarping map using the second dewarping algorithm to which the block concept is applied and providing the dewarping image to the user, the first dewarping algorithm when no event from the user occurs for a certain period of time. A dewarping map can be created using , and a dewarping image with minimized distortion can be provided to the user.

13 is a flowchart schematically illustrating an adaptive dewarping method of an image capture device according to another embodiment of the present invention.

Referring to FIG. 13 , the control unit 104 of the image capture device 100 receives a view mode setting from a user, starts image processing for generating a correction image from a fisheye image, and when an event is received (S31) , the block size of the second dewarping algorithm can be determined based on the elapsed time of the event (S33). The control unit 104 may generate a corrected image by performing dewarping using the second dewarping algorithm with the determined block size (S35).

Specifically, in step 33, the controller 104 may set at least one second threshold time for the event elapsed time according to the user's setting, and set the block size of the second dewarping algorithm for each second threshold time. The block size may be set to be smaller as the second threshold time is longer.

For example, the control unit 104 may set the 2-1st threshold time to the 2-nth threshold time, and set the block size of the second dewarping algorithm for every n second threshold times. Here, the larger n is, the longer the time, and the smaller the block size.

In the case of a first received event (first event), the controller 104 may count the elapsed time of the event from the start of the operation, and count the elapsed time of the event after the end of the previous event from the second received event (second event).

The control unit 104 may determine the block size of the second dewarping algorithm by comparing the event elapsed time with n second threshold times. For example, the control unit 104 compares the event elapsed time from the end of the first event until the second event is received with n 2-1 threshold times to 2-n threshold times, (event elapsed time > 2-n critical time) can be obtained that satisfies the condition. The control unit 104 may determine the block size corresponding to the second threshold time that satisfies the condition.

According to the present embodiment, while using the second dewarping algorithm to quickly respond to an event and provide a corrected image to the user, and when an event does not occur for a certain period of time or longer, the block size is reduced and the second dewarping algorithm is used. A corrected image with reduced distortion may be provided to the user.

14 is a flowchart schematically illustrating an adaptive dewarping method of an image capture device according to another embodiment of the present invention.

The embodiment of FIG. 14 is different from the embodiment of FIG. 13 in that the second dewarping algorithm or the first dewarping algorithm is adaptively selected by considering both the second threshold time and the first threshold time.

Referring to FIG. 14 , the controller 104 of the image capture device 100 receives a view mode setting from a user, starts image processing for generating a correction image from a fisheye image, and when an event is received (S41) , the block size of the second dewarping algorithm can be determined based on the elapsed time of the event (S43). The control unit 104 may generate a corrected image by performing dewarping using the second dewarping algorithm with the block size determined in response to the event (S45). Since step 43 is the same as step 33 in the embodiment of FIG. 13, a detailed description thereof will be omitted.

The control unit 104 compares the elapsed event time with a first threshold time (S47), and if the elapsed event time is longer than the first threshold time, the controller 104 performs dewarping using a first dewarping algorithm in response to an event received thereafter. By doing so, a corrected image can be generated (S49). The first threshold time may be longer than the second threshold time.

According to the present embodiment, a dewarping image with minimized distortion may be provided to a user by generating a dewarping map while adaptively changing a block size according to an event elapsed time.

15 is an example of an adaptive dewarping method of an image capture device according to another embodiment of the present invention.

Referring to FIG. 15 , the controller 104 may receive a view mode setting, etc. (S51), and start image processing for generating a correction image from a fisheye image. Upon receiving the initial event (first event) (S52), the controller 104 may generate a corrected image by performing dewarping using a second dewarping algorithm of the first block size (S53).

The controller 104 sets at least one second threshold time, for example, a 2-1st threshold time to a 2-nth threshold time with respect to the elapsed time of the event, and performs second dewarping for every n second threshold times. You can set the block size of the algorithm. Here, the larger n is, the longer the time, and the smaller the block size.

After the first event ends, the control unit 104 may determine the block size of the second dewarping algorithm by comparing the event elapsed time step by step with n second threshold times.

The control unit 104 compares the elapsed time of the event from the end of the first event with the 2-1 threshold time (S54), and if the event is not received for the 2-1 threshold time or longer, the control unit 104 responds to the second event received thereafter. A corrected image may be generated by performing dewarping using a second dewarping algorithm having a size of 2 blocks (S55). The second block size may be smaller than the first block size.

The control unit 104 compares the elapsed time of the event after the second event ends with the 2-2 threshold time (S56), and if the event is not received for the 2-2 threshold time or longer, the control unit 104 responds to the third event received thereafter. A corrected image may be generated by performing dewarping using a second dewarping algorithm having a size of 3 blocks (S57). The third block size may be smaller than the second block size. The 2-2 threshold time may be longer than the 2-1 threshold time. Between steps 57 and 58, the control unit 104 may reduce the block size of the second dewarping algorithm step by step by comparing the elapsed time of the event with the 2-3 threshold time to the 2-n threshold time. n may be determined according to user settings.

The control unit 104 compares the elapsed time of the event with a first threshold time (S58), and if no event occurs for the first threshold time or more, dewarping is performed using the first dewarping algorithm to generate a corrected image. Yes (S59). When an event occurs within the first threshold time, the controller 104 may generate a corrected image by performing dewarping using a second dewarping algorithm (S53).

In this embodiment, a dewarping map is rapidly generated using a large-sized block of the second dewarping algorithm by applying the block concept and a dewarping image is provided to the user, and then an event from the user does not occur for a certain period of time. In this case, a dewarping image with minimized distortion may be provided to a user by generating a dewarping map while gradually reducing a block size.

In the above-described embodiment, FIGS. 3 to 15 have been described as dewarping in the image acquisition device 100, but the embodiment of the present invention is not limited thereto, and the fisheye image of the image acquisition device 100 is received and monitored. Of course, the same can be applied to dewarping in the apparatus 200.

Embodiments of the present invention provide a dewarping image to the user by generating a dewarping map using the first dewarping algorithm adaptively according to network conditions and user terminal conditions, or block size of the second dewarping algorithm. It is possible to provide a dewarping image to a user by creating a dewarping map while changing the .

Embodiments of the present invention use the second dewarping algorithm only for an image of a corresponding channel in a situation where a dewarping process for generating a new dewarping map is required in a multi-channel environment (eg, user event, change of viewer, etc.) Then, a dewarping image may be provided to the remaining channels by regenerating a dewarping map using the first dewarping algorithm.

The image processing method of the camera system according to the present invention can be implemented as computer readable code on a computer readable recording medium. A computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. In addition, the computer-readable recording medium is distributed in computer systems connected through a network, so that computer-readable codes can be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the present invention can be easily inferred by programmers in the technical field to which the present invention belongs.

The present invention has been described with reference to an embodiment shown in the accompanying drawings, but this is only exemplary, and those skilled in the art can make various modifications and equivalent other embodiments therefrom. You will understand. Therefore, the true protection scope of the present invention should be defined only by the appended claims.

Claims (8)

delete delete Receiving an event including an image processing request for generating a dewarped image from a fisheye image; and
In response to receiving the event, generating the dewarping image by applying a dewarping algorithm to the fisheye image;
In the step of generating a dewarping image by applying the dewarping algorithm,
Blocking the fisheye image into a plurality of blocks;
estimating a transform function in units of blocks through feature point matching between a block of the fisheye image and a corresponding block of the dewarped image;
acquiring 2D coordinates of pixels in a corresponding block of the dewarped image corresponding to pixels in a block of the fisheye image based on the transform function; and
Calculating pixel values of pixels of the dewarped image corresponding to pixel values of pixels of the fisheye image;
The image processing method of the camera system, wherein the block size of the dewarping algorithm is determined based on the elapsed time from the end of the previous event to the reception of the next event. According to claim 3,
The block size is set at at least one threshold time, the elapsed time is compared with the at least one threshold time, and the dewarping algorithm is based on a threshold time that satisfies a condition in which the elapsed time is greater than the threshold time. An image processing method of a camera system further comprising determining a block size. Receiving an event including an image processing request for generating a dewarped image from a fisheye image; and
In response to receiving the event, generating the dewarping image by applying a first dewarping algorithm or a second dewarping algorithm different from the first dewarping algorithm to the fisheye image,
The dewarping image generation step,
generating a dewarping image using the second dewarping algorithm when receiving an initial event; and
and generating a dewarping image using the first dewarping algorithm when an elapsed time from the end of the first event to reception of the next event is longer than a threshold time. Receiving an event including an image processing request for generating a dewarped image from a fisheye image; and
In response to receiving the event, generating the dewarping image by applying a first dewarping algorithm or a second dewarping algorithm different from the first dewarping algorithm to the fisheye image,
The dewarping image generation step,
generating a dewarping image using the first dewarping algorithm when an elapsed time from the end of a previous event to reception of a next event is longer than a first threshold time; and
and generating a dewarping image using the second dewarping algorithm when the elapsed time is shorter than the first threshold time. According to claim 5 or 6,
In the step of generating a dewarping image by applying the first dewarping algorithm,
obtaining 2-dimensional coordinates of pixels of the fisheye image corresponding to pixels of the dewarped image in a 3-dimensional space in units of pixels; and
and calculating pixel values of pixels of the dewarped image corresponding to pixel values of pixels of the fisheye image. According to claim 5 or 6,
In the step of generating a dewarping image by applying the second dewarping algorithm,
Blocking the fisheye image into a plurality of blocks;
estimating a transform function in units of blocks through feature point matching between a block of the fisheye image and a corresponding block of the dewarped image;
obtaining 2D coordinates of pixels in a corresponding block of the dewarped image corresponding to pixels in a block of the fisheye image based on the transform function; and
and calculating a pixel value of a pixel of the dewarped image corresponding to a pixel value of a pixel of the fisheye image.

Images