JP5225047B2 - Tracking imaging apparatus, tracking imaging method, and computer program - Google Patents

Description

  The present invention relates to a tracking imaging device, a tracking imaging method, and a computer program, and is particularly suitable for tracking a target object in an image captured by an imaging unit.

  2. Description of the Related Art Conventionally, there is an image pickup apparatus that includes a drive device that can turn about two axes of a pan axis and a tilt axis as a rotation axis, and that can move an imaging direction (optical axis direction) in a desired direction. . Such an imaging device is used for a surveillance camera or the like. As such an imaging apparatus, there is an imaging apparatus having an automatic tracking function. Specifically, an imaging apparatus having an automatic tracking function detects a moving object such as a target person based on a captured image signal, and the optical axis direction is always set so that the detected moving object is always within the imaging range. Pan / tilt drive control is performed to track a moving object.

  Some imaging apparatuses having such an automatic tracking function predict the moving direction and speed of a moving object to be tracked in order to improve the tracking performance of the moving object. In Patent Document 1, the position of the moving object is converted into absolute coordinates from the position in the captured image of the moving object to be tracked and the pan / tilt angle when the captured image is captured, and this is converted in time series. Remember it. Then, based on the “absolute coordinates of the position of the moving object” stored in time series in this way, the moving direction and speed of the moving object are estimated, and the estimated result is reflected in the next pan / tilt driving angle. Do.

Also, some video cameras and digital cameras that do not have a pan / tilt drive mechanism track a moving object in the imaging region and move the tracking visual field region in the imaging region (Patent Document 2, Patent Document). 3). The image signal of the tracking visual field area to be moved within the imaging area is used for automatic focus detection (autofocus) processing or the like.
Note that there are cases in which correlation information obtained by two or more captured images at different points in time is used for extraction of a moving object in a captured image.

Japanese Patent No. 3440916 Japanese Patent Publication No. 5-60712 Japanese Patent No. 3449489

  By the way, if the moving object to be tracked is simply “moving object”, a relatively simple image area detection process such as using correlation information from two or more captured images may be performed. Good. However, in applications such as surveillance cameras, if you want to detect a specific moving object such as a "person" or "animal" that is moving (or may have stopped temporarily) and want to track it There are many.

  The processing cost of the object detection process for detecting such a specific moving object from the captured image is usually greater than the process using the correlation information described above. Therefore, in order to improve the response of detection of a moving object, it is conceivable to limit the area where object detection processing is performed to only a part of the captured image. In a tracking imaging apparatus including a pan / tilt mechanism and an automatic tracking function, it is desirable that an object to be tracked exists as close to the center of the captured image as possible. For this reason, a part of the area where the object detection process is performed is often set near the center of the captured image.

  When the object detection processing performance in the restricted area is sufficiently high and the pan / tilt drive mechanism has sufficient response performance, the tracking target captured near the center of the captured image The object can be imaged while being held near the center of the captured image. In other words, by performing pan / tilt drive based on the relative position of the object from the center of the captured image (screen), it is possible to continue capturing an image with the tracking target object held near the center of the captured image. is there.

  However, depending on the performance of the object detection processing in the restricted area, the performance of the pan / tilt drive mechanism, and the movement speed of the tracking target object, the tracking target object may be lost and tracking of the object may be interrupted. Can happen. That is, although the object to be tracked is captured in the captured image (captured image at the next time point) after performing object detection processing and pan / tilt drive based on the captured image at a certain time point, The object may be out of the area where the object detection process is performed. Then, the tracking target object is lost and tracking of the object is interrupted.

  Even if the movement speed and position of the tracking target object can be estimated as described in Patent Document 1 described above, the pan / tilt drive mechanism determines that the tracking target object movement speed and position are the same. If it does not have the ability to follow, the same problem as described above occurs.

  The present invention has been made in view of such problems, and it is intended to improve the object tracking performance when the object detection processing is reduced by limiting the object detection area as compared with the prior art. Objective.

The tracking imaging apparatus according to the present invention is configured to change the direction of the imaging optical axis in order to track an imaging unit that captures an image in the direction of the imaging optical axis and a target object in the image captured by the imaging unit. A driving unit that drives the imaging unit, a region setting unit that sets a detection processing region in a partial region of an image captured by the imaging unit, and an image in the detection processing region set by the region setting unit and the object position detecting means for detecting information indicating a relative position from the predetermined reference point in the image area to be captured by the imaging unit as position information of the target object in the detection processing area, the object position Area position calculation for calculating the position of the detection processing area at the time of the next shooting based on the relative position from the reference point which is the position information of the target object detected by the detection means And after the driving of the imaging means by the driving means is completed based on the position information of the target object detected by the object position detecting means, the reference point matches or approaches the relative position. A driving amount calculating unit that calculates a driving amount for driving the imaging unit; and a ratio determining unit that determines a predetermined ratio as a value corresponding to the driving amount calculated by the driving amount calculating unit , setting means, based on the position calculated by the area position calculation means sets the detection processing region in the next imaging, the area position calculation means, the detection processing region in the current imaging, the A position is calculated by moving the distance from the reference point to the relative position by a distance obtained by multiplying the length from the reference point to the relative position by the predetermined ratio .

In the tracking imaging method of the present invention, the imaging step of capturing an image in the direction of the imaging optical axis by the imaging unit, and the direction of the imaging optical axis in order to track the target object in the image captured by the imaging step. A driving step for driving the imaging means to change, a region setting step for setting a detection processing region in a partial region of the image captured by the imaging step, and a detection processing region set by the region setting step. An object position detecting step of processing an image and detecting information indicating a relative position from a reference point set in advance in the image area captured by the imaging step as position information of the target object in the detection processing area ; Based on the relative position from the reference point that is the position information of the target object detected by the object position detection step, An area position calculating step for calculating the position of the detection processing area and the position information of the target object detected by the object position detecting step, and after the driving of the imaging means by the driving step is completed, the reference A drive amount calculation step for calculating a drive amount for driving the imaging means so that a point coincides with or approaches the relative position, and a predetermined ratio as a value corresponding to the drive amount calculated by the drive amount calculation step and a rate determining step of determining a, in the area setting step, based on the position calculated by the area position calculation step, setting the detection processing area at the time of next imaging calculation step the area position Then, the detection processing area at the time of the current photographing is moved from the reference point to the relative position and from the reference point to the relative position. And calculates the position is moved by a distance obtained by multiplying the predetermined ratio to the length of up location.

The computer program of the present invention drives the imaging unit to change the direction of the imaging optical axis in order to track the target object in the image captured by the imaging unit that captures an image in the direction of the imaging optical axis. A driving step for instructing the driving means, an area setting step for setting a detection processing area in a partial area of the image captured by the imaging means, and an image of the detection processing area set by the area setting step. treated, and the object position detecting step of detecting information indicating a relative position from the predetermined reference point in the image area to be captured by the imaging unit as position information of the target object in the detection processing area, the object Based on the relative position from the reference point, which is the position information of the target object detected by the position detection step, the detection at the next shooting is performed. After the driving of the imaging means by the driving step is completed based on the position information of the target object detected by the region position calculating step for calculating the position of the processing region and the object position detecting step, the reference point is A driving amount calculating step for calculating a driving amount for driving the imaging unit so as to coincide with or approach the relative position, and a predetermined ratio is determined as a value corresponding to the driving amount calculated by the driving amount calculating step. to execute the percentage determining step in the computer, in the region setting step, based on the position calculated by the area position calculation step, setting the detection processing region in the next image pickup, by the area position calculation step The detection processing area at the time of the current shooting is moved from the reference point in the direction of the relative position. And calculates the position is moved by a distance obtained by multiplying the predetermined ratio in length up to the relative position.

According to the present invention, it can be changed according to the position of the target object positions of the detection processing area. As a result, even if the region for detecting an object is limited and the load of the object detection process is reduced, the tracking performance of the object can be improved as compared with the related art.

(First embodiment)
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram illustrating an example of a hardware configuration of the tracking imaging device. In FIG. 1, in the present embodiment, the tracking imaging device includes an imaging device 100 and a controller device 110.
The imaging apparatus 100 includes a camera apparatus 101 that converts an optically captured image into an electrical image signal, and a camera platform 105 that drives the camera apparatus 101 in the pan direction and the tilt direction. The pan head 105 is provided with a pan / tilt drive mechanism (pan axis drive motor and tilt axis drive motor). The pan axis drive motor is for rotating the direction of the imaging optical axis 102, which is the direction in which the camera apparatus 101 captures an image, within a predetermined angle range about the pan axis 103 (rotation axis). The tilt axis drive motor is for rotating the direction of the imaging optical axis 102 within a predetermined angle range around the tilt axis 104 (rotation axis). Note that a stepping motor, an ultrasonic motor, or the like is used as the pan axis drive motor and the tilt axis drive motor, and is controlled by a pan / tilt drive control unit 117 described later.

The controller device 110 has a configuration in which each device is connected to the system bus 111. The CPU 112 operates based on a control program stored in a non-volatile memory device 113 such as a hard disk or a flash ROM, and comprehensively controls each device connected to the system bus 111.
The volatile memory (RAM 114) is used as a work area when the CPU 112 operates. In addition, an image buffer 115 is included in a partial area of the RAM 114. In the present embodiment, the case where the work area of the CPU 112 and the image buffer 115 are included in the same RAM 114 is shown as an example. However, these may be divided into separate RAMs. Further, the image buffer 115 may be built in an image acquisition unit 118 or an object detection processing unit 119 described later.

The user I / F device 116 includes a display device (not shown) and a key input device, and notifies the user of information such as the status of the device according to a command from the CPU 112. In addition, the user I / F device 116 receives commands from the user, such as imaging, detection of a tracking target, and start / stop of a tracking operation, and transmits them to the CPU 112.
The pan / tilt drive control unit 117 is connected to the pan axis drive motor and the tilt axis drive motor of the imaging apparatus 100. The pan / tilt drive control unit 117 includes a pulse generation circuit, a driver circuit, and the like (not shown), and instructs the imaging apparatus 100 to perform desired pan / tilt drive based on a command from the CPU 112. The pan / tilt drive control unit 117 acquires current pan and tilt angle position information based on an output from a position sensor (encoder or the like) (not shown) built in the imaging apparatus 100 and notifies the CPU 112 of the position information. . Note that the pan / tilt drive control unit 117 does not have to be directly connected to the system bus 111 as in the present embodiment. In other words, the pan / tilt drive control unit 117 is provided as an independent casing outside the controller device 110, and the pan / tilt drive control unit 117 and the CPU 112 are mutually connected via a predetermined I / F such as a serial or network line. It can also be configured to communicate with.

  The image acquisition unit 118 is connected to the camera device 101 of the imaging device 100. The image acquisition unit 118 receives an instruction from the CPU 112, performs imaging control of the camera device 101, and acquires an image in the imaging range in the direction of the imaging optical axis 102. The image signal acquired by the image acquisition unit 118 is stored in the image buffer 115 as image data. Note that the image buffer 115 has a capacity capable of storing image data of a plurality of imaging ranges. Each of a plurality of image data is called a page. The image buffer 115 can be accessed not only by the image acquisition unit 118 but also by the CPU 112 and an object detection processing unit 119 described later. Access to the image buffer 115 from these plural devices is exclusively controlled by the bus arbiter.

  The object detection processing unit 119 performs processing for detecting the tracking target object 106 (target object) in the image data (in the image), and is configured by image processing hardware. The object detection processing unit 119 includes an optimal detection algorithm such as a neural network that detects a person according to the characteristics of the object 106. Further, the object detection processing unit 119 includes a processing region setting unit 120 so that only a partial region of the image data of a specific page can be designated as a detection processing region for performing the detection processing of the object 106. It is configured. The processing area setting unit 120 is a hardware register, and its value can be set by the CPU 112. In addition to this, the object detection processing unit 119 includes a register (not shown) and an interrupt output connected to the CPU 112. The object detection processing unit 119 receives a start command for detecting the object 106 from the CPU 112. Further, the object detection processing unit 119 notifies the CPU 112 of the completion of detection of the object 106 by interruption, stores the detection result of the object 106 in the register as a coordinate value in the captured image, and notifies the CPU 112 of the coordinate value. Have. In the present embodiment, a case where the object detection processing unit 119 is configured using a dedicated hardware detection circuit is shown as an example. However, when the CPU 112 is sufficiently powerful, the object detection processing unit 119 may be implemented as a software program that is operated by the CPU 112. Further, the object detection processing unit 119 may be configured using a relatively general-purpose image processing chip such as a DSP (Digital Signal Processor).

Next, an example of basic processing of the tracking control program operated by the CPU 112 will be described with reference to the flowchart of FIG.
The CPU 112 activates a tracking control program for performing the processing shown in FIG. 2 by using a command from the user I / F device 116 or power-on of the controller device 110, a command through a network line (not shown), or the like as a trigger. When started, the tracking control program performs predetermined initial settings for each device / processing unit connected to the system bus 111.
First, in step S201, the CPU 112 performs initial setting of a detection processing area in an image area to be captured. FIG. 3 is a diagram illustrating an example of an initial detection processing area.
As shown in FIG. 3, the initial detection processing area 302 is arranged as an area having a predetermined size (S _w , S _h ) centering on the center position C of the entire imaging area 310. Here, S _w is the width of the detection processing area 302, and S _h represents the height of the detection processing area 302. In the present embodiment, the size (S _w , S _h ) of the detection processing area is always constant regardless of the position of the detection processing area.

When the object tracking process is performed, the CPU 112 normally manages various coordinates with the center position C as a reference position (that is, the origin (0, 0)). However, in the circuit of the object detection processing unit 119 in the present embodiment, the position of the upper left end point Z of the entire imaging region 310 is the origin (0, 0). Therefore, the CPU 112 needs to set the processing area setting unit 120 after converting the coordinates. The CPU 112 sets the start position (a _x , a _y ) and the size (S _w , S _h ) of the detection processing area 302 in a predetermined register as settings of the processing area setting unit 120. In the present embodiment, CPU 112 is, the following equation (1), as (2), the starting position of the initial detection processing region 302 (a _x, a _y) by setting the initial detection process The center point of the region 302 is made to coincide with the center position C.
a _x = (I _w −S _w ) / 2 (1)
a _y = (I _h −S _h ) / 2 (2)
Here, I _w is the width of the entire imaging region 310, and I _h is the height of the entire imaging region 310.

  When the initial detection processing area 302 has been set as described above, the process of step S202 is performed. Thereafter, unless an end trigger such as a command from the user is input, the loop of steps S202 to S209 is repeatedly executed. It should be noted that the processing up to step S201 is automatically performed when the controller device 110 is turned on, and when a certain trigger is given to the controller device 110, the tracking state of the object 106 is changed to the processing from step S202. May be started.

  In step S <b> 202, the CPU 112 instructs the image acquisition unit 118 to execute an operation for acquiring an image captured by the camera device 101. This command is executed by the CPU 112 writing a value (command command) in a command register provided in the image acquisition unit 118. Then, the image acquisition unit 118 stores the image data in a predetermined page area of the image buffer 115 when the acquisition of the image is completed based on a command from the CPU 112. Thereafter, the image acquisition unit 118 notifies the CPU 112 of completion of acquisition of image data.

When the CPU 112 receives the image data acquisition completion notification, the CPU 112 instructs the object detection processing unit 119 to execute object detection processing in step S203. This command is made by writing a value in a command register provided in the object detection processing unit 119. In addition, the CPU 112 notifies the object detection processing unit 119 of information (page or address) related to the area of the image buffer 115 in which the image data to be processed this time is stored together with this command. The object detection processing unit 119 reads out image data of a part of the region based on the setting value of the processing region setting unit 120 from the image data in the region of the predetermined page of the image buffer 115, and uses, for example, a neural network to Perform detection processing. If an object is detected as a result of this object detection processing, the object detection processing unit 119 sets a detection success flag in the result storage register, and sets the position coordinate of the detected object in the coordinate register and the upper left end point Z. Write as origin. On the other hand, when the object detection fails, the object detection processing unit 119 sets a detection failure flag. In either case, the object detection processing unit 119 generates an interrupt signal to notify the CPU 112 of the completion of the detection process.
As described above, in the present embodiment, for example, an example of the object position detection unit is realized by performing the process of step S203.
In step S204, the CPU 112 checks the detection process result flag of the object detection processing unit 119. If the detection is successful, the process of step S206 is executed. If the detection is unsuccessful, the process of step S205 is executed.

  If the tracking target object has been successfully detected, the CPU 112 performs the following process in step S206. That is, the CPU 112 drives each of the pan axis drive motor and the tilt axis drive motor so that the center of the imaging optical axis 102 of the camera device 101 (center position C in FIG. 3) matches the relative position of the tracking target object. The amount (drive amount in the pan and tilt directions) is calculated. In this embodiment, the case where the center of the imaging optical axis 102 of the camera apparatus 101 (center position C in FIG. 3) is made to coincide with the relative position of the object to be tracked is shown as an example. However, this is not always necessary. In other words, the center of the imaging optical axis 102 of the camera device 101 (center position C in FIG. 3) may be brought closer to the relative position of the tracking target object.

FIG. 4 is a diagram illustrating an example of the state of the tracking target object in the imaging region. Specifically, FIG. 4A shows a state in which an object 106 (here, a person) to be tracked is detected in the detection processing area 302 in the imaging area 401. The object detection processing unit 119 outputs coordinates with the upper left upper point Z as the origin as the position of the tracking target object 106. As described above, the CPU 112 converts this into coordinates having the origin at the center position C of the imaging area 401 (that is, the position matching the imaging optical axis). That is, the position of the tracking target object 106 is represented by a position vector D = (D _x , D _y ) from the center position C.
Thus, in the present embodiment, for example, the center position C of the imaging region 401 is an example of a reference point, and the position represented by the position vector D from the center position C is an example of a relative position from the reference point.

Here, assuming that the driving amounts in the pan and tilt directions necessary to move the center position C of the imaging region 401 to the position of the position vector D are M _p and M _t , respectively, the driving amounts M _p and _P in the pan and tilt directions M _t may be considered to be substantially proportional to the x component and the y components D _x and D _y of the position vector D, respectively. Accordingly, the driving amounts M _p and M _t in the pan and tilt directions are simply expressed by the following equations (3) and (4), respectively.
M _p = C _p · D _x (3)
M _t = C _t · D _y (4)
Here, C _p and C _t are constants that can be determined by the viewing angle and resolution of the camera device 101 and the movement angle corresponding to the drive amount specified for each of the pan axis drive motor and the tilt axis drive motor. It is.
As described above, in the present embodiment, for example, an example of the drive amount calculation unit is realized by performing the process of step S206.

When the driving amounts M _p and M _t in the pan and tilt directions are calculated as described above, the CPU 112 performs a PT drive control process in step S207. This PT drive control process is performed by the CPU 112 writing a value in a register included in the circuit of the pan / tilt drive control unit 117. Here, the values transmitted to the pan / tilt drive control unit 117 are the driving amounts M _p and M _t in the pan and tilt directions calculated in step S206. In addition, when the pan / tilt drive mechanism (pan axis drive motor and tilt axis drive motor) starts to be driven, the CPU 112 writes a value in a command register included in the circuit of the pan / tilt drive control unit 117 to perform pan / tilt drive. This is transmitted to the control unit 117.

The pan / tilt drive control unit 117 controls the pan / tilt drive mechanism (pan axis drive motor and tilt axis drive motor) of the imaging apparatus 100, and only an angle corresponding to the drive amounts M _p and M _t specified by the CPU 112. The camera device 101 is turned. When the turning operation of the camera apparatus 101 is completed, the pan / tilt drive control unit 117 notifies the CPU 112 of the completion of the turning. The notification of the completion of the turn is made by interruption, as in the processing in other circuits. When the pan / tilt drive control unit 117 is configured to be outside the controller device 110 via a predetermined communication I / F, information exchange between the pan / tilt drive control unit 117 and the CPU 112 is performed. What is necessary is just to carry out with the command according to a predetermined communication protocol.
As described above, in the present embodiment, for example, an example of a driving unit is realized by performing the process of step S207.

  When the PT drive control process in step S207 is completed as described above, ideally, the image of the imaging region is an object to be tracked at the center position C of the imaging region 401 as shown in FIG. It is desirable that is located. When the pan / tilt driving mechanism is sufficiently responsive, the tracking target object 106 can be positioned at the center position C of the imaging region 401 as shown in FIG. However, in practice, it is technically and costly difficult to provide such a highly responsive drive mechanism. Therefore, the object 106 to be tracked also moves until the driving of the pan / tilt driving mechanism is completed. Depending on the moving speed of the object 106 at this time, after the driving of the pan / tilt driving mechanism is completed, the object 106 is within the imaging area 401 as shown in FIG. It may happen that the area 302 is out of the state. Even if the next image is acquired as it is and the object detection process is executed, the tracking imaging apparatus loses sight of the object 106 to be tracked and cannot continue tracking the object 106.

Therefore, in the present embodiment, in step S208, the CPU 112 calculates the position of the next detection processing area 302. If a series of processing cycles for each time is sufficiently short when the detection and tracking operations are continuously performed, the movement direction of the tracking target object 106 in the imaging region 401 is the pan / tilt movement direction. In many cases, it is considered that they generally coincide with each other. Accordingly, as shown in FIG. 4D, the CPU 112 uses the same position vector D as used in step S206, and a position obtained by multiplying the center position C of the imaging region 401 by the ratio α in the direction of the position vector D. Is calculated to match the center position of the detection processing area. This ratio α will be described in detail later.
As described above, the processing area setting unit 120 needs to set the coordinates (a _x , a _y ) of the start position of the detection processing area 302 with the upper left end point Z as the origin. Therefore, the CPU 112 calculates the coordinates (a _x , a _y ) of the start position of the next detection processing area 302 according to the following expressions (5) and (6).
a _x = (I _w −S _w ) / 2 + α · D _x (5)
a _y = (I _h −S _h ) / 2 + α · D _y (6)
As described above, in the present embodiment, the detection processing region 302 is calculated by the distance obtained by multiplying the length of the position vector D by the ratio α in the direction of the position vector D by calculating the expressions (5) and (6). Can be moved.
As described above, in the present embodiment, for example, an example of the region position calculation unit is realized by performing the process of step S208.

Next, in step S209, the CPU 112 sets the coordinates (a _x , a _y ) of the start position of the next detection processing area 302 calculated in step S208 in a register included in the processing area setting unit 120. As described above, in the present embodiment, the size (S _w , S _h ) of the detection processing region 302 is a fixed value. Accordingly, in step S209, the CPU 112 sets the same value as the value set in step S201 as the size (S _w , S _h ) of the next detection processing area 302. Then, returning to the process of step S202, a series of processes are repeated from the acquisition of the image.
As described above, in the present embodiment, for example, an example of an area setting unit is realized by performing the process of step S209.

  If it is determined in step S204 that detection of the tracking target object 106 has failed, the process proceeds to step S205, and the CPU 112 sets an initial detection processing area 302. In step S205, exactly the same processing as that described in step S201 is performed. In other words, if the tracking target object 106 is lost due to the tracking target object 106 going out of the imaging range, the object detection processing area 302 is returned to the center position C of the imaging area 401. .

  In the present embodiment, if an object cannot be detected by a single object detection process, it is considered that the tracking target object 106 has been lost. However, this is not necessarily required. For example, depending on the accuracy of the object detection process, for example, when the object is not detected a plurality of times in succession, it is assumed that the object 106 to be tracked is lost, and the process of returning the detection processing area 302 to the initial state is performed. May be. In this case, for example, the following process can be considered as a process that replaces steps S206 to S209 in the detection failure state until it is assumed that the tracking target object 106 has been lost. In other words, several variations such as “no pan / tilt drive or change of detection processing area”, “pan / tilt drive is performed only as before, but the detection processing area is returned to the center position C of the imaging area”, etc. Can be considered. Among these variations, the optimum one is selected according to the situation in which the tracking imaging device is used.

Further, when the object 106 to be tracked is lost, not only the detection processing area 302 is returned to the initial state but also the direction of the imaging optical axis 102, that is, the pan / tilt angle is returned to the home position. Also good. Further, when it can be expected that the object 106 is detected again in the vicinity of losing sight of the tracking target object 106, the detection processing region 302 and the direction of the imaging optical axis 102 are tracked without performing anything. You may make it hold | maintain in the state at the time of losing sight of the target object 106. FIG.
Further, no matter how fast the tracking target object 106 moves, if the detection processing area 302 is placed at a position that protrudes from the imaging area 401, it is meaningless because it only narrows the detectable range. Accordingly, in the present embodiment, the start position of the detection processing region 302 coordinates (a _x, a _y) limits the movement range of the detection processing region 302 to fit in the range represented by the following equation (7) ing.
_{(0,0) ≦ (a x,} a y) <(I w -S w, I h -S h) ··· (7)

Here, the ratio α of shifting the next detection processing area 302 in the direction of the position vector D indicating the detection position of the tracking target object 106 will be described. In the present embodiment, the CPU 112 calculates the value of the ratio α for shifting the next detection processing area 302 according to the following equation (8) every time the process of step S208 is performed, and the ratio for shifting the next detection processing area 302 α is recalculated.
α = β · | D | (8)
Here, β is a constant value determined in advance according to the driving amounts M _p and M _t in the pan and tilt directions. That is, the ratio α for shifting the next detection processing area becomes a large value when the driving amounts M _p and M _{t in} the pan and tilt directions are large, and becomes a small value when the driving amounts are small. That is, when the object tracking process is continuously performed and the movement of the object is fast, the movement amounts M _p and M _{t in the} pan / tilt direction for one detection of the object are also obtained. growing. On the other hand, when the process of tracking an object is continuously performed and the movement of the object is slow, the driving amounts M _p and M _{t in} the pan / tilt direction are also reduced. In general, if the drive amounts M _p and M _t in the pan / tilt direction are large, the time until the drive is completed also becomes long. Therefore, the faster the tracking target object 106 moves, the more the tracking target object 106 moves to the imaging region 401 as shown in FIG. 4C until the driving of the pan / tilt driving mechanism is completed. You will go farther from the center. Therefore, in such a case, the detection processing region 302 is moved greatly by setting the ratio α for shifting the next detection processing region large according to the equation (8). By doing so, it is less likely that the tracking image capturing apparatus once captures the object 106, and good tracking performance is exhibited.

Of course, the shorter the one-cycle processing from the detection of the tracking target object 106 to the completion of the driving of the pan / tilt driving mechanism, the better and more smoothly the tracking operation can be performed. Since the processing load of the object detection processing increases as the area (number of pixels) of the image data to be processed increases, the limitation of the detection processing region 302 shown in this embodiment also contributes to shortening the processing time of one cycle. doing.
As described above, in the present embodiment, for example, an example of the movement amount determination unit is realized by performing the process of step S208.

Depending on the situation in which the tracking imaging device is used, the ratio α for shifting the next detection processing area 302 depends on the response speed of the pan / tilt drive mechanism and the average moving speed of the object to be tracked. Even if the fixed value is always adjusted in advance, sufficient performance may be exhibited. As a more specific example, as shown in FIG. 5, there may be a case where the ratio α for shifting the next detection processing area can always be set to 100%.
Thus, while the object 106 to be tracked is continuously detected and the object 106 is being tracked, setting the next detection processing area 302 based on the position vector D of the object This is extremely effective for continuing the tracking of the object 106. However, from the state in which the tracking target object 106 cannot be detected, only the first time when the detection of the object 106 is successful is performed by performing the process of moving the next detection processing area 302 in this way. 106 may be easily lost.

  In the state where the tracking target object 106 is not detected, the process of returning to step S202 through steps S204 and S205 is repeated as described above. At this time, if the tracking target object 106 is detected in the next object detection processing (step S203), this object 106 is the frame of the detection processing region 302 whose center is set at the center position C of the imaging region 401. It has moved from outside into the detection processing area 302. When the “state in which the object 106 is not detected” immediately before the object detection process is continued, the PT drive control process (step S207) is not performed. Therefore, the repetition cycle for executing the object detection process in step S203 is faster than when the tracking target object 106 is detected. Here, it is assumed that the tracking target object 106 is detected at the position indicated by the position vector D in FIG. 4A in the repetition of the high-speed object detection processing, for example. In this case, it is difficult to consider that the tracking target object 106 has entered the detection processing region 302 from a direction close to the direction opposite to the direction indicated by the position vector D (right side in the figure). Rather, the tracking target object 106 is considered to have entered the detection processing area 302 from a direction close to the direction indicated by the position vector D (left side in the figure). It is also highly likely that the tracking target object 106 has moved in a direction substantially opposite to the direction indicated by the position vector D. For example, if the tracking target object 106 has moved as indicated by the position vector W in FIG. 6, a process of moving the detection processing area 302 is performed as shown in the equations (5) and (6). As a result, the object 106 is lost.

  Therefore, in step S208 of the present embodiment, the CPU 112 performs a process for determining whether or not it is the first object detection. Then, the next detection processing area 302 is set only when the detection target area 106 is not detected in the detection processing area and the state transitions to a state where the detection is successful and the first object detection is performed. Step S209 is omitted. In this case, for example, the detection processing area 302 is left at the center position in the initial state. However, it is not always necessary to leave the detection target area at the center position in the initial state. For example, since there is a high possibility that the object is moving in the direction indicated by the movement vector W, the object is slightly shifted in the opposite direction to the latest position vector D obtained in step S203 (the ratio α is set to a negative predetermined value). If this is done, the effect of making it difficult to lose sight of the object 106 can be expected.

  In this embodiment, the PT driving process in steps S206 and S207 and the detection process area setting process in steps S208 and S209 are performed sequentially. However, this is not necessarily required. For example, these may be processed by independent parallel processors such as tasks and processes operating on the OS and may be processed in parallel. Further, the image acquisition processing in step S202 and the object detection processing in step S203 are pipelined so that the image acquisition is performed even while the pan / tilt driving mechanism is being driven to further increase the speed. Is also possible.

  As described above, in this embodiment, the next detection processing area 302 is moved to a position obtained by multiplying the position vector D of the “tracking target object 106” detected in the detection processing area 302 by the ratio α. . That is, the current position information in the captured image of the tracking target object 106 is used to set the next detection processing area 302 after predicting the position to which the tracking target object 106 moves at the next detection. I did it. Therefore, it is possible to exhibit good tracking performance while reducing the load of detection processing of the tracking target object 106. In particular, even when the response speed of the pan / tilt drive mechanism is temporarily insufficient with respect to the moving speed of the object 106 to be tracked, the tracking of the object 106 is interrupted by losing sight of the object 106. The possibility of having to do can be reduced. As a result, it is not necessary to make the performance of the pan / tilt driving mechanism and the object detection processing unit 119 excessive, and a cost reduction effect can be expected.

(Second Embodiment)
Next, a second embodiment of the present invention will be described. In this embodiment, in the process corresponding to step S208 of the first embodiment described above, a process that takes into account changes in the moving speed and direction of the tracking target object is performed. As described above, the present embodiment is different from the first embodiment described above mainly in part of the processing in step S208 in FIG. For example, the hardware configuration of the tracking imaging apparatus of the present embodiment is the same as that in FIG. 1, and the basic processing flow of the tracking imaging apparatus of the present embodiment is the same as that in FIG. Therefore, in the description of the present embodiment, the same parts as those in the first embodiment described above are denoted by the same reference numerals as those in FIGS.

  7 and 8 are diagrams illustrating first and second examples of a method for estimating the speed of the tracking target object 106. In FIGS. 7 and 8, it is assumed that the object detection continues and succeeds (tracking operation continuation state).

FIG. 7A is a diagram illustrating an example of an image captured in the (n−1) cycle (n is a natural number of 2 or more) of the repetition cycle of steps S202 to S209 in the flowchart described in FIG. At this time, the tracking target object is detected at the position indicated by the position vector D _n−1 . Accordingly, the pan / tilt driving mechanism is driven so that the position indicated by the position vector D _n-1 at this time coincides with the center position C of the imaging region 401.

Then, in the next n-th cycle, as shown in FIG. 7 (b), the position indicated by the position vector D _n, the object 106 of the tracking target is detected. That is, if the time required from the time when the image shown in FIG. 7A is taken to the time when the image shown in FIG. 7B is taken is t _n−1 , the tracking target object 106 is: During time t _n−1 , the position vector D _{n has been} advanced. Therefore, the velocity vector v _n at this time may be regarded as substantially (9).
v _n = D _n / t _n-1 (9)
During time t _n−1 , the position vector D _n−1 of the tracking target object 106 is detected (step S203), and the pan / tilt driving mechanism driving process corresponding to the position vector D _n−1 (step S203). Steps S206 and S207) and various settings necessary for the next imaging are performed.

Similarly, it can be considered that the velocity vector v _n−1 at the (n−1) cycle is as shown in the following equation (10).
v _n-1 = D _n-1 / t _n-2 (10)
Furthermore, it is estimated that it is necessary to perform the driving process of the pan / tilt driving mechanism corresponding to the position vector D _n in the elapsed time from the time of imaging shown in FIG. 7B until the position vector D _n is obtained. Then, a time obtained by adding a processing time such as necessary setting thereafter is defined as t _n . That is, the time required from the time when the current shooting is performed until the next shooting is started is _denoted by t _n .

Since the time before the time t _n-1 is the actual time of the processing operation that has already been completed, the time actually taken is stored and used. On the other hand, since the time t _n is a time required for an operation to be performed in the future, such estimation calculation is performed. Of the elements that determine the time t _n , the time required to perform the drive processing of the pan / tilt drive mechanism is the response performance of the pan / tilt drive mechanism and the drive amount (movement amount) M _p in the pan / tilt direction. obtained by calculating by using the M _t. In this embodiment, the required time corresponding to the driving amounts M _p and M _t in the pan and tilt directions is stored in the nonvolatile memory device as a LUT (Look up table). The CPU 112 retrieves the required time corresponding to the driving amounts M _p and M _t corresponding to the position vector D _n from the LUT independently for each of the pan direction and the tilt direction. The CPU 112 acquires the required time in the pan direction and the tilt direction from the result of the search, and uses the longer time of the acquired required times (the longer time required for pan and tilt) to obtain the time t. Set _n . The reason why the longer one of the required times in the pan direction and the tilt direction is the longest is that the pan and tilt drive processes are started simultaneously.
Of course, if the subsequent calculation and setting processes are in time, the actual time actually taken from the imaging time of FIG. 7B is acquired after the driving process of the pan / tilt driving mechanism corresponding to the position vector D _n is completed. and it may be the time t _n by adding subsequent processing time estimated amount to the actual time.

Then, as shown in FIG. 7C, by using these values, the velocity vector v _{n +} of the tracking target object 106 is not detected until the next cycle of image capturing (that is, during time t _n ). Estimate ₁ . Then, using the estimated velocity vector v _{n + 1} , the “position vector D _{n + 1} in the next cycle of the tracking target object 106” on the imaging region 401 is estimated. FIG. 7 (c) is a diagram schematically showing the position calculation when each imaging interval is always a unit time in order to grasp the image of the estimation method. The interval varies. Therefore, a method for estimating the position vector D _{n + 1} of the object 106 at the next imaging will be described in detail with reference to FIG.

From the state of FIG. 7A to the state of FIG. 7B, the velocity vector of the tracking target object 106 changes from v _{n −1} to v _n . Since the time required at this time is t _n−1 , the speed change amount (average acceleration vector) g _n−1 per unit time is expressed by the following equation (11).
g _n-1 = (v _n −v _n−1 ) / t _n−1 = (D _n / t _n−1 −D _n−1 / t _n−2 ) / t _n−1 (11)
That is, it can be seen that the amount of change in speed per unit time (average acceleration vector) gn _-1 changes as shown in FIG. Thus, in order to obtain “the velocity vector v _{n + 1} of the tracking target object 106” at the time of the next imaging (cycle), the time t _n described above and the average acceleration vector g _{n at} this time are known. good. Therefore, in this embodiment, the CPU 112 considers that the average acceleration vectors g _n and g _n−1 are equal, and “the velocity vector v _{n +} of the tracking target object 106 at the time of the next imaging according to the following equation (12)”. Calculate (estimate) ₁ ".
v _{n + 1} = v _n + g _n · t _n (12)

Further, the CPU 112 calculates (estimates) the “position vector D _{n + 1 of the} tracking target object 106” at the time of the next shooting using the velocity vector v _{n + 1} according to the following equation (13).
D _{n + 1} = v _{n + 1} · t _n (13)
The average acceleration g _n-1 is recalculated every time the detection processing area is set, so that no error is accumulated in the average acceleration g _n-1 .
Then, as shown in FIG. 7D, the CPU 112 next time so that the position obtained by multiplying the position vector D _{n + 1} thus obtained by the ratio α described in the first embodiment is the center. The position of the detection processing area 302 is calculated and set.

FIG. 8 shows an example of calculation of the detection processing area when the tracking target object 106 moves differently from that shown in FIG. 7, but the method is the same as that shown in FIG. Therefore, detailed description is omitted. As in the present embodiment, by estimating the “velocity at the next imaging” of the object 106 to be tracked and calculating the position of the next detection processing area 302, the object 106 to be detected is shown in FIG. Even if it moves as shown, the next detection processing area 302 can be set appropriately. That is, as shown in FIG. 8A and FIG. 8B, when the object 106 is stopping (or when the object is going to change the traveling direction in the opposite direction from that which has been progressing until now). However, the next detection processing area 302 can be set at a suitable position as shown in FIG.
As described above, in the present embodiment, for example, in step S208, the CPU 112 performs the calculations of Expressions (11) to (13), thereby realizing an example of the moving speed estimation unit. In the present embodiment, for example, the position vectors D _n and D _n-1 are examples of position information of the target object that has already been detected, and the times t _n-1 and t _n-2 are times indicating the imaging interval. An example of information.

  As described above, in the present embodiment, the current and past position information in the captured image of the tracking target object 106 is used to estimate the speed and position of the object 106 at the next shooting. Therefore, in addition to the effects described in the first embodiment, there is an effect that it is possible to realize a tracking imaging apparatus that follows changes in the moving direction and speed of the object 106 and exhibits more suitable tracking performance.

In the present embodiment, the case where the information on the estimated velocity vector v _{n + 1} and the position vector D _{n + 1} is used only for setting the detection processing region 302 has been described as an example. However, in consideration of these pieces of information, the driving amounts (movement amounts) M _p and M _{t in} the pan / tilt direction may be calculated. In this way, depending on the performance of the pan / tilt driving mechanism, the tracking performance is further improved, and it is possible to make it difficult to lose sight of the object once captured. Furthermore, taking into consideration the performance of the pan / tilt drive mechanism, the estimated “amount of movement of the object 106 until the next shooting” exceeds the range that can be moved by the pan / tilt drive mechanism within a desired time. Only the detection processing area 302 may be moved to assist the pan / tilt driving mechanism.

  Further, in the present embodiment, an example in which information such as the current position of the object and the positions of the two previous captured images is used to estimate the speed of the object at the next imaging will be described as an example. did. However, it is not always necessary to do this. Further, the previous information is retained as a history, and the speed (change in speed) of the object at the next imaging is estimated using statistical processing. Good.

(Other embodiments of the present invention)
Each unit constituting the tracking imaging apparatus and each step of the tracking imaging method in the embodiment of the present invention described above can be realized by operating a program stored in a RAM or ROM of a computer. This program and a computer-readable recording medium recording the program are included in the present invention.

  In addition, the present invention can be implemented as, for example, a system, apparatus, method, program, storage medium, or the like. Specifically, the present invention may be applied to a system including a plurality of devices. The present invention may be applied to an apparatus composed of a single device.

  The present invention includes a software program (in the embodiment, a program corresponding to the flowchart shown in FIG. 2) that directly or remotely supplies a software program that implements the functions of the above-described embodiments. The present invention also includes a case where the system or apparatus computer achieves this by reading and executing the supplied program code.

  Accordingly, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention.

  In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, and the like.

  Examples of the recording medium for supplying the program include a floppy (registered trademark) disk, hard disk, optical disk, magneto-optical disk, MO, CD-ROM, CD-R, and CD-RW. In addition, there are magnetic tape, nonvolatile memory card, ROM, DVD (DVD-ROM, DVD-R), and the like.

  As another program supply method, a browser on a client computer is used to connect to an Internet home page. The computer program itself of the present invention or a compressed file including an automatic installation function can be downloaded from the homepage by downloading it to a recording medium such as a hard disk.

  It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the present invention.

  In addition, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to users, and key information for decryption is downloaded from a homepage via the Internet to users who have cleared predetermined conditions. Let It is also possible to execute the encrypted program by using the downloaded key information and install the program on a computer.

  Further, the functions of the above-described embodiments are realized by the computer executing the read program. In addition, based on the instructions of the program, an OS or the like running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments can also be realized by the processing.

  Further, the program read from the recording medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Thereafter, the CPU of the function expansion board or function expansion unit performs part or all of the actual processing based on the instructions of the program, and the functions of the above-described embodiments are realized by the processing.

  It should be noted that each of the above-described embodiments is merely a specific example for carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner. . That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

1 is a block diagram illustrating an example of a hardware configuration of a tracking imaging apparatus according to a first embodiment of this invention. It is a flowchart which shows the 1st Embodiment of this invention and demonstrates an example of the basic process of a tracking control program. It is a figure which shows the 1st Embodiment of this invention and shows an example of an initial detection process area | region. It is a figure which shows the 1st Embodiment of this invention and shows an example of the mode of the object of tracking object in an imaging region. It is a figure which shows the 1st Embodiment of this invention and shows an example of a mode that it moves only the same as the position vector of the object obtained by this object detection process, and sets the next detection process area | region. In the first embodiment of the present invention, when an object moves in a direction opposite to the position vector of the object obtained in the current object detection process, the next detection processing area is set using the position vector. It is a figure explaining the problem of. It is a figure which shows the 2nd Embodiment of this invention and demonstrates the 1st example of the method of estimating the speed of the object of tracking object. It is a figure which shows the 2nd Embodiment of this invention and demonstrates the 1st example of the method of estimating the speed of the object of tracking object. It is a figure which shows the 2nd Embodiment of this invention and demonstrates an example of the estimation method of the position vector of the object at the time of the next imaging.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Image pick-up apparatus 101 Camera apparatus 102 Optical axis 103 Pan axis 104 Tilt axis 105 Pan head 106 Object 110 Controller apparatus 112 CPU
113 Nonvolatile memory device 114 RAM
115 Image buffer 117 Pan / tilt drive control unit 118 Image acquisition unit 119 Object detection processing unit 120 Processing region setting unit

Claims (8)

Imaging means for imaging an image in the direction of the imaging optical axis;
Drive means for driving the imaging means to change the direction of the imaging optical axis in order to track the target object in the image imaged by the imaging means;
Area setting means for setting a detection processing area in a partial area of the image imaged by the imaging means;
Relative position from a reference point set in advance in the image area imaged by the imaging means as the position information of the target object in the detection processing area by processing the image of the detection processing area set by the area setting means an object position detecting means for detecting information indicating,
Area position calculating means for calculating the position of the detection processing area at the time of next shooting based on the relative position from the reference point which is position information of the target object detected by the object position detecting means;
Based on the position information of the target object detected by the object position detecting means, after the driving of the imaging means by the driving means is completed, the imaging means is adjusted so that the reference point matches or approaches the relative position. Drive amount calculating means for calculating a drive amount for driving;
Ratio determining means for determining a predetermined ratio as a value according to the driving amount calculated by the driving amount calculating means ;
The area setting means sets the detection processing area at the time of the next imaging based on the position calculated by the area position calculation means ,
The area position calculating means is configured to calculate the detection processing area at the time of the current photographing by a distance obtained by multiplying the length from the reference point to the relative position by the predetermined ratio from the reference point to the relative position. A tracking imaging apparatus characterized by calculating a moved position . Based on the position information of the target object detected by the object position detection means, and having a movement speed estimation means for estimating the movement speed of the target object,
Said area setting means is estimated by the movement speed estimation means, based on the moving speed information of the target object, according to claim 1, characterized in that to set the detection processing area at the time of next imaging Tracking imaging device. The moving speed estimation means calculates the amount of change in speed of the target object based on position information of the target object already detected by the object position detection means and time information indicating an imaging interval by the imaging means. The tracking imaging apparatus according to claim 2 , wherein a moving speed of the target object at the next imaging is calculated using a change amount of the speed of the target object. Said drive means includes a drive mechanism for rotating said imaging means as the rotation axis pan axis, any claim 1-3, characterized in that it comprises a drive mechanism for rotating said imaging means tilt axis as a rotation axis The tracking imaging device according to claim 1. The object location detecting means uses a neural network, tracking imaging apparatus according to any one of claim 1 to 4, characterized in that to detect the position information of the target object. The area setting unit is configured to perform a next imaging based on position information of the target object detected by the object position detection unit when the detection of the position of the target object is continued by the object position detection unit. tracking imaging apparatus according to any one of claim 1 to 5, characterized in that to set the detection processing region in. An imaging step of imaging an image in the direction of the imaging optical axis by an imaging means;
A driving step of driving the imaging means to change the direction of the imaging optical axis in order to track the target object in the image captured by the imaging step;
An area setting step for setting a detection processing area in a partial area of the image captured by the imaging step;
Relative position from a reference point set in advance in the image area imaged by the imaging step as the position information of the target object in the detection processing area is processed by processing the image of the detection processing area set by the area setting step and the object position detecting step of detecting information indicating,
An area position calculating step for calculating a position of the detection processing area at the time of the next shooting based on a relative position from the reference point that is position information of the target object detected by the object position detecting step;
Based on the position information of the target object detected by the object position detection step, after the driving of the imaging means by the driving step is completed, the imaging means is adjusted so that the reference point matches or approaches the relative position. A driving amount calculating step for calculating a driving amount for driving;
A ratio determining step of determining a predetermined ratio as a value according to the driving amount calculated by the driving amount calculating step ;
In the region setting step , based on the position calculated in the region position calculating step , the detection processing region at the time of the next imaging is set ,
In the area position calculation step, the detection processing area at the time of the current photographing is the distance obtained by multiplying the length from the reference point to the relative position by the predetermined ratio from the reference point to the relative position. A tracking imaging method characterized by calculating a moved position . Instructs the drive means to drive the image pickup means to change the direction of the image pickup optical axis in order to track the target object in the image picked up by the image pickup means for picking up an image in the direction of the image pickup optical axis. A driving step;
An area setting step for setting a detection processing area in a partial area of the image captured by the imaging means;
Relative position from a reference point set in advance in the image area captured by the imaging means as the position information of the target object in the detection processing area is processed by processing the image of the detection processing area set in the area setting step and the object position detecting step of detecting information indicating,
An area position calculating step for calculating a position of the detection processing area at the time of the next shooting based on a relative position from the reference point that is position information of the target object detected by the object position detecting step;
Based on the position information of the target object detected by the object position detection step, after the driving of the imaging means by the driving step is completed, the imaging means is adjusted so that the reference point matches or approaches the relative position. A driving amount calculating step for calculating a driving amount for driving;
Causing the computer to execute a ratio determining step of determining a predetermined ratio as a value corresponding to the driving amount calculated by the driving amount calculating step ;
In the region setting step , based on the position calculated in the region position calculating step , the detection processing region at the time of the next imaging is set ,
In the area position calculation step, the detection processing area at the time of the current photographing is the distance obtained by multiplying the length from the reference point to the relative position by the predetermined ratio from the reference point to the relative position. A computer program for calculating a moved position .

Images