JP5811923B2 - Information processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an information processing apparatus, an image processing method, and a program.

  At present, we may want to investigate the number of flying objects such as birds for various purposes. For example, as part of environmental protection activities, it is conceivable to investigate the number of wild birds to understand the current state of wildlife. For example, the number of flying objects is counted during the flight of the flying objects.

  As a method of counting the flying objects flying in the space, a person goes to a place where the flying object is present and counts by visual observation, or a flying object is imaged using an imaging device, and the human image is captured. A method of counting by looking at can be considered. Also, a method of estimating the number of flying objects captured in the captured image by applying an image processing technique such as pattern recognition to the captured image capturing the flying object is also conceivable. For example, there is a method in which pattern information indicating the shape, color, etc. of a flying object to be counted is prepared, a region that matches the pattern information is detected from a captured image, and the number of detected regions is estimated as the number of flying objects. Conceivable.

  The azimuth and elevation angles of the two video cameras are controlled so that the moving body is displayed in the center of the screen, and the equations of two straight lines connecting the moving body and the two video cameras are calculated. A position measurement system that calculates three-dimensional coordinates of a moving body has been proposed. Further, there has been proposed a bird damage prevention apparatus that estimates the number of wild bird arrivals from a captured image and controls to change the currently implemented bird damage countermeasure to a different bird damage countermeasure when the number of arrivals is larger than a threshold value.

JP 2001-349707 A JP 2008-72 A

  By the way, in the survey of the number of flying objects, there is a case where it is desired to investigate a group including a large number of flying objects such as a flock of wild birds flying all at once from a breeding place (colony). However, when such a large number of flying objects are imaged, a plurality of flying objects may be superimposed on the captured image in the depth direction. For this reason, in the conventional image processing method of extracting a region that matches the pattern information indicating the shape and color from the captured image, it is difficult to distinguish a plurality of flying objects that are overlapped. It was difficult to estimate the population accurately.

  In one aspect, an information processing apparatus is provided that processes captured images obtained by capturing a plurality of flying objects. The information processing apparatus includes a storage unit and a calculation unit. For each of a plurality of sample images with different numbers of overlapping objects, the storage unit is surrounded by an area whose brightness is less than the threshold, and the number of spot areas whose brightness is greater than or equal to the threshold and the sample image Sample information including the number of flying objects is stored. The calculation unit calculates the number of spot regions included in the captured image, selects sample information corresponding to the calculated number of spot regions from sample information corresponding to a plurality of sample images, and the selected sample information indicates Based on the number of flying objects, the number of flying objects in the captured image is estimated.

  In one aspect, an image processing method by a computer that processes captured images of a plurality of flying objects is provided. In the image processing method, the number of spot regions that are surrounded by an area with brightness less than the threshold and whose brightness is equal to or greater than the threshold is calculated from the captured image. Depending on the number of spot areas calculated from sample information corresponding to multiple sample images with different number of overlapping objects, including the number of spot areas in the sample image and the number of objects in the sample image. Select sample information. Based on the number of flying objects indicated by the selected sample information, the number of flying objects shown in the captured image is estimated.

  In one aspect, a program for processing a captured image in which a plurality of flying objects are captured is provided. The computer that executes the program calculates the number of spot areas that are surrounded by areas whose brightness is less than the threshold and whose brightness is greater than or equal to the threshold from the captured image. Depending on the number of spot areas calculated from sample information corresponding to multiple sample images with different number of overlapping objects, including the number of spot areas in the sample image and the number of objects in the sample image. Select sample information. Based on the number of flying objects indicated by the selected sample information, the number of flying objects shown in the captured image is estimated.

  In one aspect, the estimation accuracy of the number of flying objects shown in the captured image is improved.

It is a figure which shows the information processing apparatus of 1st Embodiment. It is a figure which shows the information processing system of 2nd Embodiment. It is a block diagram which shows the hardware example of a monitoring apparatus. It is a figure which shows the example of a relationship between an imaging direction and the shape of a bird. It is a figure which shows the example of a relationship between the number of layers, the brightness, and the number of light spots. It is a figure (continuation) which shows the example of a relationship between the number of layers, the brightness, and the number of light spots. It is a figure which shows the example of a change of the number of rows in a flock of birds. It is a block diagram which shows the function example of a simulation apparatus and a monitoring apparatus. It is a figure which shows the example of arrangement | positioning of the bird and camera in a three-dimensional model. It is a figure which shows the example of an image condition table. It is a figure which shows the example of a camera coordinate table. It is a figure which shows the example of the area | region where the flock of birds in the captured image was reflected. It is a figure which shows the example of the calculation method of the distance between a camera and a bird. It is a flowchart which shows the example of a procedure of reference data generation. It is a flowchart (continuation) which shows the example of a procedure of reference data generation. It is a figure which shows the example of a reference data table. It is a flowchart which shows the example of a procedure of bird number estimation. It is a flowchart (continuation) which shows the example of a procedure of bird number estimation. It is a flowchart which shows the example of a procedure of flight direction determination. It is a figure which shows the example of the estimation result of the number of birds.

Hereinafter, the present embodiment will be described with reference to the drawings.
[First Embodiment]
FIG. 1 is a diagram illustrating the information processing apparatus according to the first embodiment. The information processing apparatus 10 processes a captured image 13 obtained by copying a plurality of flying objects. The captured image 13 is generated by, for example, one or more imaging devices connected to the information processing device 10 and input to the information processing device 10. The flying object to be imaged may be a flying creature such as a wild bird or an artificial object. For example, the imaging device that generates the captured image 13 and the information processing device 10 are installed near a breeding place for wild birds, and are used to investigate the number of wild birds flying from the breeding place.

  The information processing apparatus 10 includes a storage unit 11 and a calculation unit 12. The storage unit 11 may be a volatile memory such as a RAM (Random Access Memory) or a non-volatile storage device such as an HDD (Hard Disk Drive) or a flash memory. The arithmetic unit 12 may be a processor such as a central processing unit (CPU) or a digital signal processor (DSP), or may be another electronic circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). The processor executes, for example, a program stored in the memory. The processor may include a dedicated electronic circuit for image processing in addition to an arithmetic unit and a register for executing program instructions.

  The memory | storage part 11 memorize | stores sample information about each of several sample images from which the overlapping number (for example, number of layers) of the flying object of a depth direction differs. For example, the storage unit 11 stores sample information 11a corresponding to the sample image # 1, sample information 11b corresponding to the sample image # 2, and sample information 11c corresponding to the sample image # 3. For example, sample image # 1 has an overlap number of 2, sample image # 2 has an overlap number of 3, and sample image # 3 has an overlap number of 4. Each sample information includes the number of spot regions included in the sample image and the number of flying objects shown in the sample image. The spot area is surrounded by an area where the brightness is less than the threshold (for example, an area where pixels having the brightness less than the threshold are continuous), and the area where the brightness is the threshold or more (for example, the brightness is the threshold or more) Area where pixels are continuous). The area detected as the spot area may be limited to a size larger than a certain number of pixels.

  The sample images # 1 to # 3 may be images generated by simulation or images obtained by capturing actual flying objects. The memory | storage part 11 should just memorize | store sample information 11a, 11b, 11c, and does not need to memorize | store sample image # 1-# 3. For example, an apparatus other than the information processing apparatus 10 generates sample information 11a, 11b, and 11c from sample images # 1 to # 3 obtained by simulation, and the sample information 11a, 11b, and 11c is transmitted to the information processing apparatus 10. You may make it store.

  The calculation unit 12 acquires the captured image 13 and calculates the number of spot regions included in the captured image 13 by image processing. And the calculating part 12 selects the sample information (for example, sample information 11b) according to the number of the spot area | regions calculated from the captured image 13 among the sample information 11a, 11b, 11c memorize | stored in the memory | storage part 11. FIG. The computing unit 12 estimates the number of flying objects shown in the captured image 13 based on the number of flying objects indicated by the selected sample information.

  For example, the calculation unit 12 corrects the number of spot regions indicated by each sample information based on at least one of the distance between the imaging device that generated the captured image 13 and the flying object and the size of the captured image 13. To do. And the calculating part 12 selects sample information by comparing the correction result for every sample information, and the number of the spot area | regions calculated from the captured image 13 (for example, the sample information from which both are closest is selected). . The distance between the imaging device and the flying object can also be calculated by the triangulation method from the difference in the position where the flying object is reflected in the two captured images generated using the two imaging devices, It can also be estimated from the size of the flying object on the image. For example, the calculation unit 12 corrects the number of flying objects indicated by the selected sample information based on at least one of the distance between the imaging device and the flying object and the size of the captured image, and the correction result is reflected in the captured image 13. Estimate the number of flying objects.

  Note that the sample information 11a, 11b, and 11c may further include a brightness value indicating the proportion of the area in the sample image where the brightness is equal to or greater than the threshold value. In this case, the calculation unit 12 can calculate the brightness value of the captured image 13 and select sample information according to the number of spot areas of the captured image 13 and the brightness value of the captured image 13. For example, the calculation unit 12 has a brightness value close to the brightness value of the captured image 13 (for example, the difference between the two is less than or equal to a predetermined value), and the number of corrected spot areas is the number of spot areas of the captured image 13. It is conceivable to select sample information that is close to the number (for example, the difference between the two is less than or equal to a predetermined value).

  In addition, the storage unit 11 may store sample information for each of a plurality of sample images having different combinations of the number of flying objects in the depth direction and the flying direction of the flying object. In this case, the calculation unit 12 can determine the flight direction of the flying object shown in the captured image 13 and can narrow down candidate sample information to be selected based on the determined flight direction. The flight direction can be determined, for example, by detecting a change in the position and size of the flying object on the image from two or more captured images generated by the same imaging device at different timings. Thereby, it is possible to cope with a flying object in which the appearance of the spot region differs depending on the flight direction.

  According to the information processing apparatus 10 of the first embodiment, information indicating the relationship between the number of spot regions that appear in an image due to the overlapping of a plurality of flying objects in the depth direction and the number of flying objects that appear in the image is referred to. . Thereby, even if a plurality of flying objects are reflected in the depth direction of the captured image 13 (for example, even if a group of flying objects forms a plurality of layers), the spot area of the captured image 13 From the number, the number of flying objects can be estimated with high accuracy. In the second embodiment described below, an example in which the number of birds appearing in a captured image is estimated is given.

[Second Embodiment]
FIG. 2 illustrates an information processing system according to the second embodiment. The information processing system according to the second embodiment estimates the number of birds appearing in a captured image from a captured image obtained by capturing a flock of birds flying in the air (for example, a flock of birds flying from a breeding ground). This information processing system is used, for example, to investigate the number of birds as part of environmental protection activities. This information processing system includes cameras 21 and 22, a simulation device 100, and a monitoring device 200. The cameras 21 and 22 and the monitoring apparatus 200 are installed in the vicinity of the bird's habitat, and the simulation apparatus 100 is installed in a place far from the bird's habitat.

  The cameras 21 and 22 are imaging devices that generate still images (or may be moving images), and are connected to the monitoring device 200. The cameras 21 and 22 capture a flock of birds flying in the air and output the captured images to the monitoring device 200. The cameras 21 and 22 are installed such that they are separated from each other by a certain distance, the optical axes are parallel to each other, and the lens surfaces face the same direction. By installing the two cameras in this way, the distance between the flock of birds and the cameras 21 and 22 can be calculated from the two captured images (details of distance calculation will be described later). The cameras 21 and 22 are installed so that the elevation angle (angle between the horizontal plane and the optical axis) becomes constant toward the air.

  In order to automatically capture the flock of birds outdoors, the cameras 21 and 22 generate a captured image and detect it in the monitoring device 200 only when a shadow that may indicate a flock of birds is detected within the viewing angle. You may make it provide. When generating still images, the cameras 21 and 22 detect a plurality of consecutive images at predetermined time intervals (for example, about several hundred milliseconds to several seconds) when detecting a shadow that may indicate at least a flock of birds. A captured image is generated and provided to the monitoring apparatus 200. By using a plurality of captured images having different imaging timings, it is possible to determine the flight direction of the bird with respect to the cameras 21 and 22. The cameras 21 and 22 may include a stage for changing the elevation angle so that the elevation angle can be instructed from a remote location. Further, three or more cameras may be connected to the monitoring apparatus 200. In addition, when the distance to the bird can be estimated by a method other than triangulation as described later, one camera connected to the monitoring device 200 can be used.

  The simulation device 100 is a computer as a terminal device operated by a user. However, the simulation apparatus 100 can be a server computer accessed from a terminal device. The simulation apparatus 100 simulates the projection of a flock of birds flying in the air using CAD (Computer Aided Design) software that can generate a three-dimensional model. The simulation is performed for a plurality of patterns by changing conditions such as the arrangement of a plurality of birds (for example, how many layers the birds are flying in) and the direction of looking at the flock of birds. And the simulation apparatus 100 produces | generates the reference data which show the relationship between the direction which looks at a flock of birds, the characteristic which the projection of a flock of birds has, and the number of birds from a simulation result. The reference data is provided to the monitoring apparatus 200 and used for estimating the number of birds.

  The monitoring device 200 is a computer that estimates the number of flying birds using captured images acquired from the cameras 21 and 22. The monitoring device 200 may be a terminal device provided with a user interface, or may be accessed from the terminal device via the network 30. The monitoring apparatus 200 confirms the imaging conditions such as the elevation angle of the cameras 21 and 22 and the flight direction of the flock of birds with respect to the cameras 21 and 22, and features of the projection of the flock of birds from the captured images (for example, areas with shadows) And the sky area without shadows). Then, the monitoring apparatus 200 compares the reference data generated by the simulation apparatus 100 with the imaging conditions and the features extracted from the captured image, and estimates the number of birds that appear in the captured image.

  Note that the reference data generated by the simulation apparatus 100 may be recorded in a nonvolatile storage device included in the monitoring apparatus 200 before the monitoring apparatus 200 is arranged near the bird's habitat, or via the network 30. You may provide to the monitoring apparatus 200. FIG. Further, the reference data may be recorded on a portable recording medium, and the monitoring device 200 after the arrangement may be read from the portable recording medium. In addition, the estimation result of the number of birds of the monitoring device 200 may be automatically reported to a predetermined computer via the network 30, or the user directly confirms the input / output device of the monitoring device 200. You may make it do. Therefore, the simulation apparatus 100 and the monitoring apparatus 200 may or may not be connected to the network 30 depending on the reference data providing method and the bird count reporting method. In the following, it is assumed that the simulation apparatus 100 and the monitoring apparatus 200 are connected to the network 30.

  FIG. 3 is a block diagram illustrating an example of hardware of the monitoring device. The monitoring apparatus 200 includes a CPU 201, RAM 202, HDD 203, image signal processing unit 204, input signal processing unit 205, disk drive 206, communication unit 207, and interface 208. Each of the above units is connected to a bus. The CPU 201 is an example of the calculation unit 12 according to the first embodiment, and the RAM 202 is an example of the storage unit 11 according to the first embodiment.

  The CPU 201 is a processor including an arithmetic unit that executes program instructions. The CPU 201 loads at least a part of the program and data stored in the HDD 203 into the RAM 202 and executes the program. Note that the CPU 201 may include a plurality of processor cores, the monitoring apparatus 200 may include a plurality of processors, and the processes described below may be executed in parallel using a plurality of processors or processor cores.

  The RAM 202 is a volatile memory that temporarily stores programs executed by the CPU 201 and data used for calculation. Note that the monitoring apparatus 200 may include a type of memory other than the RAM, or may include a plurality of volatile memories.

  The HDD 203 is a nonvolatile storage device that stores an OS (Operating System), software programs such as firmware and application software, and data. The monitoring device 200 may include other types of storage devices such as a flash memory and an SSD (Solid State Drive), and may include a plurality of nonvolatile storage devices.

  The image signal processing unit 204 outputs an image to the display 23 connected to the monitoring device 200 in accordance with a command from the CPU 201. As the display 23, a CRT (Cathode Ray Tube) display, a liquid crystal display, or the like can be used.

  The input signal processing unit 205 acquires an input signal from the input device 24 connected to the monitoring apparatus 200 and notifies the CPU 201 of the input signal. As the input device 24, a pointing device such as a mouse or a touch panel, a keyboard, or the like can be used.

  The disk drive 206 is a drive device that reads programs and data recorded on the recording medium 25. Examples of the recording medium 25 include a magnetic disk such as a flexible disk (FD) and an HDD, an optical disk such as a CD (Compact Disc) and a DVD (Digital Versatile Disc), and a magneto-optical disk (MO). Can be used. The disk drive 206 stores the program and data read from the recording medium 25 in the RAM 202 or the HDD 203 in accordance with a command from the CPU 201.

  The communication unit 207 is an interface that can communicate with another computer (for example, the simulation apparatus 100) using the network 30. The communication unit 207 may be a wired interface connected to a wired network or a wireless interface connected to a wireless network.

  The interface 208 includes terminals connected to the cameras 21 and 22. A digital signal cable such as a USB (Universal Serial Bus) cable can be used for connection between the interface 208 and the cameras 21 and 22. The interface 208 acquires captured images from the cameras 21 and 22 and stores them in the RAM 202 or the HDD 203.

  The simulation apparatus 100 can also be realized using the same hardware as the monitoring apparatus 200. However, the simulation apparatus 100 does not need to include the disk drive 206 and the interface 208, and in the case of a server computer, the simulation apparatus 100 does not need to include the image signal processing unit 204 and the input signal processing unit 205. Further, the monitoring device 200 may not include the disk drive 206, and may not include the image signal processing unit 204 and the input signal processing unit 205 when accessible from other computers.

  FIG. 4 is a diagram illustrating an example of the relationship between the imaging direction and the bird shape. Even when the cameras 21 and 22 capture the same flock of birds, the projection of the flock of birds shown in the captured image differs depending on the direction from which the flock is taken. In the example of FIG. 4, a flock of six birds flying in one layer is imaged from directly below (at an elevation angle of 90 °) and horizontally from the three directions of front, side, and diagonal (at an elevation angle of 0 °). A projection of a flock of birds shown in a captured image when captured is shown.

  Taking a flock of birds from directly below, these six birds are shown apart from each other, as shown in the lower right of FIG. When taking a flock of birds horizontally from the front with respect to the direction of flight, as shown in the upper right of FIG. 4, a bird and other birds that fly behind it appear at least partially overlapping. If a flock of birds is taken horizontally from the direction of flight, as shown in the lower left of FIG. 4, a bird and other birds flying side by side appear to overlap at least partially. If a flock of birds is taken horizontally from an angle of 45 ° with respect to the flight direction (left front of the bird in FIG. 4), as shown in the upper left of FIG. 4, one bird and other birds flying diagonally behind it , At least partially overlap.

  As described above, how birds overlap on the captured image differs depending on whether the flock of birds is imaged from the front, side, or diagonal direction with respect to the flight direction. Also, how birds overlap is also different depending on the elevation angle, such as whether the flock of birds was taken from directly below, taken from the horizontal direction, or at an intermediate angle. In addition, the way in which birds overlap is different depending on the number of layers formed when a plurality of birds can fly by forming a plurality of layers in a direction perpendicular to the horizontal plane. Further, the way in which birds overlap is also different depending on the number of rows formed when a plurality of birds can fly in a plurality of rows from the top to the back. Therefore, when generating the reference data, the simulation apparatus 100 projects the flock of birds while changing the number of layers / rows of the flock of birds and the imaging direction (front / horizontal / diagonal) / elevation angle with respect to the flight direction of the birds. To simulate.

  FIG. 5 is a diagram illustrating a relationship example between the number of layers and the number of brightness / light spots. When flocks of birds are flying in a plurality of layers in a direction perpendicular to the horizontal plane, when the cameras 21 and 22 look up at the sky at an elevation angle of 90 °, the birds overlap in the depth direction of the captured image. It is reflected.

  At this time, a bird in one layer and a bird in another layer rarely overlap each other on the captured image, and often appear slightly shifted (partially overlap). For this reason, as the number of layers in the flock increases, the brightness of the captured image decreases and the number of light spots increases.

  The brightness of the captured image consists of pixels whose brightness is above the threshold because natural light is not blocked (bright pixels) and pixels whose brightness is below the threshold because dark light is blocked by birds (dark pixels). Indicates the ratio. The value indicating the brightness of the captured image is calculated as, for example, the number of bright pixels / the number of dark pixels, or the number of bright pixels / (the number of bright pixels + the number of dark pixels). As the number of layers increases, the area in which the bird is photographed increases and the area in which the sky is photographed decreases, so the brightness of the captured image decreases. The light spot is a region composed of bright pixels surrounded by dark pixels, and has a size equal to or larger than a threshold value. As the number of layers increases, a region surrounded by the projection of a bird in one layer and the projection of a bird in another layer appears more in the captured image, and the number of light spots increases.

  As an example, FIG. 5A shows an image obtained by capturing a flock of eight birds flying in one layer from an elevation angle of 90 °. The captured image of (A) does not include a light spot. FIG. 5B shows a group of eight birds flying in the first layer and seven birds flying in the second layer, that is, a group of fifteen birds flying in two layers. The image imaged from is shown. The captured image of (B) does not include a light spot, but the brightness is reduced as compared with the captured image of (A).

  FIG. 6 is a diagram (continued) showing an example of the relationship between the number of layers and the number of brightness / light spots. FIG. 6C shows a group of eight birds flying in the first layer, seven birds flying in the second layer, and eight birds flying in the third layer, that is, 23 birds. The image which image | photographed the flock which flies by 3 layers from the elevation angle of 90 degrees is shown. The captured image (C) includes seven light spots, and the projection density of the bird is higher and the brightness is lower than the captured image (B) described above. In FIG. 6D, 8 birds fly to the 1st layer, 7 birds fly to the 2nd layer, 8 birds fly to the 3rd layer, and 7 birds fly to the 4th layer. The image which imaged the flock which flies, ie, the flock which 30 birds fly by 4 layers, from 90 degrees of elevation angles is shown. The captured image of (D) includes 13 light spots, and the projection density of the bird is further increased and the brightness is reduced as compared with the captured image of (C) described above.

  Thus, the number of layers in the flock of birds affects the brightness of the captured image and the number of light spots appearing in the captured image. Therefore, the monitoring apparatus 200 extracts brightness and the number of light spots from the captured image as feature information, and estimates the number of layers and the number of birds in the captured image.

  FIG. 7 is a diagram illustrating an example of a change in the number of rows in a flock of birds. A flock of birds may form a row by arranging a plurality of birds on a line perpendicular to the flight direction in a horizontal plane. When a group of birds flying in a plurality of rows is imaged from the front at an elevation angle of 0 °, a plurality of birds are shown overlapping in the depth direction of the captured image. In the second embodiment, the number of rows of bird flocks is counted excluding the first bird.

  As an example, FIG. 7A shows an image obtained by capturing a group of six birds flying in one layer and two rows from an elevation angle of 0 ° and an elevation angle of 90 °. On the image captured at an elevation angle of 0 °, the birds in the front row and the birds in the back row are shown overlapping. FIG. 7B shows an image obtained by capturing a group of 10 birds flying in one layer and three rows from an elevation angle of 0 ° and an elevation angle of 90 °. The image captured at an elevation angle of 0 ° in (B) is less bright than the captured image in (A). FIG. 7C shows an image of a flock of 15 birds flying in one layer and four rows from an elevation angle of 0 ° and an elevation angle of 90 °. The image captured at an elevation angle of 0 ° in (C) is further reduced in brightness than the captured image in (B). In the example of FIG. 7, a light spot does not appear in an image captured from an elevation angle of 0 °, but a light spot may appear depending on the arrangement of birds.

  Thus, when imaging a flock of birds at an elevation angle of 0 °, the number of columns affects the brightness of the captured image and the number of light spots appearing in the captured image. Therefore, the simulation apparatus 100 simulates the projection of the flock of birds while changing the number of rows of the flock of birds. In the simulation method described below, the number of columns is made variable only when the elevation angle = 0 ° in order to suppress the amount of reference data. However, the number of columns may be made variable at other elevation angles.

FIG. 8 is a block diagram illustrating functional examples of the simulation apparatus and the monitoring apparatus.
The simulation apparatus 100 includes a model condition storage unit 110, a 3D model generation unit 120, a reference image processing unit 130, and a reference data storage unit 140. The model condition storage unit 110 and the reference data storage unit 140 can be realized as a storage area secured in the RAM or HDD included in the simulation apparatus 100. The three-dimensional model generation unit 120 and the reference image processing unit 130 can be realized as a program module executed by a processor.

  The model condition storage unit 110 stores model condition information indicating conditions for generating a three-dimensional model representing a flock of birds and simulating projection of the flock of birds. The model condition information includes information indicating the number of layers and the number of rows of bird flock models to be prepared, and information indicating from which direction the flock is observed on the three-dimensional model. included.

  The three-dimensional model generation unit 120 has a CAD function for processing a three-dimensional model. The three-dimensional model generation unit 120 arranges a plurality of bird shapes in a virtual three-dimensional space based on the model condition information stored in the model condition storage unit 110, and generates a three-dimensional model representing a flock of birds. Generate. Then, the three-dimensional model generation unit 120 arranges a virtual camera on the three-dimensional model based on the model condition information, and generates a reference image representing the shape of the flock of birds viewed from the virtual camera. . The three-dimensional model generation unit 120 generates a plurality of reference images while changing the number of layers and the number of rows of birds and the direction in which the birds are observed according to the model condition information.

  The reference image processing unit 130 has an image processing function for processing a digital two-dimensional image. The reference image processing unit 130 analyzes each of the plurality of reference images generated by the three-dimensional model generation unit 120, and calculates the brightness of the reference image and the number of light spots appearing in the reference image as feature information. In addition, the reference image processing unit 130 confirms with the 3D model generation unit 120 the number of bird shapes arranged on the 3D model when each reference image is generated. Then, the reference image processing unit 130 generates feature data such as brightness and the number of light spots and reference data indicating the number of birds for each reference image, and stores the reference data in the reference data storage unit 140.

  The reference data storage unit 140 stores the reference data generated by the reference image processing unit 130. The reference data stored in the reference data storage unit 140 is copied to the monitoring device 200 before or after the monitoring device 200 is installed near the bird habitat. As a method of replicating the reference data, a method of transferring from the simulation apparatus 100 to the monitoring apparatus 200 using the network 30 and a method of using a portable recording medium such as a CD or a DVD are conceivable. The reference image is an example of the sample image described in the first embodiment, and the reference data is an example of the sample information 11a, 11b, and 11c described in the first embodiment.

  The monitoring device 200 includes a reference data storage unit 210, a captured image acquisition unit 220, a captured image processing unit 230, and a bird count calculation unit 240. The reference data storage unit 210 can be realized as a storage area secured in a storage device such as the RAM 202 or the HDD 203 provided in the monitoring device 200. The captured image acquisition unit 220, the captured image processing unit 230, and the bird count calculation unit 240 can be realized as a program module executed by the CPU 201.

  The reference data storage unit 210 stores reference data generated by the simulation apparatus 100. Further, the reference data storage unit 210 stores at least a part of model condition information included in the simulation apparatus 100 in order to associate the reference data with the imaging condition of the reference image from which the reference data is generated. However, the imaging condition information may be embedded in the reference data copied from the simulation apparatus 100.

  The captured image acquisition unit 220 acquires digital signals of the captured images from the cameras 21 and 22 via the interface 208 and provides the captured images to the captured image processing unit 230. At this time, the captured image acquisition unit 220 captures the image captured by the camera 21 at time t, the image captured by the camera 21 after a predetermined time (or before the predetermined time) from the time t, and the camera 22 receives the time t (or , An image captured at a time close to time t). When the captured images acquired from the cameras 21 and 22 are moving images, the captured image acquisition unit 220 cuts out the still image from the moving images.

  The captured image acquisition unit 220 also provides the captured image processing unit 230 with information indicating the installation state of the cameras 21 and 22. The information on the installation state includes the distance between the cameras 21 and 22 (the distance between the optical axes) and the elevation angle of the cameras 21 and 22. Information on the distance between the optical axes and the elevation angle is set in the monitoring device 200 when the cameras 21 and 22 are installed near the bird's habitat. However, when the cameras 21 and 22 include a stage that automatically adjusts the elevation angle, the captured image acquisition unit 220 may be able to acquire information on the current elevation angle from the cameras 21 and 22.

  The captured image processing unit 230 has an image processing function for processing a digital two-dimensional image. The captured image processing unit 230 analyzes the captured image acquired from the captured image acquisition unit 220 and calculates the brightness of the captured image and the number of light spots appearing in the captured image as feature information. In addition, the captured image processing unit 230 determines the flight direction of the flock of birds with respect to the cameras 21 and 22 from the two captured images having different imaging timings, and the cameras 21 and 22 from the two captured images generated by the different cameras. The distance between the bird and the flock of birds is calculated by the triangulation method. The captured image processing unit 230 generates imaging data including feature information such as brightness and the number of light spots and imaging condition information such as a flight direction and an elevation angle, and provides the imaging data to the bird count calculation unit 240.

  The bird count calculation unit 240 estimates the number of birds captured in the captured image based on the reference data stored in the reference data storage unit 210 and the captured image data acquired from the captured image processing unit 230. Specifically, the bird count calculation unit 240 narrows down the plurality of reference images prepared by the simulation to those having the same (same or close) imaging conditions such as a flight direction and an elevation angle. Then, the bird count calculation unit 240 selects, from the narrowed-down reference images, images having the same characteristics (such as brightness and the number of light spots) that are common (same or close to each other), and performs imaging using the selected number of birds in the reference image. Calculate the approximate number of birds in the image. In comparison of brightness and the number of light spots and estimation of the number of birds in the captured image, the numerical values are corrected as appropriate according to the distance between the cameras 21 and 22 and the flock of birds.

  FIG. 9 is a diagram illustrating an arrangement example of birds and cameras in the three-dimensional model. In the three-dimensional model generated by the three-dimensional model generation unit 120, a bird shape and a virtual camera are arranged in a virtual three-dimensional space defined by mutually orthogonal x-axis, y-axis, and z-axis.

  In order to model a flock by arranging a plurality of bird shapes, the following characteristics of the flock are considered. (A) When each bird is too close to the fellow bird, each bird tries to leave the bird (separation). (B) Each bird tries to fly in the same direction as the fellow birds around it (Alignment). (C) Each bird tries to approach the center of the flock within a range that is not too close to the fellow bird (Cohesion). Birds often fly almost horizontally due to their body structure, and the inclination in the vertical direction often falls within the range of -20 ° to + 20 °. The nature of the flock of birds is described, for example, in the following paper (Craig Reynolds, "Flocks, Herds, and Schools: A Distributed Behavioral Model", Proc. Of ACM SIGGRAPH '87 Conference, pp.25- 34, 1987.).

  A three-dimensional model is generated by arranging a plurality of bird shapes so as to satisfy the above-mentioned property of a flock of birds. At this time, the center of the flock is located at the origin (0, 0, 0), and each bird is flying in the horizontal direction facing the positive direction of the x axis. Although only one bird shape is shown in FIG. 9 for simplification, a plurality of bird shapes are actually arranged in a virtual three-dimensional space.

  The modeled flock of birds is observed from multiple directions by a virtual camera. The virtual cameras are arranged so as to face the origin (0, 0, 0), respectively. FIG. 9 shows a virtual camera that observes a flock of birds at an elevation angle of 0 ° from the position of coordinates (20, 0, 0) on the x axis, and the position of coordinates (0, 20, 0) on the y axis. Describes a virtual camera that observes a flock of birds at an elevation angle of 0 ° and a virtual camera that observes a flock of birds at an elevation angle of 90 ° from the position of coordinates (0, 0, −20) on the z-axis. Has been. The distance between each virtual camera and the origin is fixed at 20. The distance = 1 on the three-dimensional model corresponds to the actual distance 1 m, for example.

  Here, in determining the placement of virtual cameras in the simulation, it is considered from which direction the flock of birds can be observed by the cameras 21 and 22. First, it is considered that the cameras 21 and 22 are unlikely to be installed at a position higher than the flock of birds. Therefore, in the simulation, the elevation angle of the virtual camera is set to 0 ° or more and 90 ° or less if the direction in which the sky is looked up is a positive angle. Further, as the direction of flight of the flock of birds with respect to the cameras 21 and 22, the following eight ways can be considered when the elevation angle is other than 90 °.

(1) Direction to the camera (bird's head is in front)
(2) Direction away from the camera (bird's tail is in front)
(3) Crossing the camera's field of view from left to right (the bird's right wing appears in the foreground)
(4) Direction to cross the camera view from right to left (bird's left wing appears in front)
(5) Direction approaching the camera diagonally to the left with respect to the optical axis (the right front of the bird appears in the foreground)
(6) The direction of moving away from the camera diagonally to the left with respect to the optical axis (the left rear of the bird appears in front)
(7) Direction approaching the camera diagonally to the right with respect to the optical axis (the left front of the bird appears in the foreground)
(8) Direction away from the camera obliquely to the right with respect to the optical axis (the right rear of the bird appears in front)
However, even if the projection of a flock of birds is turned over, the brightness and the number of light spots are not affected. 6) (7) and (8) can be classified. Therefore, in the simulation, for each elevation angle, the shape of the flock may be observed from three locations corresponding to “front”, “lateral”, and “oblique”.

  The flight direction when the elevation angle is 90 ° does not affect the brightness and the number of light spots even if the projection is rotated. In the following description of the second embodiment, the flight direction when imaging a flock of birds at an elevation angle of 90 ° is treated as “lateral”.

  In a three-dimensional model, a reference image can be generated by observing a bird flock model from a virtual camera. At this time, a virtual light source is arranged on the opposite side of the virtual camera across the origin. As a result, in the reference image, a region where no bird is present is a bright region whose luminance is greater than or equal to a threshold, and a region where a bird is present is a dark region whose luminance is less than the threshold. In generating reference data from a reference image, it is only necessary to distinguish between a bright area and a dark area. Therefore, the three-dimensional model generation unit 120 may generate a color image as a reference image, or may generate a grayscale image or a black and white image binarized according to luminance.

  FIG. 10 is a diagram illustrating an example of the image condition table. The image condition table 111 is stored in the model condition storage unit 110 of the simulation apparatus 100. The image condition table 111 includes items of the number of layers, the number of columns, the elevation angle, the direction, and an image ID (Identification).

  In the number of layers item, the number of layers formed in the bird flock model (the number of overlaps in the z-axis direction) is set. In the example of FIG. 10, five types of layers from 1 layer to 5 layers are listed. In the column number item, the number of columns formed in the bird flock model (the number of overlaps in the x-axis direction) is set. In the example of FIG. 10, five types of columns from 1 column to 5 columns are listed. In the item of elevation angle, the elevation angle of a virtual camera is set. In the example of FIG. 10, there are four elevation angles of 0 °, 30 °, 60 °, and 90 °. Of course, other elevation angles such as 45 ° can be set. The direction item lists one or more directions that require observation among “front”, “horizontal”, and “oblique”. In the image ID item, for the combination of the number of layers, the number of columns, the elevation angle, and the direction, identification information for identifying a reference image generated under the conditions is set.

  A bird flock model is generated for each combination of number of layers and number of columns. Further, the position of the virtual camera is determined by the combination of the elevation angle and the direction (the z coordinate is determined from the elevation angle, and the x coordinate and the y coordinate are determined from the direction). One reference image is generated for a combination of one bird flock model and one virtual camera. For example, the reference image # 1 is generated under the condition that the number of layers = 1, the number of columns = 1, the elevation angle = 30 °, and the direction = front, the number of layers = 1, the number of columns = 2, the elevation angle = 0 °, and the direction = horizontal. Reference image # 9 is generated under the conditions.

  Here, in the second embodiment, as shown in FIG. 10, when observing a model with the number of columns = 1, a virtual camera set to an elevation angle = 30 °, 60 °, 90 ° is used. When observing a model with the number of columns = 2, 3, 4, and 5, a virtual camera set to an elevation angle = 0 ° is used. When the elevation angle is other than 90 °, the reference image is generated in three ways: direction = front, side, and diagonal, and when the elevation angle is 90 °, the reference image is generated in one direction = side. Therefore, 19 reference images are generated per layer number, and 95 reference images are generated in total.

  However, the condition of the reference image shown in FIG. 10 is an example, and it is possible to generate a reference image with other conditions. For example, the number of columns of the model observed with a virtual camera with an elevation angle of 30 °, 60 °, and 90 ° may be a predetermined number and may not be one. In addition, a model with the number of columns = 1 may be observed with a virtual camera with an elevation angle = 0 °, and a model with the number of columns = 2,3,4,5 may have an elevation angle = 30 °, 60 °, 90 °. You may observe with a virtual camera.

  FIG. 11 is a diagram illustrating an example of a camera coordinate table. The camera coordinate table 112 is stored in the model condition storage unit 110 of the simulation apparatus 100. The camera coordinate table 112 includes items of elevation angle, direction, and coordinates.

  Similar to the image condition table 111, a virtual camera elevation angle is set in the elevation angle item. In the example of FIG. 11, four of 0 °, 30 °, 60 °, and 90 ° are listed. In the direction item, any one of “front”, “horizontal”, and “oblique” is set. However, as described above, the direction when the elevation angle is 90 ° is only “horizontal”. In the coordinate item, a coordinate indicating where the virtual camera should be arranged with respect to the combination of the elevation angle and the direction is set. In the example of FIG. 11, the coordinates are set so that the distance from the origin is 20.

  For example, coordinates (14.14, 14.14, 0) are set as positions satisfying the imaging condition of elevation angle = 0 ° and direction = slanting. The virtual camera arranged at this position picks up a flock of birds horizontally from 45 ° diagonally to the left. Also, coordinates (17.32, 0, −10) are set as positions satisfying the imaging condition of elevation angle = 30 ° and direction = front. The virtual camera arranged at this position takes an image by looking up at the front of the flock of birds from below 30 °. Also, coordinates (0, 10, -17.32) are set as positions satisfying the imaging condition of elevation angle = 60 ° and direction = horizontal. The virtual camera arranged at this position takes an image by looking up the left side of the flock of birds from below 60 °.

  FIG. 12 is a diagram illustrating an example of a region where a flock of birds is captured in a captured image. The captured images generated by the cameras 21 and 22 may include a plurality of areas where birds are densely captured. There are two such regions in the captured image 40 shown in FIG. The captured image processing unit 230 of the monitoring apparatus 200 cuts out a region where birds are densely captured as a target region, and estimates the number of birds for each target region. For example, the target areas 41 and 42 are cut out from the captured image 40, and the number of birds is estimated for each of the target area 41 and the target area 42.

  Here, the target area is an area where dark pixels whose luminance is less than the threshold value are densely packed together. The captured image processing unit 230 includes, for example, a minimum rectangular area within a range in which dark pixels of a certain number of pixels or more are included in the area and can be extracted so that dark pixels do not cover the rectangular frame. , Detected as a target area. However, the rectangular area may be extracted with a margin so that the dark pixels in the area and the rectangular frame are at least a certain distance apart. In the captured image 40, there are two regions where dark pixels are densely packed, and a region of bright pixels whose luminance is equal to or higher than a threshold spreads between them, so that they are detected as different target regions.

  In FIG. 12 and the above description, the shape of the target region is a rectangle, but may be another shape such as an ellipse. In the above description, the captured image processing unit 230 extracts the target region from the captured image when estimating the number of birds. However, when generating the reference data, the reference image processing unit 130 also calculates the target region from the reference image. The process which extracts is performed. However, unlike the captured image, one target region is extracted from the reference image in principle.

  FIG. 13 is a diagram illustrating an example of a method for calculating the distance between the camera and the bird. By using the captured images generated by the two cameras 21 and 22, the captured image processing unit 230 can calculate the distance between the cameras 21 and 22 and the flock of birds by a triangulation method.

  First, the captured image processing unit 230 detects a dark region in which a bird is captured from a captured image generated by the camera 21, and includes about several hundred pixels that include both the detected dark region and a bright region in which the sky is captured. Is detected as a reference pattern. The reference pattern may be a target region that is a unit for calculating the number of birds or a part thereof. Next, the captured image processing unit 230 detects an area that most closely matches the reference pattern from the captured image generated by the camera 22. It can be said that the areas detected from the cameras 21 and 22 are the same birds. Then, the captured image processing unit 230 includes the center coordinates (x1, y1) of the reference pattern included in the captured image of the camera 21 and the center coordinates (x2, y2) of the region corresponding to the reference pattern included in the captured image of the camera 22. ) Is calculated.

  Thereafter, the captured image processing unit 230 converts x1, which is the x coordinate of the center coordinates (x1, y1), into an angle θ1 in the horizontal direction with respect to the optical axis of the camera 21, and uses the x coordinates of the center coordinates (x2, y2). A certain x2 is converted into an angle θ2 in the horizontal direction with respect to the optical axis of the camera 22. Here, when the captured image processing unit 230 knows in advance the correspondence between the number of pixels from the center of the captured image and the angle with respect to the optical axis, which is determined according to the viewing angles of the cameras 21 and 22 and the resolution of the captured image. To do. The captured image processing unit 230 calculates the distance r between the cameras 21 and 22 and the flock of birds as shown in FIG. 13 based on the angles θ1 and θ2 calculated as described above and the optical axis distance l. That is, the distance r is calculated from the following formula: r × (tan (θ1) + tan (θ2)) = 1.

  Note that the captured image processing unit 230 may be able to calculate the distance r by a method other than the above triangulation. For example, one bird appears in the captured image generated by the camera 21 without overlapping other birds, and the monitoring device 200 has information indicating the correspondence between the imaging distance and the size of the bird on the captured image. When registered, the captured image processing unit 230 can estimate the distance r from the one bird. In this case, it is only necessary that at least one camera is connected to the monitoring apparatus 200.

FIG. 14 is a flowchart illustrating an exemplary procedure for generating reference data. The process shown in this flowchart is executed in the simulation apparatus 100.
(Step S <b> 11) The three-dimensional model generation unit 120 refers to the image condition table 111 stored in the model condition storage unit 110 and selects one layer number of the flock of birds.

(Step S <b> 12) The three-dimensional model generation unit 120 refers to the image condition table 111 and selects one row of bird flocks.
(Step S13) The three-dimensional model generation unit 120 arranges a plurality of bird shapes in a virtual three-dimensional space, and selects the number of layers selected in step S11 (the number of overlaps in the z-axis direction) and the selection in step S12. A bird flock model having the number of rows (the number of overlaps in the x-axis direction) is generated. The number of bird shapes arranged in the y-axis direction may be a predetermined number. At this time, the specific arrangement of the plurality of bird shapes may be automatically calculated by the three-dimensional model generation unit 120 so as to satisfy the above-mentioned property of the flock of birds, or may be designated by the user. May be.

  (Step S14) The three-dimensional model generation unit 120 determines whether the number of columns is a predetermined value (for example, one column). If the number of columns is a predetermined value, the process proceeds to step S15. If the number of columns is not a predetermined value (for example, 2 columns, 3 columns, 4 columns, 5 columns), the process proceeds to step S18.

  (Step S15) The three-dimensional model generation unit 120 refers to the camera coordinate table 112 stored in the model condition storage unit 110 and confirms the coordinates of three points corresponding to an elevation angle of 30 ° and a direction = front, side, and diagonal. Then, virtual cameras are arranged at these three points. Then, the three-dimensional model generation unit 120 generates a reference image viewed from each virtual camera and assigns an image ID.

  (Step S16) The three-dimensional model generation unit 120 refers to the camera coordinate table 112, confirms the coordinates of three points corresponding to an elevation angle of 60 ° and a direction = front, side, and diagonal, and virtually sets the three points. Place the camera. Then, the three-dimensional model generation unit 120 generates a reference image viewed from each virtual camera and assigns an image ID.

  (Step S <b> 17) The three-dimensional model generation unit 120 refers to the camera coordinate table 112, confirms coordinates corresponding to an elevation angle of 90 ° and direction = horizontal, and places a virtual camera at that point. Then, the three-dimensional model generation unit 120 generates a reference image viewed from the arranged virtual camera and assigns an image ID. Thereafter, the process proceeds to step S19.

  (Step S18) The three-dimensional model generation unit 120 refers to the camera coordinate table 112 stored in the model condition storage unit 110 and confirms the coordinates of three points corresponding to an elevation angle of 0 ° and a direction = front, side, and diagonal. Then, virtual cameras are arranged at these three points. Then, the three-dimensional model generation unit 120 generates a reference image viewed from each virtual camera and assigns an image ID.

  (Step S19) The three-dimensional model generation unit 120 determines whether all the column numbers have been selected in Step S12 with respect to the number of layers selected in Step S11. If the total number of columns is selected, the process proceeds to step S20. If there is an unselected number of columns, the process proceeds to step S12.

  (Step S20) The three-dimensional model generation unit 120 determines whether all the layer numbers have been selected in Step S11. If the total number of layers is selected, the process proceeds to step S21 described below. If there is an unselected number of layers, the process proceeds to step S11.

FIG. 15 is a flowchart (continued) illustrating an example of a procedure for generating reference data.
(Step S21) The reference image processing unit 130 selects one of a plurality of reference images (for example, 95 reference images) generated through the above steps S15 to S18.

  (Step S22) The reference image processing unit 130 includes the number of bird shapes (number of birds) arranged when the reference image selected in Step S21 is generated, and a virtual camera that took the reference image. The distance from the origin is confirmed with the 3D model generation unit 120.

  (Step S23) The reference image processing unit 130 cuts out a target area in which the shape of the flock of birds is captured from the reference image selected in step S21 by the method described above. That is, the reference image processing unit 130 extracts a rectangular region (or a predetermined shape such as an ellipse) that surrounds a set of dark pixels whose luminance is less than the threshold value. In detecting a pixel whose luminance is less than the threshold value, the reference image processing unit 130 may binarize the reference image and convert it into a monochrome image.

  (Step S24) The reference image processing unit 130 calculates the area of the target region cut out in step S24. When the target region is a rectangle, the area can be calculated as the number of vertical pixels × the number of horizontal pixels. When the target region has a shape other than a rectangle such as an ellipse, for example, the area can be calculated by substituting the number of vertical pixels and the number of horizontal pixels into a predetermined mathematical expression.

  (Step S25) The reference image processing unit 130 calculates the brightness of the target region cut out in step S24. The brightness of the target area represents the ratio between the number of bright pixels whose luminance is greater than or equal to a threshold in the target area and the number of dark pixels whose luminance is lower than the threshold. The value indicating the brightness is calculated, for example, as the number of bright pixels / the number of dark pixels, or the number of bright pixels / the area of the target region.

  (Step S26) The reference image processing unit 130 calculates the number of light spots from the target region cut out in step S23 by the method as described above. That is, the reference image processing unit 130 is an area composed of pixels surrounded by pixels whose luminance is less than the threshold, and whose size (for example, the area, the number of pixels on one side, etc.) is equal to or larger than the threshold. Are detected as light spots, and the number of detected light spots is counted.

  (Step S27) The reference image processing unit 130 generates reference data including distance, area, brightness, the number of light spots, and the number of birds in association with the image ID of the reference image selected in Step S21. Then, the reference image processing unit 130 determines whether all the reference images generated by the 3D model generation unit 120 have been selected in step S21. If all reference images have been selected, the process ends. If there is an unselected reference image, the process proceeds to step S21.

  FIG. 16 is a diagram illustrating an example of the reference data table. The reference data table 141 is generated by the reference image processing unit 130 of the simulation apparatus 100 and stored in the reference data storage unit 140. Further, the reference data table 141 is copied from the reference data storage unit 140 to the reference data storage unit 210 of the monitoring device 200. The reference data table 141 includes items of image ID, distance, area, brightness, number of light spots, and number of birds.

  In the image ID item, an image ID defined in the image condition table 111 is set. However, the reference data table 141 may include information on imaging conditions such as an elevation angle and a flight direction corresponding to the image ID instead of the image ID. In the distance item, a distance between the virtual camera that has captured the reference image and the origin is set. In the example of FIG. 16, the distance is fixed at 20. In the area item, the area of the target region in which the shape of the flock of birds is reflected is set. In the brightness item, a value indicating the brightness of the target area, that is, the ratio between the bright pixels and the dark pixels included in the target area is set. In the item of light spot, the number of light spots appearing in the reference image is set. In the item of the number of birds, the number of bird shapes reflected in the reference image is set.

FIG. 17 is a flowchart illustrating an exemplary procedure for estimating the number of birds. The process shown in this flowchart is executed in the monitoring apparatus 200.
(Step S31) The captured image acquisition unit 220 acquires the captured image Img1 (t) at time t from the camera 21. The captured image acquisition unit 220 acquires the captured image Img1 (t + 1) at time t + 1 after a predetermined time (for example, about several hundred milliseconds to several seconds) from the camera 21 from the camera 21. However, the captured image acquisition unit 220 may use a captured image at time t−1 that is a predetermined time earlier. Further, the captured image acquisition unit 220 acquires a captured image Img2 (t) at time t (or a time sufficiently close to time t) from the camera 22.

  (Step S <b> 32) The captured image acquisition unit 220 checks the elevation angles of the cameras 21 and 22 and the distance between the optical axes between the camera 21 and the camera 22. The elevation angle and the distance between the optical axes are registered in the monitoring apparatus 200 when the cameras 21 and 22 are installed, for example.

  (Step S33) The captured image processing unit 230 detects, from the captured image Img1 (t), one or more target regions in which a flock of birds is captured by the method described above. In other words, the captured image processing unit 230 detects a rectangular region (or a predetermined shape such as an ellipse) that surrounds a set of dark pixels whose luminance is less than the threshold so that no dark pixels are placed on the frame.

(Step S34) The captured image processing unit 230 selects one of the target areas detected from the captured image Img1 (t) in step S33.
(Step S35) The captured image processing unit 230 uses the target area of the captured image Img1 (t) selected in Step S34 and the area of the captured image Img1 (t + 1) corresponding to the target area to perform the cameras 21, 22. Determine the flight direction of the flock of birds against. The region of the captured image Img1 (t + 1) corresponding to the target region may be detected by the same method as the target region, or may be detected by pattern matching using the target region as a reference pattern. The flight direction is any one of “front”, “lateral”, and “oblique”. Details of the determination method will be described later.

  (Step S36) The captured image processing unit 230 calculates the distance between the cameras 21 and 22 and the flock of birds using the captured image Img1 (t) and the captured image Img2 (t) by the method as described above. To do. For example, the captured image processing unit 230 calculates the angle θ1, from the center coordinates of the target area of the captured image Img1 (t) selected in step S34 and the central coordinates of the area of the captured image Img1 (t + 1) corresponding to the target area. θ2 is calculated. The captured image processing unit 230 calculates the distance by the triangulation method from the angles θ1 and θ2 and the distance between the optical axes confirmed in step S32. However, the captured image processing unit 230 may calculate the distance using the image pattern of the region other than the target region selected in step S34, or calculates the distance only once for the entire captured image Img1 (t). You may make it do.

  (Step S37) The captured image processing unit 230 calculates the area and brightness of the target region of the captured image Img1 (t) selected in Step S34. The area can be calculated from the number of vertical pixels and the number of horizontal pixels of the target region. The brightness can be calculated from the number of pixels whose luminance is greater than or equal to a threshold value and the number of pixels whose luminance is less than the threshold value. The captured image processing unit 230 counts the number of light spots that appear in the target region. The light spot in the target region is a region that is surrounded by pixels whose luminance is less than the threshold and whose luminance is equal to or higher than the threshold, and whose size (for example, the area and the number of pixels on one side) is equal to or higher than the threshold. . Here, as the size threshold, a value depending on the distance calculated in step S36 is used. For example, when the threshold value of the size used when detecting the light spot of the reference image in the simulation apparatus 100 is N, N × distance of the captured image / distance of the reference image (for example, 20) is the size used here. The threshold value.

  (Step S38) The captured image processing unit 230 associates with the target area of the captured image Img1 (t) selected in Step S34, and captures the captured image data including the elevation angle, the flight direction, the distance, the area, the brightness, and the number of light spots. Generate. Then, the captured image processing unit 230 determines whether all the detected target areas have been selected in step S34. If all target areas have been selected, the process proceeds to step S41. If there is an unselected target area, the process proceeds to step S34.

FIG. 18 is a flowchart (continued) showing an example of the procedure for estimating the number of birds.
(Step S41) The bird count calculation unit 240 selects one target region detected from the captured image Img1 (t) by the captured image processing unit 230, and acquires imaging data of the selected target region.

  (Step S42) Based on the model condition information stored in the reference data storage unit 210, the bird count calculation unit 240 searches for a reference image having the same elevation angle and flight direction as the target region selected in step S41. Reference data of a reference image is acquired from the reference data storage unit 210. At this time, a plurality of reference images having different numbers of layers are searched. If there is no reference image having the same elevation angle as the target region selected in step S41, a reference image having the closest elevation angle is searched.

(Step S43) The bird count calculation unit 240 corrects the number of light spots of each reference image searched in step S42 so as to match the scale of the target region selected in step S41. Specifically, for each acquired reference data, the bird count calculation unit 240 calculates the number of light spots indicated by the reference data × (distance indicated by the imaging data / distance indicated by the reference data) ² × (area indicated by the imaging data / reference The area indicated by the data) is calculated as the number of light spots after correction.

  (Step S44) The bird count calculation unit 240 selects, from the reference images searched in step S42, a reference image that is closest to the target area selected in step S41 and has the same brightness / light spot number. Specifically, the number-of-birds calculation unit 240 has a difference between the brightness indicated by the imaging data and the brightness indicated by the reference data within a threshold, and the number of light spots indicated by the imaging data and the number of light spots corrected in step S43. A reference image whose difference is within a threshold is selected. As a result, the number of layers of the flock of birds appearing in the target area is estimated. If there is no reference image that is sufficiently close in both the target area and the brightness and the number of light spots, the reference image that has the closest brightness is selected.

In the above description, the number of light spots in the reference image is corrected to match the scale of the target area. However, the number of light spots in the target area may be corrected to match the scale of the reference image.
(Step S45) The bird count calculation unit 240 corrects the number of birds in the reference image selected in step S44 so as to match the scale of the target area selected in step S41. Specifically, the bird number calculation unit 240 calculates the number of birds indicated by the reference data × (distance indicated by the imaging data / distance indicated by the reference data) ² × (area indicated by the imaging data / area indicated by the reference data). . Then, the bird count calculation unit 240 estimates that the corrected bird count is the number of birds that appear in the target area.

  (Step S46) The bird count calculation unit 240 determines whether or not all target regions detected from the captured image Img1 (t) have been selected in step S41. If all target areas have been selected, the process proceeds to step S47. If there is an unselected target area, the process proceeds to step S41.

(Step S47) The number-of-birds calculation unit 240 adds up the number of birds in each target area calculated in Step S45, and estimates that the number of birds appears in the captured image Img1 (t).
FIG. 19 is a flowchart illustrating a procedure example of the flight direction determination. The processing shown in this flowchart is executed in step S35 described above.

  (Step S351) The captured image processing unit 230 calculates the area of the target region detected from the captured image Img1 (t). Further, the captured image processing unit 230 calculates the coordinates (center coordinates) of the center point of the target area in the captured image Img1 (t). The center coordinates are represented by, for example, an x coordinate and ay coordinate when the upper left of the captured image Img1 (t) is the origin.

(Step S352) The captured image processing unit 230 calculates an area and center coordinates for the region of the captured image Img1 (t + 1) corresponding to the target region, as in the target region.
(Step S353) The captured image processing unit 230 calculates a distance between the center coordinates in the captured image Img1 (t) calculated in Step S351 and the center coordinates in the captured image Img1 (t + 1) calculated in Step S352. Then, the captured image processing unit 230 determines whether the calculated coordinate distance is less than the threshold value. If the coordinate distance is less than the threshold, the process proceeds to step S354. If the coordinate distance is greater than or equal to the threshold, the process proceeds to step S355.

  (Step S354) The captured image processing unit 230 determines that the flight direction of the flock of birds with respect to the cameras 21 and 22 is “front”. This is because if the change in the position of the flock of birds on the captured image is small, the flock of birds is considered to have moved (1) in the direction toward the camera or (2) in the direction away from the camera. is there. Then, the process ends.

  (Step S355) The captured image processing unit 230 calculates the difference between the area in the captured image Img1 (t) calculated in step S351 and the area in the captured image Img1 (t + 1) calculated in step S352. Then, the captured image processing unit 230 determines whether the calculated area difference is less than the threshold value. If the area difference is less than the threshold, the process proceeds to step S356. If the area difference is greater than or equal to the threshold, the process proceeds to step S357.

  (Step S356) The captured image processing unit 230 determines that the flight direction of the flock of birds with respect to the cameras 21 and 22 is “horizontal”. This is because if the position of the flock of birds is large and the change in size is small, the flock of birds will either (3) cross the camera field of view from left to right, or (4) the camera field of view from right to left. This is because it is considered that the image is moving in a crossing direction or is picked up at an elevation angle of 90 °. Then, the process ends.

  (Step S357) The captured image processing unit 230 determines that the flight direction of the flock of birds with respect to the cameras 21 and 22 is “oblique”. This is because when the position of the flock of birds is large and the size of the flock is large, the flock of birds (4) crosses the camera field of view from right to left, and (5) approaches the camera diagonally to the left with respect to the optical axis. Direction, (6) move away from the camera diagonally to the left with respect to the optical axis, (7) move away from the camera obliquely to the right with respect to the optical axis, or (8) move away from the camera obliquely to the right with respect to the optical axis. It is because it is thought that it is doing.

  FIG. 20 is a diagram illustrating an example of a bird count estimation result. Here, with respect to one target area detected from the captured image, imaging data of elevation angle = 60 °, flight direction = diagonal, distance = r, area = s0, brightness = b0, and number of light spots = m0 is generated. To do.

  First, the monitoring device 200 includes five reference images (reference images # 6, # 25, and # 44) having different number of layers corresponding to the elevation angle = 60 ° and the flight direction = oblique from the reference images # 1 to # 95. , # 63, # 82). Here, the reference image # 6 corresponds to the number of layers = 1, the reference image # 25 corresponds to the number of layers = 2, the reference image # 44 corresponds to the number of layers = 3, and the reference image # 63 corresponds to the number of layers = 4 corresponds to the number of layers = 5.

Next, the monitoring apparatus 200 corrects the number of light spots of each retrieved reference image. For example, for reference image # 25, when distance = 20, area = s25, brightness = b25, and number of light spots = m25, the number of corrected light spots is m25 × (r / 20) ² × (s0 / s25 ) Is calculated. Similarly, the number of light spots m6, m44, m63, and m82 of the reference images # 6, # 44, # 63, and # 82 are also corrected. The monitoring apparatus 200 compares the brightness b0 with the brightness b6, b25, b44, b63, and b82 of the reference images # 6, # 25, # 44, # 63, and # 82. Moreover, the monitoring apparatus 200 compares the number of light spots m0 with the number of light spots after correction of the reference images # 6, # 25, # 44, # 63, and # 82.

Here, it is assumed that the brightness b25 of the reference image # 25 is closest to the brightness b0, and the corrected light spot number of the reference image # 25 is closest to the light spot number m0. Then, the monitoring apparatus 200 estimates that the flock of birds reflected in the target area forms two layers. Then, when the number of birds in the reference image # 25 = t25, the monitoring apparatus 200 estimates that t25 × (r / 20) ² × (s0 / s25) is the number of birds reflected in the target area. In this way, the monitoring device 200 can estimate the number of layers of birds and the number of birds from a captured image in which a plurality of birds overlap.

  According to the information processing system of the second embodiment, by simulating the projection of a flock of birds, reference data indicating the influence on the brightness and the number of light spots when the number of layers and the number of columns change is obtained. Prepared. Then, by comparing the brightness and the number of light spots of the captured images generated by the cameras 21 and 22 with the reference data, the number of layers of flocks, the number of columns, and the number of birds are estimated. For this reason, even if a plurality of birds are captured in the captured image, the number of birds can be estimated with higher accuracy than the method of detecting the region in which the birds are captured by pattern recognition. Further, in the second embodiment, projection of a flock of birds is simulated while changing imaging conditions such as an elevation angle and a flight direction. Therefore, it is possible to further improve the estimation accuracy of the number of birds by narrowing down the reference data to be used in consideration of the positional relationship between the cameras 21 and 22 and the flock of birds.

  As described above, the information processing of the first embodiment can be realized by causing the information processing apparatus 10 to execute a program, and the information processing of the second embodiment is performed by the simulation apparatus 100 or the monitoring apparatus 200. This can be realized by executing each program. Such a program can be recorded on a computer-readable recording medium (for example, the recording medium 25). As the recording medium, for example, a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like can be used. Magnetic disks include FD and HDD. Optical discs include CD, CD-R (Recordable) / RW (Rewritable), DVD, and DVD-R / RW.

  When distributing the program, for example, a portable recording medium in which the program is recorded is provided. It is also possible to store the program in a storage device of another computer and distribute the program via a network. The computer stores, for example, a program recorded on a portable recording medium or a program received from another computer in a storage device (for example, the HDD 203), and reads and executes the program from the storage device. However, a program read from a portable recording medium may be directly executed, or a program received from another computer via a network may be directly executed. Further, at least a part of the information processing described above can be realized by an electronic circuit such as a DSP, ASIC, or PLD (Programmable Logic Device).

DESCRIPTION OF SYMBOLS 10 Information processing apparatus 11 Memory | storage part 11a, 11b, 11c Sample information 12 Calculation part 13 Captured image

Claims (7)

An information processing apparatus that processes captured images of a plurality of flying objects,
For each of a plurality of sample images having different number of flying objects, the number of spot areas that are surrounded by an area whose brightness is less than the threshold and the brightness is equal to or greater than the threshold, and the number of flying objects reflected in the sample image A storage unit for storing sample information including:
The number of spot areas included in the captured image is calculated, sample information corresponding to the calculated number of spot areas is selected from the sample information corresponding to the plurality of sample images, and the number of flying objects indicated by the selected sample information And an arithmetic unit for estimating the number of flying objects shown in the captured image. The sample information corresponding to each sample image further includes a brightness value indicating the proportion of the area where the brightness is equal to or greater than the threshold in the sample image,
The information processing apparatus according to claim 1, wherein the calculation unit calculates a brightness value of the captured image, and selects sample information according to the calculated number of spot regions and the calculated brightness value. The storage unit stores sample information for each of a plurality of sample images having different combinations of the number of overlaps and the flight direction of the flying object,
The calculation unit determines a flight direction of the plurality of flying objects from two or more captured images obtained by copying the plurality of flying objects, and narrows down sample information candidates to be selected based on the determined flight direction. The information processing apparatus described.   The calculation unit corrects the number of spot areas indicated by each sample information based on at least one of the distance between the plurality of flying objects and the imaging device and the size of the captured image, and corrects each sample information. The information processing apparatus according to claim 1, wherein sample information is selected by comparing a result with the calculated number of spot areas.   The arithmetic unit corrects the number of flying objects indicated by the selected sample information based on at least one of a distance between the plurality of flying objects and an imaging device and a size of the captured image, and the correction result is captured by the imaging The information processing apparatus according to claim 1, wherein the information processing apparatus estimates the number of flying objects in the image. A computer-based image processing method for processing captured images of a plurality of flying objects,
The computer is
From the captured image, calculate the number of spot areas that are surrounded by an area with a brightness less than the threshold, and the brightness is greater than or equal to the threshold,
Depending on the number of spot areas calculated from sample information corresponding to multiple sample images with different number of overlapping objects, including the number of spot areas in the sample image and the number of objects in the sample image. Selected sample information,
An image processing method for estimating the number of flying objects shown in the captured image based on the number of flying objects indicated by the selected sample information. A program for processing captured images of a plurality of flying objects,
On the computer,
From the captured image, calculate the number of spot areas that are surrounded by an area with a brightness less than the threshold, and the brightness is greater than or equal to the threshold,
Depending on the number of spot areas calculated from sample information corresponding to multiple sample images with different number of overlapping objects, including the number of spot areas in the sample image and the number of objects in the sample image. Selected sample information,
A program for executing processing for estimating the number of flying objects shown in the captured image based on the number of flying objects indicated by the selected sample information.

Images