JP7492059B2 - Periphery monitoring device and surroundings monitoring method - Google Patents

Description

The present invention primarily relates to a surroundings monitoring device that uses camera images.

Patent Document 1 discloses an intruding object monitoring system that uses an imaging device such as a camera as an image input means. Patent Document 1 also discloses that when detecting an intruding ship at sea, a mask area is automatically set to eliminate false detections due to light reflections and the like at sea.

JP 2002-279429 A

However, the configuration of Patent Document 1 creates a mask area based on changes in pixel values, so when light reflections on the ocean are confusable with ships, the mask area cannot be set correctly, resulting in false detections or missed detections, leaving room for improvement.

The present invention has been made in consideration of the above circumstances, and its purpose is to appropriately set a target target area that enables highly accurate target detection.

The problem that the present invention aims to solve is as described above. Next, we will explain the means for solving this problem and its effects.

The perimeter monitoring device of the present invention includes a nautical chart data storage unit that stores nautical chart data, a captured image input unit that inputs an image captured by a camera, an area setting unit that sets a detection target area based on the nautical chart data, a target information acquisition unit that acquires target information of a target detected in the detection target area, and a composite image generation unit that generates a composite image by combining the target information at a position in the captured image corresponding to the position where the target was detected.

The target information acquisition unit acquires target information of the target detected by performing image recognition on the detection target area, and the boundary separating the inside and outside of the detection target area where the image recognition is performed is represented by a closed polygonal line figure, and the area setting unit stores position information of multiple vertices of the polygonal line figure.

In addition, in the surroundings monitoring device of the present invention, the composite image generating unit generates the composite image including a graphic representing the detection target area.

In addition, in the perimeter monitoring device of the present invention, the detection target area in which the image recognition is performed can be modified and then specified by a user operation.

1 is a block diagram showing an overall configuration of a surroundings monitoring device according to an embodiment of the present invention; FIG. 2 is a side view showing various devices provided in a port monitoring facility. FIG. 2 is a diagram showing an example of a captured image input from a camera. A conceptual diagram illustrating three-dimensional scene data constructed by placing virtual reality objects in a three-dimensional virtual space, and a projection screen placed in the three-dimensional virtual space. FIG. 4 is a diagram showing a composite video output by a data synthesis unit. A conceptual diagram showing how a virtual reality object of a land-water boundary is placed in a three-dimensional virtual space. FIG. 13 is a diagram showing a composite image in which a land-water boundary is composited with a captured image. 4 is a flowchart illustrating a process performed in the surroundings monitoring device.

Next, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the overall configuration of a perimeter monitoring device 1 according to one embodiment of the present invention. FIG. 2 is a side view showing various devices provided in a port monitoring facility 4.

The perimeter monitoring device 1 shown in FIG. 1 is installed in, for example, a port monitoring facility 4 as shown in FIG. 2. The perimeter monitoring device 1 can generate images based on images captured by a camera (imaging device) 3, on which information that supports monitoring of the movements of ships (monitoring targets) in the port is superimposed. Therefore, the perimeter monitoring device 1 functions as an image generating device. The images generated by the perimeter monitoring device 1 are displayed on the display 2.

Display 2 can be configured, for example, as a display placed at a monitoring desk where an operator performs monitoring duties at port monitoring facility 4. However, display 2 is not limited to the above, and can be, for example, the display of a portable computer carried by a monitoring assistant who monitors the surrounding situation from port monitoring facility 4.

The perimeter monitoring device 1 generates a composite image, which is an output image to the display 2, by combining an image of the surroundings captured by a camera 3 installed in a port monitoring facility 4 with a graphic that represents additional display information (described in detail later) of the surroundings in a virtual reality manner.

Next, referring mainly to FIG. 1, we will explain the camera 3 and various devices electrically connected to the perimeter monitoring device 1.

Camera 3 is configured as a visible light video camera that captures the surroundings of port monitoring facility 4. Camera 3 is configured as a wide-angle camera, and is mounted facing slightly downward in a high location with good visibility. This camera 3 has a live output function, and can generate video data (image data) as the result of the capture in real time and output it to perimeter monitoring device 1.

In principle, camera 3 takes pictures from a fixed point. However, camera 3 is attached via a rotation mechanism (not shown), and its shooting direction can be changed by receiving a signal from perimeter monitoring device 1 instructing pan/tilt operation.

In this embodiment, the perimeter monitoring device 1 is electrically connected to the camera 3, as well as various other devices such as an AIS receiver 9 and a radar device (target detection unit) 12.

The AIS receiver 9 receives AIS information transmitted from ships. The AIS information includes various information such as the position (latitude and longitude) of the ship navigating the monitored port, the length and width of the ship, the type and identification information of the ship, the speed, course and destination of the ship, etc.

The radar device 12 can detect targets such as ships present in the harbor being monitored. The radar device 12 also has a known target tracking function (Target Tracking, TT) that can capture and track targets, and can determine the position and velocity vector (TT information) of the target.

The surroundings monitoring device 1 is connected to a keyboard 36 and a mouse 37 that are operated by the user. The user can issue various instructions related to the generation of images by operating the keyboard 36 and the mouse 37. These instructions include pan/tilt operations of the camera 3, etc.

Next, the configuration of the perimeter monitoring device 1 will be described in detail, mainly with reference to FIG. 1.

As shown in FIG. 1, the perimeter monitoring device 1 includes a captured image input unit 21, an additional display information acquisition unit (target information acquisition unit) 17, a camera position/orientation setting unit 25, a radar position/orientation setting unit 26, an image recognition unit (target detection unit) 28, an image recognition area setting unit (area setting unit) 31, a tracking target area setting unit (area setting unit) 32, a nautical chart data storage unit 33, and a composite image generation unit 20.

Specifically, the perimeter monitoring device 1 is configured as a known computer, and includes a CPU, ROM, RAM, and HDD, which are not shown. Furthermore, the perimeter monitoring device 1 includes a GPU for performing three-dimensional image processing at high speed, which will be described later. For example, the HDD stores software for executing the perimeter monitoring method of the present invention. This hardware and software work together to cause the perimeter monitoring device 1 to function as a captured image input unit 21, an additional display information acquisition unit 17, a camera position/orientation setting unit 25, a radar position/orientation setting unit 26, an image recognition unit 28, an image recognition area setting unit 31, a tracking target area setting unit 32, a nautical chart data storage unit 33, and a composite image generation unit 20, etc. Reference numeral 35 denotes a processing circuitry.

The captured image input unit 21 can input the image data output by the camera 3 at, for example, 30 frames per second. The captured image input unit 21 outputs the input image data to the image recognition unit 28 and the composite image generation unit 20 (data synthesis unit 23 described below).

The additional display information acquisition unit 17 acquires information (additional display information, target information) to be additionally displayed on the image captured by the camera 3, based on the information input from the AIS receiver 9 and the radar device 12 to the perimeter monitoring device 1 and the information acquired by the image recognition unit 28. Various types of additional display information are possible, but for example, it can be the position and speed of the ship obtained from the AIS receiver 9, the position and speed of the target obtained from the radar device 12, the position and speed of the target obtained from the image recognition unit 28, etc. The additional display information acquisition unit 17 outputs the acquired information to the composite image generation unit 20. The additional display information will be described in detail later.

The camera position/orientation setting unit 25 can set the position (shooting position) and orientation of the camera 3 in the port monitoring facility 4. The position information of the camera 3 is specifically information indicating the latitude, longitude, and altitude. The orientation information of the camera 3 is specifically information indicating the azimuth angle and depression angle. This information can be obtained, for example, by performing measurements during the installation work of the camera 3. As described above, the orientation of the camera 3 can be changed within a predetermined angle range, but when the camera 3 is panned/tilted, the latest orientation information is set in the camera position/orientation setting unit 25 accordingly. The camera position/orientation setting unit 25 outputs the set information to the composite image generation unit 20, the additional display information acquisition unit 17, and the image recognition area setting unit 31.

The radar position/orientation setting unit 26 can set the position and orientation of the antenna of the radar device 12. The antenna position information is specifically information indicating latitude and longitude. The antenna orientation information is specifically information indicating direction. Although an antenna normally rotates within a horizontal plane, the antenna orientation here refers to the direction (reference direction) that serves as the reference for the detection direction of the radar device 12. This information can be obtained by performing measurements during the installation work of the antenna.

The antenna position and orientation information set in the radar position/orientation setting unit 26 of the perimeter monitoring device 1 is also set in the radar device 12. The radar position/orientation setting unit 26 outputs the set information to the additional display information acquisition unit 17 and the tracking target area setting unit 32.

The image recognition unit 28 recognizes targets such as ships, divers, and drifting objects by cutting out parts of the images acquired from the captured image input unit 21 that are thought to be ships, etc., and comparing them with a pre-registered target image database. Specifically, the image recognition unit 28 detects moving objects using a frame difference method or the like, cuts out areas where differences occur, and compares them with the image database. As a matching method, any suitable publicly known method such as template matching can be used. Image recognition can also be achieved using other publicly known methods, such as a neural network. The image recognition unit 28 outputs information on the position recognized on the image for each recognized target to the additional display information acquisition unit 17.

The image recognition area setting unit 31 sets an area within the image input to the image recognition unit 28 from the captured image input unit 21 that is to be the target of image recognition by the image recognition unit 28. In this embodiment, since the image recognition unit 28 is assumed to recognize objects floating on the water surface, the image recognition area setting unit 31 normally sets only a portion of the area in which the water surface appears as the target area for image recognition.

The boundary separating the inside and outside of the target area for image recognition can be represented, for example, by a closed polygonal line figure. The image recognition area setting unit 31 stores the position information (latitude and longitude) of multiple vertices of the polygonal line figure.

The camera 3 is usually installed so as to capture a large image of the water surface, but as shown in the example of Figure 3, the field of view of the camera 3 may include land as well as the water surface. In this case, by setting the target area for image recognition in the image recognition area setting unit 31 so as to exclude the land portion, it is possible to prevent the image recognition unit 28 from recognizing, for example, a car traveling on a road as a target. As a result, it is possible to improve the recognition accuracy and reduce the processing load.

The tracking target area setting unit 32 in FIG. 1 sets an area in which the radar device 12 will track targets using the target tracking function described above. Since the radar device 12 is designed to detect objects floating on the water surface, the tracking target area setting unit 32 also sets the target tracking target to exclude land areas, just like the image recognition area setting unit 31. This allows only targets on the water surface to be properly detected.

The boundary separating the inside and outside of the target area tracked by the radar device 12 can be represented by a closed polygonal line figure, similar to the target area for image recognition. The tracking target area setting unit 32 stores the position information (latitude and longitude) of multiple vertices of the polygonal line figure.

The chart data storage unit 33 stores chart data. For example, an electronic navigational chart is used as the chart data. The chart data includes vector data of the boundary between the water surface and land (land-water boundary). The boundary can be expressed in any manner, but for example, it is possible to express the outline of the land area as a closed figure made up of broken lines, and to describe the position information (latitude and longitude) of each vertex in order.

The nautical chart data storage unit 33 outputs vector data indicating the land-water boundary to the image recognition area setting unit 31 and the tracking target area setting unit 32. This makes it easy to set areas for the image recognition area setting unit 31 and the tracking target area setting unit 32.

The composite image generating unit 20 generates a composite image to be displayed on the display 2. The composite image generating unit 20 includes a three-dimensional scene generating unit (display data generating unit) 22 and a data synthesis unit (display output unit) 23.

As shown in FIG. 4, the three-dimensional scene generating unit 22 constructs a three-dimensional virtual reality scene by arranging virtual reality objects 41v, 42v, ... corresponding to the additional display information in the three-dimensional virtual space 40. This generates three-dimensional scene data (three-dimensional display data) 48, which is data of the three-dimensional scene. Details of the three-dimensional scene will be described later.

The data synthesis unit 23 generates figures that three-dimensionally express the additional display information by drawing the three-dimensional scene data 48 generated by the three-dimensional scene generation unit 22, and performs processing to output the synthetic image shown in FIG. 5, that is, an image obtained by synthesizing the figures 41f, 42f, ... with the image captured by the camera 3. As shown in FIG. 5, in this synthetic image, figures 41f, 42f, ... indicating the additional display information are superimposed on the image captured by the camera 3. The data synthesis unit 23 outputs the generated synthetic image to the display 2. The details of the figure generation processing and the data synthesis processing will be described later.

Next, the additional display information acquired by the aforementioned additional display information acquisition unit 17 will be described in detail. Figure 3 is a conceptual diagram illustrating an example of additional display information to be displayed on the surroundings monitoring device 1.

The additional display information is information that is additionally displayed on the image captured by the camera 3, and various types are possible depending on the purpose and function of the equipment connected to the perimeter monitoring device 1. For example, for the AIS receiver 9, the received AIS information (e.g., the position and orientation of the ship, etc.) can be the additional display information. For the radar device 12, the position and speed of the detected target can be the additional display information. This information is input from each equipment to the perimeter monitoring device 1 in real time.

In addition, in this embodiment, the additional display information includes the position and speed of the target identified by the image recognition unit 28 through image recognition.

The target position and velocity vector contained in the information obtained from the radar device 12 are relative to the position and orientation of the antenna of the radar device 12. Therefore, the additional display information acquisition unit 17 converts the target position and velocity vector obtained from the radar device 12 into an earth reference based on the information obtained from the radar position/orientation setting unit 26.

Similarly, the target position and velocity vector obtained from the image recognition unit 28 are relative to the position and orientation of the camera 3. Therefore, the additional display information acquisition unit 17 converts the target position and velocity vector obtained from the image recognition unit 28 into an earth reference based on the information obtained from the camera position/orientation setting unit 25.

An example of additional display information is described below. In the situation shown in the camera image of FIG. 3, the AIS information acquired by the AIS receiver 9 detects that a large vessel 41r is sailing toward the right side of the camera image within the harbor. The radar device 12 detects that a small vessel 42r is sailing toward the right side of the camera image at high speed. Furthermore, the image recognition unit 28 detects that a small vessel 43r is sailing toward the near side at high speed.

Each piece of additional display information includes at least information indicating the position (latitude and longitude) of the point on the sea surface (water surface) where it is located. For example, the additional display information indicating ships 41r, 42r, and 43r includes information indicating the positions of the ships 41r, 42r, and 43r.

Next, the construction of a three-dimensional scene by the three-dimensional scene generating unit 22 and the synthesis of images by the data synthesis unit 23 will be described in detail with reference to FIG. 4. FIG. 4 is a conceptual diagram illustrating three-dimensional scene data 48 generated by placing virtual reality objects 41v, 42v, ... in a three-dimensional virtual space 40, and a projection screen 51 placed in the three-dimensional virtual space 40.

The three-dimensional virtual space 40 in which the virtual reality objects 41v, 42v, ... are placed by the three-dimensional scene generation unit 22 is configured with a Cartesian coordinate system as shown in FIG. 4. The origin of this Cartesian coordinate system is set at a point at zero altitude directly below the installation position of the camera 3. In the three-dimensional virtual space 40, the xz plane, which is a horizontal plane including the origin, is set to simulate the sea surface (water surface). In the example of FIG. 4, the coordinate axes are set so that the +z direction always coincides with the azimuth angle of the camera 3, the +x direction is to the right, and the +y direction is upward. Each point (coordinate) in this three-dimensional virtual space 40 is set to correspond to a real position around the camera 3.

Figure 4 shows an example in which virtual reality objects 41v, 42v, 43v are placed in a three-dimensional virtual space 40 in response to the port situation shown in Figure 3. In this embodiment, the virtual reality objects 41v, 42v, ... include a downward-pointing cone indicating the position of the recognized target (i.e., ships 41r, 42r, 43r) and an arrow indicating the velocity vector of the target. Both the cone and the arrow are three-dimensional shapes. The downward-pointing cone indicates that the target is located directly below it. The direction of the arrow indicates the direction of the target's velocity, and the length of the arrow indicates the magnitude of the velocity.

Each virtual reality object 41v, 42v, 43v is positioned on the xz plane or slightly above it so as to reflect the relative position of the additional display information it represents with respect to the camera 3, based on the azimuth angle of the camera 3. When determining the positions at which these virtual reality objects 41v, 42v, ... are positioned, a calculation is performed using the position and orientation of the camera 3 set by the camera position/orientation setting unit 25 shown in FIG. 1.

The three-dimensional scene generating unit 22 generates three-dimensional scene data 48 as described above. In the example of FIG. 4, the virtual reality objects 41v, 42v, ... are positioned with an orientation reference that has the origin directly below the camera 3, so that when the azimuth angle of the camera 3 changes from the state of FIG. 3, a new three-dimensional scene is constructed in which the virtual reality objects 41v, 42v, ... are rearranged, and the three-dimensional scene data 48 is updated. In addition, when the content of the additional display information is changed, for example, by the ships 41r, 42r, 43r moving from the state of FIG. 3, the three-dimensional scene data 48 is updated to reflect the latest additional display information.

Furthermore, the data synthesis unit 23 places a projection screen 51 in the three-dimensional virtual space 40, which determines the position and range in which the image captured by the camera 3 is projected. The position and orientation of the viewpoint camera 55 (described below) are set so that both the projection screen 51 and the virtual reality objects 41v, 42v, ... are included in the field of view, thereby achieving image synthesis.

The data synthesis unit 23 positions the viewpoint camera 55 so as to simulate the position and orientation of the camera 3 in the real space in the three-dimensional virtual space 40. The data synthesis unit 23 also positions the projection screen 51 so as to directly face the viewpoint camera 55. With regard to simulating the position of the camera 3, the position of the camera 3 can be obtained based on the setting value of the camera position/orientation setting unit 25 shown in FIG. 1.

The azimuth angle of the viewpoint camera 55 in the three-dimensional virtual space 40 does not change even if the azimuth angle changes due to the panning of the camera 3. Instead, when the data synthesis unit 23 pans the camera 3, it repositions the virtual reality objects 41v, 42v, ... in the three-dimensional virtual space 40 so that they rotate in the horizontal plane around the origin by the amount of the changed azimuth angle.

The depression angle of the viewpoint camera 55 is controlled so that it is always equal to the depression angle of the camera 3. In response to the change in depression angle caused by the tilting operation of the camera 3 (change in the depression angle of the viewpoint camera 55), the data synthesis unit 23 changes the position and orientation of the projection screen 51 placed in the three-dimensional virtual space 40 so that it always faces the viewpoint camera 55 directly.

Then, the data synthesis unit 23 generates a two-dimensional image by performing a known rendering process on the three-dimensional scene data 48 and the projection screen 51. More specifically, the data synthesis unit 23 places a viewpoint camera 55 in the three-dimensional virtual space 40, and defines a view frustum 56 that determines the range to be subjected to the rendering process, with the viewpoint camera 55 as its vertex and its line of sight as its central axis. Next, the data synthesis unit 23 converts the vertex coordinates of polygons that are located inside the view frustum 56, among the polygons that make up each object (virtual reality objects 41v, 42v, ... and the projection screen 51), into coordinates of a two-dimensional virtual screen that corresponds to the display area of the composite image on the display 2, by perspective projection. Then, based on the vertices placed on this virtual screen, a two-dimensional image is generated by performing pixel generation and processing at a predetermined resolution.

The two-dimensional image generated in this manner includes figures obtained by drawing the three-dimensional scene data 48 (in other words, figures resulting from the rendering of the virtual reality objects 41v, 42v, ...). In addition, in the process of generating the two-dimensional image, the image captured by the camera 3 is placed so as to be pasted onto the position corresponding to the projection screen 51. This allows the data synthesis unit 23 to synthesize the images.

Projection screen 51 is curved to fit a spherical shell centered on viewpoint camera 55, preventing distortion of the captured image due to perspective projection. Camera 3 is a wide-angle camera, and lens distortion occurs in the captured image as shown in FIG. 3, but this lens distortion is removed when the captured image is pasted onto projection screen 51. Any method can be used to remove lens distortion, but one possible method is to use a lookup table that associates pixel positions before and after correction. This allows for good alignment between the captured image and the three-dimensional virtual space 40 shown in FIG. 4.

Next, we will explain the relationship between the image captured by camera 3 and the composite image using an example.

Figure 5 shows the result of combining the captured image shown in Figure 3 with the above-mentioned two-dimensional image created by rendering the three-dimensional scene data 48 in Figure 4. However, in Figure 5, the part in which the image captured by camera 3 appears is shown by a dashed line for the sake of convenience so that it can be easily distinguished from the other parts (this is the same in other figures showing the composite image). In the composite image in Figure 5, figures 41f, 42f, and 43f representing additional display information are positioned so as to overlap the captured image.

The above figures 41f, 42f, ... are generated as a result of drawing the three-dimensional shapes of the virtual reality objects 41v, 42v, ... constituting the three-dimensional scene data 48 shown in FIG. 4 from a viewpoint that is in the same position and orientation as the camera 3. Therefore, even when the figures 41f, 42f, ... are superimposed on a realistic image captured by the camera 3, it is possible to ensure that almost no strange appearance occurs.

This makes it possible to obtain a natural and highly realistic augmented reality image in which the mark indicating the ship and the arrow indicating the speed appear to be floating in the air above the water surface. In addition, by gazing at the sea surface projected on the display 2, the user can obtain the necessary information without missing anything, as the figures 41f, 42f, ... representing the virtual reality are comprehensively within the user's field of vision.

As described above, lens distortion is removed from the captured image input from camera 3 when it is projected onto projection screen 51 in three-dimensional virtual space 40. However, data synthesis unit 23 re-adds lens distortion to the rendered synthesized image by inverse conversion using the above-mentioned lookup table. As can be seen by comparing Figures 5 and 3, this makes it possible to obtain a synthesized image that is less likely to look unnatural in relation to the camera images before synthesis. However, adding lens distortion may be omitted.

As shown in FIG. 5, the data synthesis unit 23 further synthesizes text information describing information useful for monitoring in positions near the figures 41f, 42f, and 43f in the synthesized image. The content of the text information is arbitrary, and various contents can be displayed, for example, information identifying the ship, information indicating the size of the ship, information indicating which device the information was obtained from, etc. Information identifying the ship can be obtained, for example, from AIS information. Information indicating the size of the ship can be obtained from AIS information, but it can also be obtained by calculation from the size of the image detected during image recognition by the image recognition unit 28, or from the size of the tracking echo obtained by the radar device 12. This makes it possible to realize a monitoring screen rich in information.

As shown in FIG. 5, in this embodiment, a figure 41f representing additional display information based on the AIS information obtained from the AIS receiver 9, a figure 42f representing additional display information based on the image recognition result by the image recognition unit 28, and a figure 43f representing additional display information based on the radar tracking result of the radar device 12 are synthesized into the captured image. This makes it possible to realize an integrated display, and the user can easily monitor information obtained from various information sources in a single synthesized image. As a result, the monitoring burden can be effectively reduced.

In FIG. 5, additional display information based on AIS information obtained from the AIS receiver 9 is displayed for the ship 41r. It is possible that the image recognition unit 28 recognizes the same ship 41r, or that the radar device 12 tracks it. Even in this case, the synthetic image generation unit 20 prioritizes displaying additional display information based on AIS information in the area of the ship 41r. This allows information with a high degree of reliability to be displayed with priority.

A compass scale 46 is displayed in the composite image of FIG. 5. The compass scale 46 is formed in an arc shape that connects the left and right edges of the screen. The compass scale 46 has numerical values written on it that indicate the compass corresponding to the image. This allows the user to intuitively grasp the compass direction that the camera 3 is looking in.

For example, if the camera 3 is tilted and the compass scale 46 overlaps with other figures 41f, 42f, 43f in the composite image, or overlaps with the water surface in the captured image, the data composition unit 23 automatically moves the position where the compass scale 46 is composited in the vertical direction of the image. This prevents the compass scale 46 from interfering with other displays.

Next, we will explain how to set the image recognition area using nautical chart data.

As described above, the image recognition area setting unit 31 sets the area in which the image recognition unit 28 performs image recognition. With the image captured by the camera 3 displayed on the display 2, the user can specify the above-mentioned area by specifying the target area for image recognition with the mouse 37 or the like.

When the user specifies the target area for image recognition, the nautical chart data storage unit 33 outputs vector data representing the land-water boundary from among the stored nautical chart data to the image recognition area setting unit 31. The image recognition area setting unit 31 outputs this boundary line vector data to the composite image generation unit 20.

The three-dimensional scene generation unit 22 of the composite image generation unit 20 generates three-dimensional scene data 48 in which a virtual reality object 49v representing the land-land boundary is placed on the xz plane, as shown in FIG. 6. The position at which the virtual reality object 49v is placed is based on the azimuth angle of the camera 3, and is placed so as to reflect the relative position of the land-land boundary with respect to the camera 3. The data synthesis unit 23 then performs rendering processing in exactly the same manner as described above, and outputs a composite image as shown in FIG. 7 to the display.

In the composite image of Figure 7, figure 49f, which is the result of rendering virtual reality object 49v, is placed in the composite image as if it were placed on the water surface in the image captured by camera 3. Since the boundary vector data should match the real boundary line, the shape of the water surface reflected by camera 3 and the shape of figure 49f are consistent.

The user can therefore easily and appropriately set the image recognition area by referring to figure 49f. For example, the user can directly specify the area surrounded by figure 49f shown in FIG. 7 as the image recognition area. The user can also deform the area by using the mouse or other methods to correct any misalignment with the captured image, and then specify it as the image recognition area.

The boundary vector data stored in the nautical chart data storage unit 33 can also be used to set the tracking target area in the tracking target area setting unit 32. The tracking target area can be calculated and determined based on the antenna position/orientation set in the radar position/orientation setting unit 26 and the boundary vector data so as to be limited to water areas. This allows the user to easily and accurately set the tracking target area.

The boundaries of the target area for image recognition and the target area for radar tracking can also be defined using other boundaries rather than the land-water boundary line. For example, contour lines corresponding to a specified depth from the data included in the nautical chart data can be used as the recognition boundary. Also, a boundary line can be created by offsetting the land-water boundary line a specified distance toward the water area, and this boundary line can be used as the recognition boundary. This makes it easier to realize monitoring according to the purpose, for example, when monitoring only large ships, the target area can be limited to an area with a specified water depth or greater.

The parameters of image recognition or radar tracking may be changed according to the information contained in the nautical chart data. For example, since large ships cannot navigate in shallow water areas, when the water depth obtained based on the nautical chart data is small, the image recognition unit 28 may perform template matching with an image database of small ships, or the radar device 12 may track only small radar echoes. This allows for appropriate monitoring.

Figure 8 shows a flowchart of a series of processes performed by the surroundings monitoring device 1.

When the flow in FIG. 8 starts, the perimeter monitoring device 1 stores the nautical chart data input from outside in the nautical chart data storage unit 33 as a preparatory step (step S101). Next, the perimeter monitoring device 1 inputs the captured image captured by the camera 3 from the captured image input unit 21 (step S102). The composite image generation unit 20 of the perimeter monitoring device 1 generates a composite image by combining this captured image with the boundary vector data obtained from the nautical chart data storage unit 33, and outputs the composite image to the display 2 (step S103).

The user sets the image recognition area appropriately while watching the screen of the display 2 (step S104). Steps S103 and S104 may be omitted, and the surroundings monitoring device 1 may automatically set the boundary of the image recognition area so that it is the same as the boundary line vector data.

After that, normal operation is started. More specifically, the perimeter monitoring device 1 inputs the captured image captured by the camera 3 from the captured image input unit 21 (step S105). Next, the image recognition unit 28 of the perimeter monitoring device 1 performs image recognition on the image recognition area, thereby acquiring target information of the detected target (step S106). Next, the composite image generation unit 20 generates a composite image by combining the target information at the position of the captured image corresponding to the position where the target was detected, and outputs the composite image to the display 2 (step S107). Then, the process returns to step S105, and the processes of steps S105 to S107 are repeated.

As described above, the perimeter monitoring device 1 of this embodiment includes a nautical chart data storage unit 33, a captured image input unit 21, an image recognition area setting unit 31, an additional display information acquisition unit 17, and a composite image generation unit 20. The nautical chart data storage unit 33 stores nautical chart data. The captured image input unit 21 inputs the captured image of the camera 3. The image recognition area setting unit 31 sets the detection target area based on the nautical chart data. The additional display information acquisition unit 17 acquires target information of targets detected in the detection target area. The composite image generation unit 20 generates a composite image by combining the target information at a position in the captured image corresponding to the position where the target was detected.

This makes it easy to set the area for image recognition using land-water boundaries in nautical chart data.

The above describes a preferred embodiment of the present invention, but the above configuration can be modified, for example, as follows:

The pan/tilt function described above may be omitted from camera 3, making it impossible to change the shooting direction.

When the three-dimensional scene generating unit 22 generates the three-dimensional scene data 48, in the above embodiment, as described in FIG. 4, the virtual reality objects 41v, 42v, ... are positioned based on a camera orientation reference with the position of the camera 3 as the origin. However, the virtual reality objects 41v, 42v, ... may be positioned based on a true north reference in which the +z direction is always true north, rather than based on the camera orientation. In this case, when the azimuth angle changes due to the panning operation of the camera 3, instead of repositioning the virtual reality objects 41v, 42v, ..., the azimuth angle of the viewpoint camera 55 is changed to simulate the change in the position and orientation of the camera 3 in the three-dimensional virtual space 40, and rendering is performed, thereby obtaining exactly the same rendering result as in the case of the camera orientation reference described above.

In addition, instead of using the position directly below the camera 3 as the origin, the coordinate system of the three-dimensional virtual space 40 may be set to a fixed point appropriately determined on the Earth, with the +z direction being due north and the +x direction being due east, for example.

The devices (sources of additional display information) connected to the perimeter monitoring device 1 are not limited to those described in FIG. 1, and may include other devices. Examples of such devices include infrared cameras and acoustic sensors.

The target area set in the image recognition area setting unit 31 and the target area set in the tracking target area setting unit 32 can be represented by other figures, such as smooth curves, instead of polygonal line figures. Also, the target area can be set in the form of raster data (mask image) instead of vector data.

The figures representing the additional display information are not limited to those shown in FIG. 5. For example, a figure that is a rendering of a three-dimensional model of the ship can be displayed at the position where the ships 41r, 42r, and 43r are detected. This allows a more realistic display to be realized. In the three-dimensional virtual space 40 of FIG. 4, the three-dimensional model of the ship is placed in an orientation that matches the orientation of the ship obtained from AIS information, etc. The size of the three-dimensional model of the ship placed in the three-dimensional virtual space 40 may be changed depending on the size of the ship obtained from AIS information, etc.

The three-dimensional scene generating unit 22 can be omitted, and the figures 41f, 42f, and 43f of the additional display information can be configured not to be displayed by augmented reality. For example, instead of a figure that is a rendered three-dimensional cone shape, a planar triangle that is simply pointing downward can be displayed. Even without constructing a three-dimensional scene, the correspondence between each pixel in the captured image and the corresponding location on the water surface can be calculated based on the position and orientation in which the camera 3 is installed. Therefore, the above triangle can be displayed at a position on the image that corresponds to the detected position on the water surface. Similarly, even without constructing a three-dimensional scene, a composite image such as that shown in FIG. 7 in which a boundary line figure is composited can be realized.

The perimeter monitoring device 1 is not limited to being installed in ground facilities, but may also be installed on a moving object such as a ship.

1 Surroundings monitoring device 3 Camera 17 Additional display information acquisition unit (target information acquisition unit)
20 Composite image generating unit 21 Shooting image input unit 28 Image recognition unit 31 Image recognition area setting unit (area setting unit)
32 Tracking target area setting unit (area setting unit)
33 Nautical chart data storage unit

Claims (16)

a nautical chart data storage unit for storing nautical chart data;
a captured image input unit for inputting captured images from a camera;
an area setting unit that sets a boundary that separates an inside and an outside of a detection target area using a polygonal line figure based on the nautical chart data;
a target information acquisition unit that acquires target information of a target detected by image recognition inside the detection target area in the captured image ;
a composite image generating unit that generates a composite image by combining the target information at a position of the photographed image corresponding to the position where the target is detected;
Equipped with
A perimeter monitoring device characterized by: The surroundings monitoring device according to claim 1,
The boundary separating the inside and outside of the detection target area in which the image recognition is performed is represented by a closed polygonal line figure.
A perimeter monitoring device characterized by: The surroundings monitoring device according to claim 1 or 2,
the region setting unit stores position information of a plurality of vertices of the polygonal line figure;
A perimeter monitoring device characterized by: The surroundings monitoring device according to any one of claims 1 to 3,
the region setting unit sets an inside of the detection target region so as to include at least a part of a region where a water surface appears and exclude a land portion;
A perimeter monitoring device characterized by: The surroundings monitoring device according to any one of claims 1 to 4,
the synthetic image generating unit generates the synthetic image including a graphic representing the detection target region.
Perimeter monitoring devices. The surroundings monitoring device according to claim 5,
The detection target area in which the image recognition is performed can be designated after being transformed by a user operation.
Perimeter monitoring devices. The surroundings monitoring device according to any one of claims 1 to 6,
The detection target area is an area separated by a boundary line represented by a plurality of pieces of position information.
A perimeter monitoring device characterized by: The surroundings monitoring device according to claim 7,
The detection target area is an area delimited by a water-land boundary or a bathymetry line included in the nautical chart data.
A perimeter monitoring device characterized by: The surroundings monitoring device according to claim 7,
The detection target area is an area separated by a boundary line a predetermined distance away from a land-land boundary line included in the nautical chart data toward the water area side.
A perimeter monitoring device characterized by: The surroundings monitoring device according to any one of claims 1 to 9,
The photographed image input unit inputs photographed images captured by a camera installed on the ground.
A perimeter monitoring device characterized by: The surroundings monitoring device according to any one of claims 1 to 10,
The target information acquisition unit acquires target information of a target detected by the radar device in the detection target area.
A perimeter monitoring device characterized by: The surroundings monitoring device according to any one of claims 1 to 11,
The target information represents at least one of a target position, a target speed, and a target size.
A perimeter monitoring device characterized by: The surroundings monitoring device according to any one of claims 1 to 12,
The target information acquisition unit is capable of acquiring the target information based on AIS information,
The synthetic image generation unit is capable of synthesizing the target information based on the AIS information and the target information based on information other than the AIS information into a single image.
A perimeter monitoring device characterized by: The surroundings monitoring device according to claim 13,
When both the target information based on the AIS information and the target information based on information other than the AIS information are obtained for the same target, the synthetic image generation unit preferentially synthesizes the target information based on the AIS information.
A perimeter monitoring device characterized by: The surroundings monitoring device according to any one of claims 1 to 14,
The composite image generating unit is capable of combining a direction scale indicating a direction with the captured image,
The composite image generating unit is capable of automatically changing a vertical position at which the orientation scale is composited with the captured image.
A perimeter monitoring device characterized by: Stores chart data,
Input the images captured by the imaging device,
Based on the nautical chart data, a boundary is set to separate an inside and an outside of a detection target area using a polygonal line diagram;
acquiring target information of a target detected by image recognition within the detection target area in the captured image ;
generating a composite image by combining the target information with the captured image at a position corresponding to the position where the target was detected;
A perimeter monitoring method comprising:

Images