JP6554825B2 - Image processing apparatus, image processing system, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing system, an image processing method, and a program.

  2. Description of the Related Art In recent years, image monitoring systems have been developed that analyze video data obtained by recording with an imaging device such as a video camera to detect an object, or detect a motion or abnormality of an object on a video, or video. It is remarkable. As such an image monitoring system, an image of an inspection object is recorded by an image, and a signal generated on the inspection apparatus side (for example, a detection signal of a proximity sensor or a photoelectric sensor) is used as a trigger. A system that performs image determination based on a frame of data has been proposed (see Patent Document 1). In the technique described in Patent Document 1, when it is necessary to perform image determination, an inspection object is reflected in a normal position for image determination from a past frame among frames constituting recorded video data. The current frame is searched, and image determination is performed on the searched frame.

  However, since the technique described in Patent Document 1 uses a signal generated on the inspection apparatus side as a trigger for image determination, there is a problem that the timing of the trigger is fixed and the timing adjustment is not flexible. .

  The present invention has been made in view of the above, and provides an image processing apparatus, an image processing system, an image processing method, and a program capable of freely generating a trigger timing that is a reference for timing of image determination and the like. The purpose is to do.

In order to solve the above-described problems and achieve the object, the present invention provides a display unit, a playback unit that displays video data captured by the imaging unit on the display unit, and the video data displayed on the display unit. In the first display area of the display means, the trigger area designating means for designating a trigger area, and the trigger from the video data for a first predetermined time based on the time point when the trigger area is designated by the trigger area designating means. Detection means for detecting a change point in an image in the area and extracting an image in the trigger area of the frame of the change point as a first image; and the display in which video data imaged by the imaging means is displayed an image in the region corresponding to the trigger region in the second display area of the device compared to the first image, and generating means for generating a trigger according to the comparison result Characterized by comprising a.

  According to the present invention, it is possible to freely generate a trigger timing that is a reference for timing such as image determination.

FIG. 1 is a diagram illustrating an example of the overall configuration of the image processing system according to the first embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of the information processing apparatus according to the first embodiment. FIG. 3 is a diagram illustrating an example of a functional block configuration of the information processing apparatus according to the first embodiment. FIG. 4 is a diagram illustrating an example in which a plurality of real-time videos are displayed on the main screen displayed on the display device of the information processing apparatus according to the first embodiment. FIG. 5 is a diagram illustrating an example of a state in which video data is displayed on the setting screen displayed on the display device of the information processing apparatus according to the first embodiment. FIG. 6 is a diagram illustrating an example in which a trigger area is specified in the video display unit on the setting screen displayed on the display device of the information processing apparatus according to the first embodiment. FIG. 7 is a diagram illustrating an example of mark display when a change point in the trigger region is found on the setting screen displayed on the display device of the information processing device according to the first embodiment. FIG. 8 is a diagram for explaining an operation of detecting a change point in the trigger area according to the first embodiment. FIG. 9 is a diagram illustrating an example of a dialog displayed when two change points in the trigger area are found on the setting screen displayed on the display device of the information processing apparatus according to the first embodiment. FIG. 10 is a diagram illustrating an example of a dialog displayed when a change point in the trigger area is not found on the setting screen displayed on the display device of the information processing apparatus according to the first embodiment. FIG. 11 is a diagram illustrating an example in which a detection area is designated in the video display unit on the setting screen displayed on the display device of the information processing device according to the first embodiment. FIG. 12 is a diagram illustrating an example in which the second detection area is designated in the video display unit on the setting screen displayed on the display device of the information processing device according to the first embodiment. FIG. 13 illustrates a state in which a trigger area set with video data different from the video data displayed on the video display unit is selected on the setting screen displayed on the display device of the information processing apparatus according to the first embodiment. It is a figure which shows the example of. FIG. 14 shows a state in which a trigger area set with video data different from the video data displayed on the video display unit is selected on the setting screen displayed on the display device of the information processing apparatus according to the first embodiment. It is a figure which shows the example of the state of. FIG. 15 illustrates detection corresponding to a trigger region set with video data different from the video data displayed on the video display unit on the setting screen displayed on the display device of the information processing apparatus according to the first embodiment. It is a figure which shows the example which designates an area | region. FIG. 16 is a diagram illustrating an example of setting the detection area attribute information designated by the video display unit on the setting screen displayed on the display device of the information processing apparatus according to the first embodiment. FIG. 17 is a diagram illustrating an example of a state in the case where the size of the detection area already designated in the video display unit is changed on the setting screen displayed on the display device of the information processing device according to the first embodiment. . FIG. 18 is a diagram illustrating an example of a state in the case where the size of the trigger region designated in the video display unit is changed on the setting screen displayed on the display device of the information processing device according to the first embodiment. . FIG. 19 is a diagram illustrating an example of a state in which video data is displayed on the sensitivity setting preview screen displayed on the display device of the information processing device according to the first embodiment. FIG. 20 is a diagram illustrating an example of a determination result of the difference between the extracted images with respect to the detection reference image on the sensitivity setting preview screen displayed on the display device of the information processing apparatus according to the first embodiment. FIG. 21 is a diagram illustrating an example of a result of re-determination of the difference between the extracted images with respect to the detection reference image at different sensitivities on the sensitivity setting preview screen displayed on the display device of the information processing device according to the first embodiment. . FIG. 22 is a diagram illustrating an example in which a dialog for confirming whether to apply the changed sensitivity on the sensitivity setting preview screen displayed on the display device of the information processing apparatus according to the first embodiment is displayed. . FIG. 23 is a diagram illustrating an example of a state in which the sensitivity changed on the sensitivity setting preview screen is reflected on the setting screen displayed on the display device of the information processing apparatus according to the first embodiment. FIG. 24 is a flowchart illustrating an example of a setting operation for designating a trigger area and a detection area on a setting screen displayed on the display device of the information processing apparatus according to the first embodiment. FIG. 25 is a diagram illustrating an example of a state after the trigger region and the detection region are designated on the main screen displayed on the display device of the information processing device according to the first embodiment. FIG. 26 is a flowchart illustrating an example of an image determination operation performed by the information processing apparatus according to the first embodiment. FIG. 27 shows the display state of the trigger area when the trigger of the trigger area occurs on the main screen displayed on the display device of the information processing apparatus according to the first embodiment, and the detection at the detection timing of the detection area. It is a figure which shows the example of the display state of an area | region. FIG. 28 is a diagram illustrating an example of a dialog displayed when a reference image in the trigger area is not found on the setting screen displayed on the display device of the information processing device according to the second embodiment. FIG. 29 is a diagram illustrating an operation of detecting a change point in the trigger area according to the second embodiment. FIG. 30 is a flowchart illustrating an example of a setting operation for designating a trigger area and a detection area on a setting screen displayed on the display device of the information processing apparatus according to the second embodiment.

  Hereinafter, embodiments of an image processing apparatus, an image processing system, an image processing method, and a program according to the present invention will be described in detail with reference to the drawings. In addition, the present invention is not limited by the following embodiments, and constituent elements in the following embodiments are easily conceivable by those skilled in the art, substantially the same, and so-called equivalent ranges. Is included. Furthermore, various omissions, substitutions, changes, and combinations of the constituent elements can be made without departing from the scope of the following embodiments.

[First Embodiment]
(Overall configuration of image processing system)
FIG. 1 is a diagram illustrating an example of the overall configuration of the image processing system according to the first embodiment. The overall configuration of the image processing system 1 of the present embodiment will be described with reference to FIG.

  As illustrated in FIG. 1, the image processing system 1 according to the present embodiment includes imaging devices 2 a to 2 f, an information processing device 3, and a network 4.

  The imaging devices 2a to 2f capture (capture) a subject by converting light from the subject into an electrical signal, and video data that is a moving image (for example, 10 [FPS]) composed of a plurality of frames (image data). Is a video camera that generates For example, the imaging devices 2a to 2f image production facilities and the like that produce products, and generate video data for detecting an abnormality with respect to a workpiece that is a product.

  Note that the imaging devices 2a to 2f are simply referred to as “imaging device 2” when referred to without distinction or collectively. In FIG. 1, the image processing system 1 is configured to include six imaging devices 2, but is not limited thereto, and may be configured to include other numbers of imaging devices 2. Good.

  The information processing device 3 is a PC (Personal Computer), a workstation, or the like that functions as an image processing device that performs image determination based on video data captured by the imaging device 2. Further, the information processing apparatus 3 is connected to an external device 10 such as a production facility so that communication according to, for example, a fieldbus standard is possible. Image determination processing based on video data by the information processing device 3 will be described later.

  The network 4 is, for example, an Ethernet (registered trademark) standard network for connecting the imaging devices 2 a to 2 f and the information processing device 3. In this case, in the network 4, data communication is performed using a protocol such as TCP (Transmission Control Protocol) / IP (Internet Protocol). In this case, the imaging devices 2a to 2f and the information processing device 3 have MAC addresses for communication using the TCP / IP protocol, and are assigned IP addresses such as private IP addresses. The specific configuration of the network 4 is, for example, a star wiring format in which the imaging devices 2a to 2f and the information processing device 3 are connected to a switching hub having a plurality of ports by a LAN (Local Area Network) cable.

  The network 4 shown in FIG. 1 shows an example in which communication is performed by TCP / IP. However, the present invention is not limited to this. For example, a VGA (Video Graphics Array) terminal or a USB (USB) is connected to the information processing apparatus 3 side. There may be a configuration in which a plurality of (Universal Serial Bus) ports are provided and the plurality of imaging devices 2 are connected to the information processing device 3 by VGA cables or USB cables.

(Hardware configuration of information processing device)
FIG. 2 is a diagram illustrating an example of a hardware configuration of the information processing apparatus according to the first embodiment. A hardware configuration of the information processing apparatus 3 according to the present embodiment will be described with reference to FIG.

  As shown in FIG. 2, the information processing apparatus 3 includes a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, an external storage device 104, a display 105, A network I / F 106, a keyboard 107, a mouse 108, a DVD (Digital Versatile Disc) drive 109, and an external device I / F 111 are provided.

  The CPU 101 is a device that controls the overall operation of the information processing apparatus 3. The ROM 102 is a nonvolatile storage device that stores a program for the information processing device 3. The RAM 103 is a volatile storage device used as a work area for the CPU 101.

  The external storage device 104 is a storage device such as an HDD (Hard Disk Drive) or an SSD (State Solid Drive) that stores video data captured by the imaging device 2 and various data such as setting information.

  The display 105 is a display device that displays various information such as a cursor, a menu, a window, characters or images, or a screen of an application that executes image determination processing by the information processing device 3. The display 105 is, for example, a CRT (Cathode Ray Tube) display, a liquid crystal display, a plasma display, an organic EL (Electroluminescence) display, or the like. For example, the display 105 is connected to the main body of the information processing apparatus 3 by a VGA cable, an HDMI (registered trademark) (High-Definition Multimedia Interface) cable, or the like, or by an Ethernet cable.

  The network I / F 106 is an interface for connecting to the network 4 and performing data communication. The network I / F 106 is, for example, a NIC (Network Interface Card) that enables communication using a TCP / IP protocol. Specifically, the information processing device 3 acquires video data from the imaging device 2 via the network 4 and the network I / F 106.

  The keyboard 107 is an input device for selecting characters, numbers, various instructions, moving a cursor, setting setting information, and the like. The mouse 108 is an input device for selecting and executing various instructions, selecting a processing target, moving a cursor, setting setting information, and the like.

  The DVD drive 109 is a device that controls reading, writing, and deletion of data with respect to the DVD 110 as an example of a removable storage medium.

  The external device I / F 111 is an interface for connecting to the external device 10 and performing data communication. The external device I / F 111 is an interface card that enables communication according to the fieldbus standard, for example. Specifically, the information processing apparatus 3 performs data communication with the external device 10 via the external device I / F 111.

  The CPU 101, the ROM 102, the RAM 103, the external storage device 104, the display 105, the network I / F 106, the keyboard 107, the mouse 108, the DVD drive 109, and the external device I / F 111 communicate with each other via a bus 112 such as an address bus and a data bus. Connected as possible. When the display 105 is connected by an Ethernet cable, it is connected to the network I / F 106. In this case, data communication is performed using a protocol such as TCP / IP.

(Block configuration of information processing device)
FIG. 3 is a diagram illustrating an example of a functional block configuration of the information processing apparatus according to the first embodiment. The block configuration of the information processing apparatus 3 will be described with reference to FIG.

  As shown in FIG. 3, the information processing apparatus 3 includes a video reception unit 301, a storage unit 302, an input device 303, a first reproduction unit 304 (reproduction unit), and a trigger area designation unit 305 (trigger area designation unit). ), A change point detection unit 306 (detection unit), a detection region designation unit 307 (detection region designation unit), a setting unit 308 (setting unit), a second reproduction unit 309, an image extraction unit 310, and a difference Determination unit 311, video distribution unit 312, trigger generation unit 313 (generation unit), detection region determination unit 314 (detection region determination unit), display control unit 315, and external output unit 316 (external output unit) And a display device 317 (display means).

  The video reception unit 301 is a functional unit that performs data communication with the imaging device 2 via the network 4 and receives video data from the imaging device 2. The video receiving unit 301 stores the received video data in the storage unit 302. The video receiving unit 301 is realized by the network I / F 106 shown in FIG. In FIG. 3, the network 4 is not shown for the sake of simplicity.

  The storage unit 302 is a storage device that stores the video data received by the video receiving unit 301 and various setting information. The storage unit 302 identifies and stores which imaging device 2 the video data received by the video reception unit 301 is. The storage unit 302 is realized by, for example, the RAM 103 or the external storage device 104 illustrated in FIG.

  The input device 303 is a device that performs operation input for image determination processing executed by the information processing device 3. The input device 303 is realized by the keyboard 107 and the mouse 108 shown in FIG.

  The first playback unit 304 sends the video data acquired from the storage unit 302 to the display control unit 315 in accordance with an operation signal from the input device 303 operated by the user, and displays the video data to the display control unit 315. This is a functional unit that plays back and displays. Specifically, the first reproduction unit 304 reproduces and displays video data on the video display unit 502 of the setting screen 500 displayed on the display device 317 described later with reference to FIG. The first reproduction unit 304 is realized by a program executed by the CPU 101 shown in FIG.

  The trigger area specifying unit 305 is a trigger that determines the trigger timing in the display area of the video data on the display device 317 (the video display unit 502 described later in FIG. 5 and the like) in accordance with an operation signal from the input device 303 operated by the user. It is a functional part that specifies an area. The trigger area designation unit 305 causes the storage unit 302 to store information on the designated trigger area. The trigger area information is information such as coordinates indicating the position of the trigger area in the display area of the video data and the shape of the trigger area, for example. The trigger area specifying unit 305 is realized by a program executed by the CPU 101 shown in FIG.

  When the trigger region is specified by the trigger region specifying unit 305, the change point detecting unit 306 is for a predetermined time before and after the time of the frame of the video data reproduced and displayed by the first reproducing unit 304 at the specified time. A frame is acquired, and in the acquired frame, a difference (first difference) with respect to a pixel value from the immediately preceding frame is calculated for each frame, and a change point for an image in the trigger region is detected based on the calculated difference. It is a functional part to do. The change point will be described later. As a difference calculation method, for example, pixel values in the trigger areas of two frames are compared, and the total number of pixels having pixel values different from each other by a predetermined sensitivity or more is calculated as a difference. Also, if the frame is a grayscale image, the pixel value matches the luminance value, so the luminance values in the trigger area of the two frames are compared, and the difference in the total number of pixels whose luminance values differ by more than a predetermined sensitivity May be calculated as When the frame is an image composed of RGB pixel values, the pixel values of R (red), G (green), and B (blue) are compared, and the pixel value is the same as described above. The sum of the number of pixels that differ by more than a predetermined sensitivity may be calculated as a difference. Alternatively, edge detection processing may be performed on the frame image, and the sum of the edge portions may be calculated as a difference. The change point detection unit 306 is realized by a program executed by the CPU 101 shown in FIG.

  The detection area designating unit 307 is a detection area that is a target of image determination processing that detects an abnormality in the display area of the video data (video display section 502) on the display device 317 in accordance with an operation signal from the input device 303 operated by the user. Is a functional part for designating. The detection area designation unit 307 causes the storage unit 302 to store information on the designated detection area. The information on the detection area is information such as coordinates indicating the position of the detection area in the display area of the video data and the shape of the trigger area, for example. The detection area designating unit 307 can designate a plurality of detection areas in the video data display area, as will be described later. The detection area designating unit 307 is realized by a program executed by the CPU 101 shown in FIG.

  The setting unit 308 is a functional unit that sets setting information and stores the setting information in the storage unit 302 in accordance with an operation signal from the input device 303 operated by the user. In addition, the setting unit 308 sends information that needs to be displayed among the set setting information from the storage unit 302 to the display control unit 315, and causes the display control unit 315 to display the setting information on the display device 317. The setting unit 308 is realized by a program executed by the CPU 101 shown in FIG.

  The second playback unit 309 sends the video data acquired from the storage unit 302 to the display control unit 315 in accordance with an operation signal from the input device 303 operated by the user, and displays the video data to the display control unit 315. This is a functional unit that plays back and displays. Specifically, the second reproduction unit 309 reproduces and displays video data on the video display unit 702 of the sensitivity setting preview screen 700 displayed on the display device 317 described later with reference to FIG. The second playback unit 309 is realized by a program executed by the CPU 101 shown in FIG.

  The image extraction unit 310 includes an image (extraction image) of a portion corresponding to the detection area specified by the detection area specification unit 307 in the display area (video display unit 702) where the video data is reproduced and displayed by the second reproduction unit 309. ). The image extraction unit 310 is realized by a program executed by the CPU 101 shown in FIG.

  The difference determination unit 311 is a functional unit that calculates a difference between a detection reference image that is an image in a detection region at a time at which a detection region is designated and an extracted image extracted by the image extraction unit 310. The calculation method of the difference is in accordance with the calculation method of the change point detection unit 306 described above. The difference determination unit 311 is realized by a program executed by the CPU 101 shown in FIG.

  The video distribution unit 312 sends real-time video data acquired from the video receiving unit 301 to the display control unit 315 according to an operation signal from the input device 303 operated by the user, and sends the video data to the display control unit 315. This is a functional unit to be displayed on the display device 317. Specifically, the video distribution unit 312 distributes and displays the video data on the video display units 401a to 401f of the main screen 400 displayed on the display device 317 described later with reference to FIGS. The video distribution unit 312 is realized by a program executed by the CPU 101 shown in FIG.

  The trigger generation unit 313 compares the difference image with the image of the trigger region of the video data frame in the trigger region in the video display units 401a to 401f of the main screen 400 shown in FIG. It is a functional unit that generates a signal. The trigger generation unit 313 is realized by a program executed by the CPU 101 shown in FIG.

  The detection area determination unit 314 detects the detection reference image (second image) in the detection areas in the video display units 401a to 401f before or after the set time when the trigger signal generated by the trigger generation unit 313 is received. ) And the image in the detection area of the frame of the video data to determine whether or not there is an abnormality. The detection area determination unit 314 is realized by a program executed by the CPU 101 illustrated in FIG.

  The display control unit 315 is a functional unit that controls the display operation of the display device 317. Specifically, the display control unit 315 displays the video data acquired by the first playback unit 304 and the second playback unit 309, the setting information set by the setting unit 308, and the trigger area specified by the trigger area specifying unit 305. And the detection area designated by the detection area designation unit 307 and the like are displayed on the video distribution unit 312. The display control unit 315 is realized by a program (driver) executed by the CPU 101 shown in FIG.

  The external output unit 316 is a functional unit that outputs an abnormal signal to the external device 10 when the detection region determination unit 314 determines abnormality. The external output unit 316 is realized by the external device I / F 111 shown in FIG. The external output unit 316 outputs an abnormality signal. However, the present invention is not limited to this, and a batch file for notifying abnormality by e-mail is transmitted to the external device 10 or the main screen 400. It is also possible to take measures such as notifying abnormality (for example, blinking with a lamp icon).

  The display device 317 is a device that displays various data according to the control of the display control unit 315. In the present embodiment, in particular, the display device 317 displays a main screen 400, a setting screen 500, a sensitivity setting preview screen 700, and the like, which will be described later, by a program (application) executed by the CPU 101 shown in FIG. The display device 317 is realized by the display 105 illustrated in FIG.

  In addition, the first reproduction unit 304, the trigger region designation unit 305, the change point detection unit 306, the detection region designation unit 307, the setting unit 308, the second reproduction unit 309, the image extraction unit 310, and the difference determination unit 311 illustrated in FIG. The video distribution unit 312, the trigger generation unit 313, the detection region determination unit 314, and the display control unit 315 are not all software programs, but are ASIC (Application Specific Integrated Circuit) or FPGA (Field-Programmable Gate). Array) or the like may be realized by a hardware circuit.

  In addition, the first reproduction unit 304, the trigger region designation unit 305, the change point detection unit 306, the detection region designation unit 307, the setting unit 308, the second reproduction unit 309, the image extraction unit 310, the difference determination unit 311, and the video distribution unit 312. The trigger generation unit 313, the detection area determination unit 314, and the display control unit 315 conceptually show functions, and are not limited to such a configuration.

  In addition, for the sake of simplification of the following description, an expression that the display control unit 315 displays the received data on the display device 317 is simply a function unit that has sent the data to the display control unit 315. It is expressed that it is displayed on (or a screen displayed by the display device 317). For example, when the first reproduction unit 304 sends the video data to the display control unit 315 and causes the display control unit 315 to reproduce and display the video data on the display device 317, the first reproduction unit 304 simply displays the video data. It is expressed that playback is displayed on the display device 317.

(Main screen structure)
FIG. 4 is a diagram illustrating an example in which a plurality of real-time videos are displayed on the main screen displayed on the display device of the information processing apparatus according to the first embodiment. The configuration of the main screen 400 and the operation for displaying a plurality of real-time videos will be described with reference to FIG.

  A main screen 400 shown in FIG. 4 is displayed on the display device 317 by a program (application) executed by the CPU 101. The main screen 400 displays video data received from a plurality of imaging devices 2 in real time, and a trigger signal is generated at the trigger timing by the trigger generation unit 313 in the trigger region specified by the trigger region specifying unit 305. It is. The main screen 400 is a screen for determining whether there is an abnormality in the detection region designated by the detection region designation unit 307 after the set time or the set time when the trigger signal is received by the detection region judgment unit 314. As shown in FIG. 4, the main screen 400 includes video display units 401 a to 401 f, a monitoring start button 411, a monitoring stop button 412, and a monitoring setting button 413.

  The video display units 401a to 401f are real-time video data acquired from the imaging devices 2a to 2f via the video reception unit 301 in accordance with an operation signal from the input device 303 operated by the user by the video distribution unit 312. Is a display unit on which each is displayed. Note that the video display units 401a to 401f are simply referred to as the “video display unit 401” when referred to without distinction or collectively.

  When the monitoring start button 411 is pressed by a user's operation of the input device 303 (click operation using the mouse 108, etc., the same applies hereinafter), the video distribution unit 312 causes real-time video data to be displayed on the video display units 401a to 401f, respectively. It is a button that is displayed. When the monitor start button 411 is pressed and video data is displayed on the video display unit 401, a trigger signal is generated at the trigger timing by the trigger generation unit 313 in the trigger region specified by the trigger region specifying unit 305. Then, determination of the presence or absence of abnormality in the detection region is started after the set time when the trigger signal is received by the detection region determination unit 314 or after the set time.

  When the monitoring stop button 412 is pressed by an operation of the input device 303 by the user, the trigger generation unit 313 generates a trigger signal at the trigger timing, and the detection region determination unit 314 determines whether there is an abnormality in the detection region. And a button for stopping the real-time video data display operation by the video distribution unit 312.

  A monitoring setting button 413 is a button for causing the setting unit 308 to display a setting screen 500 shown in FIG. 5 and the like on the display device 317 when the user presses the input device 303.

(Configuration of configuration screen)
FIG. 5 is a diagram illustrating an example of a state in which video data is displayed on the setting screen displayed on the display device of the information processing apparatus according to the first embodiment. The configuration of the setting screen 500 will be described with reference to FIG.

  The setting screen 500 is a screen for setting the trigger region and the detection region, and setting the sensitivity, threshold value, and timing of image determination in the detection region. As shown in FIG. 5, the setting screen 500 includes a camera selection unit 501, a video display unit 502, a playback button 503a, a pause button 503b, a fast forward button 503c, a fast reverse button 503d, and a stop button 503e. A seek bar 504, a slider 505, a time display unit 506, a trigger region information unit 507, a reference image display unit 508a, a difference image display unit 508b, a detection region information unit 509, a detection region button unit 510, A setting reading button 511, a setting writing button 512, a setting reflection button 513, and a close button 514 are included.

  The camera selection unit 501 is an operation unit that selects which video data is to be displayed on the video display unit 502 among the video data of the imaging devices 2a to 2f that the user desires to specify the trigger region and the detection region. For example, the video data captured by the imaging devices 2a to 2f are associated with “1” to “6”, respectively. In the example of FIG. 5, “6”, that is, the video data of the imaging device 2f is selected. An example is shown.

  The video display unit 502 is a display unit that reproduces and displays video data acquired from the storage unit 302 according to an operation signal from the input device 303 operated by the user by the first playback unit 304. In the example of FIG. 5, when the user selects the “6” camera selection unit 501 by operating the input device 303, the first playback unit 304 displays the video data captured by the imaging device 2 f as the display device 317. To display.

  The playback button 503a is a button for starting playback of the video data displayed on the video display unit 502 by the first playback unit 304 when pressed by an operation of the input device 303 by the user. The pause button 503b is a button for temporarily stopping the video data reproduced and displayed on the video display unit 502 by the first reproduction unit 304 when pressed by an operation of the input device 303 by the user. The fast-forward button 503c is a button for fast-forwarding and displaying the video data reproduced and displayed on the video display unit 502 by the first reproduction unit 304 when pressed by the user operating the input device 303. The rewind button 503d is a button for fast rewinding display of video data reproduced and displayed on the video display unit 502 by the first reproduction unit 304 when pressed by the user operating the input device 303. The stop button 503e is a button for stopping the video data reproduced and displayed on the video display unit 502 by the first reproduction unit 304 when pressed by the operation of the input device 303 by the user.

  The seek bar 504 is a bar-shaped body that indicates how long the video data reproduced and displayed on the video display unit 502 is based on the position of the slider 505 disposed thereon. The slider 505 is a shape body and operation unit that slides to a position on the seek bar 504 corresponding to the imaging time of the video data reproduced and displayed on the video display unit 502 (time when the displayed frame is captured). Conversely, when the slider 505 is slid by the operation of the input device 303 by the user, the first playback unit 304 displays a frame of video data at the imaging time corresponding to the position on the seek bar 504 where the slider 505 exists. Displayed on the part 502.

  The time display unit 506 is a display unit that displays the imaging date and imaging time of the video data reproduced and displayed on the video display unit 502 by the first reproduction unit 304.

  The trigger region information unit 507 is a display unit that displays the name of the trigger region specified by the trigger region specifying unit 305 and the sensitivity and threshold value for the change point detection unit 306 to detect the above-described change point. The reference image display unit 508 a is a display unit that displays the reference image extracted by the change point detection unit 306. The difference image display unit 508 b is a display unit that displays a difference image (described later) extracted by the change point detection unit 306.

  The detection area information unit 509 includes the name, sensitivity, threshold value, detection delay time (indicated as “delay time” in FIG. 5) of the detection area designated by the detection area designating unit 307, and a monitoring method (“monitoring” in FIG. 5). (Notation), presence / absence of action (indicated as “action” in FIG. 5), and image determination method (indicated as “image determination” in FIG. 5) (hereinafter, these pieces of information about the detection area are referred to as attribute information). And a selection operation unit.

  The detection area button unit 510 is a group of buttons for editing attribute information about the detection area specified by the detection area specifying unit 307 and displayed in the detection area information unit 509. The detection area button unit 510 includes an action button 510a, an image determination selection button 510b, a deletion button 510c, a sensitivity determination button 510d, a sensitivity increase button 510e, a sensitivity decrease button 510f, a threshold increase button 510g, and a threshold decrease. Button 510h. The function of each button of the detection area button unit 510 will be described later.

  When the setting read button 511 is pressed by the user's operation of the input device 303, the setting unit 308 reads setting information such as the trigger area and the detection area information stored in the storage unit 302, and the setting screen 500 The setting information is displayed on each display unit. When the setting write button 512 is pressed by an operation of the input device 303 by the user, setting information such as trigger area and detection area information set on the setting screen 500 by the setting unit 308 is stored in the storage unit 302 ( These buttons are stored in the external storage device 104). When the setting reflection button 513 is pressed by an operation of the input device 303 by the user, setting information such as trigger area and detection area information set on the setting screen 500 by the setting unit 308 is displayed on the main screen 400 and setting. The button is valid until an application such as the screen 500 is closed (for example, temporarily stored in the RAM 103).

  The close button 514 is a button for causing the setting unit 500 to close the setting screen 500 and causing the display device 317 to display the main screen 400 when pressed by the user operating the input device 303.

(Specify trigger area and detect change point)
FIG. 6 is a diagram illustrating an example in which a trigger area is specified in the video display unit on the setting screen displayed on the display device of the information processing apparatus according to the first embodiment. FIG. 7 is a diagram illustrating an example of mark display when a change point in the trigger region is found on the setting screen displayed on the display device of the information processing device according to the first embodiment. FIG. 8 is a diagram for explaining an operation of detecting a change point in the trigger area according to the first embodiment. FIG. 9 is a diagram illustrating an example of a dialog displayed when two change points in the trigger area are found on the setting screen displayed on the display device of the information processing apparatus according to the first embodiment. FIG. 10 is a diagram illustrating an example of a dialog displayed when a change point in the trigger area is not found on the setting screen displayed on the display device of the information processing apparatus according to the first embodiment. The trigger area designation and change point detection will be described with reference to FIGS.

  A setting screen 500 shown in FIG. 6 shows a state in which the trigger area 530 is designated in the video display unit 502 in which video data is played back and displayed by the first playback unit 304. Specifically, first, during playback and display of the video data on the video display unit 502, the user pauses by the operation of the input device 303 at the timing when the desired image portion that is the basis of the occurrence of the trigger is displayed. When the button 503b is pressed, the first reproduction unit 304 pauses the video data reproduced and displayed on the video display unit 502, and causes the time display unit 506 to display the imaging date and imaging time of the frame at that time. . Next, when the user operates the input device 303 (for example, dragging the mouse 108), the trigger area specifying unit 305 specifies and displays the trigger area 530 on the video display unit 502 (first display area).

  A setting screen 500 illustrated in FIG. 7 illustrates a state in which a change point is detected in the trigger region 530 specified by the trigger region specifying unit 305 by the change point detecting unit 306. Specifically, when the trigger region 530 is designated by the trigger region designation unit 305, the change point detection unit 306 first determines the frame of the video data being reproduced and displayed by the first reproduction unit 304 at the designated time point. Frames for a predetermined time (for example, 2 seconds before and after the imaging time) (first predetermined time) before and after the imaging time are acquired, and an image in the trigger area 530 at the time when the trigger area 530 is designated is extracted as a reference image. Here, the change point refers to a point in time when a predetermined change in the image in the trigger region 530 is taken as a reference for the timing at which the trigger generation unit 313 generates the trigger signal. Hereinafter, a specific method for detecting change points by the change point detection unit 306 will be described.

  As shown in FIG. 8A, the change point detection unit 306 acquires 13 frames (frame (0) to frame (12)) as frames for a predetermined time. In this case, the change point detection unit 306 acquires frames for a predetermined time before and after the frame (6) as the middle. In addition, in order to simplify the description, the image of the part corresponding to the trigger region 530 in each frame is also referred to as frame (0) to frame (12), respectively. Then, the change point detection unit 306 calculates a difference in pixel values from the immediately preceding frame for each acquired frame. The difference calculation method is as described above. The graph shown in FIG. 8B is a graph in which the change point detection unit 306 extracts the frames exemplified in FIG. 8A and expresses the differences calculated for each frame in time series. In FIG. 8B, for example, the difference between the frame (1) indicates the difference between the frame (1) and the frame (0) that is the immediately preceding frame. Here, the IN point and OUT point as change points are defined as follows. That is, the IN point is the time when the image in the trigger area 530 starts to change in successive frames. The OUT point is a point in time when a change in the image in the trigger area 530 starts to subside in successive frames. The change point detection unit 306 calculates the difference between the acquired frame and the immediately preceding frame for each frame. However, the change point detecting unit 306 is not limited to the immediately preceding frame. The difference from the previous frame may be calculated.

  The change point detection unit 306 finds an IN point candidate and an OUT point candidate based on the difference calculated for each frame. Specifically, when the calculated difference becomes equal to or greater than a predetermined threshold (first threshold) for two or more consecutive frames, the change point detection unit 306 first determines the frame whose difference is equal to or greater than the predetermined threshold. The frame is a candidate frame for the IN point. In the example of FIG. 8B, the difference of frame (1) is less than the threshold, the difference of frame (2) is greater than or equal to the threshold, and the difference of frame (3) is also greater than or equal to the threshold, and frames (2) and (3 ) Of two frames is equal to or greater than the threshold, so the frame (2) where the difference first exceeds the threshold is the IN point candidate frame. Similarly, the frame (7) is a candidate frame for the IN point. In addition, when the difference for only one frame is not less than the threshold and not more than two frames, there is a possibility of noise, so that frame is not a candidate frame for the IN point.

  On the other hand, when the calculated difference is continuously less than a predetermined threshold value (first threshold value) for two frames or more, the change point detection unit 306 first determines the frame whose difference is less than the predetermined threshold value as the OUT point. Candidate frames. In the example of FIG. 8B, the difference of the frame (4) is greater than or equal to the threshold, the difference of the frame (5) is less than the threshold, and the difference of the frame (6) is also less than the threshold, and the frames (5) and (6) Since the difference between the two frames is less than the threshold value, the frame (5) in which the difference is initially less than the threshold value is the OUT point candidate frame. Here, the difference of the frame (12) is less than the predetermined threshold value, but the subsequent frame (13) is a frame other than the frame extracted by the change point detection unit 306, and therefore the OUT point candidate Do not frame. Also, if the difference for only one frame is less than the threshold value and not continuous for two frames or more, there is a possibility of noise, so that frame is not a candidate frame for the OUT point.

  As described above, the change point detection unit 306 specifies the frames (2) and (7) as the IN point candidate frames and the frame (5) as the OUT point candidate frames in the example of FIG. To do. As described above, when a plurality of IN point candidate frames are identified, the change point detection unit 306 sets the frame closest to the middle frame (6) of the acquired frames, that is, the frame (7) as the IN point frame. decide. In the example of FIG. 8, only one frame (frame (5)) is specified as the OUT point candidate frame. However, as with the IN point candidate, a plurality of frames are OUT point candidate frames. It may be specified as. As described above, when a plurality of OUT point candidate frames are identified, the change point detection unit 306 determines the frame closest to the middle frame (6) of the acquired frames as the OUT point candidate, similarly to the IN point candidate. Decide on a frame. In the example of FIG. 8, since the OUT point candidate frame is only the frame (5), the change point detection unit 306 determines the frame (5) as the OUT point frame. As described above, the change point detection unit 306 detects the change points (IN point and OUT point) by determining the IN point frame and the OUT point frame from the acquired frames for a predetermined time. When the change point is detected, the change point detection unit 306 detects a moving object in the trigger region 530 using a frame before the change point frame, a frame after the change point frame, or a frame before and after the change point frame. Vector information indicating whether a moving object has entered from the storage unit 302 may be stored in the storage unit 302. In this case, the trigger generation unit 313 uses the vector information stored in the storage unit 302 to check whether the moving object has moved from the same direction as the vector information, and triggers based on the check result. A signal may be generated.

  Note that the method of detecting the change point in the trigger region 530 by the change point detection unit 306 is not limited to the above-described method, and how the change point is determined as long as the change point defines the timing at which the trigger is generated. It may be detected.

  In addition, as described above, the change point detection unit 306, when the difference is equal to or greater than the predetermined threshold continuously for two frames or more, first determines the frame whose difference is equal to or greater than the predetermined threshold as the IN point candidate. Although it is assumed to be a frame, it is not limited to this. That is, the difference may not be continuous for two frames or more, but may be a case where another number of frames or more are continuously equal to or greater than a predetermined threshold. The same applies to the case of finding a candidate frame for the OUT point.

  FIG. 8 shows an example in which both the IN point and OUT point frames are found. However, the change point detection unit 306 may find only the IN point frame or only the OUT point frame. In this case, these IN point or OUT point is detected as a change point. In this case, the change point detection unit 306 extracts an image in the trigger region 530 of the detected change point frame as a difference image (first image). The change point detection unit 306 stores the extracted reference image and the difference image in the frame of the change point in the storage unit 302, displays the reference image on the reference image display unit 508a, and displays the difference image on the difference image display unit 508b. Let Furthermore, the change point detection unit 306 displays a trigger mark 540 on the seek bar 504 at a portion corresponding to the time of the detected change point frame. When a change point is detected by the change point detection unit 306, the trigger area specifying unit 305 causes the storage unit 302 to store information on the specified trigger area 530. The setting unit 308 also names the trigger region 530 specified by the trigger region specifying unit 305 and a predetermined sensitivity (for example, “50”) used to detect the change point of the change point detection unit 306. A predetermined threshold value (for example, “20%”) (default value) is displayed on the trigger area information unit 507. As illustrated in FIG. 7, the setting unit 308 assumes that the trigger region is the trigger region designated for the video data of the imaging device “6” (imaging device 2f), as shown in FIG. ".

  The setting screen 500 shown in FIG. 9 is displayed when the change point detection unit 306 detects two change points (IN point and OUT point) in the trigger region 530 specified by the trigger region specification unit 305 as described above. The change point selection dialog 600 is displayed. As described above, when two or more change points are detected, the change point detection unit 306 generates one of the two change points (IN point and OUT point) closest to the time when the reference image is extracted for generating a trigger. The display device 317 displays a change point selection dialog 600 for selecting whether or not to use. The change point selection dialog 600 includes a yes button 601, a no button 602, and a close button 603. A close button 603 is a button for closing the change point selection dialog 600.

  When the user presses the yes button 601 by operating the input device 303, the change point detection unit 306 detects the IN point as a change point, and extracts an image in the trigger region 530 of the frame at the IN point as a difference image. . Then, the change point detection unit 306 stores the extracted reference image and the difference image in the frame of the IN point in the storage unit 302, displays the reference image on the reference image display unit 508a, and displays the difference image as the difference image display unit 508b. To display. Furthermore, the change point detection unit 306 displays a trigger mark 540 on the seek bar 504 at a portion corresponding to the detected time of the frame at the IN point.

  On the other hand, when the user presses the No button 602 by operating the input device 303, the change point detection unit 306 detects the OUT point as a change point, and uses the image in the trigger region 530 of the frame at the OUT point as a difference image. Extract. Then, the change point detection unit 306 stores the extracted reference image and the difference image in the frame of the OUT point in the storage unit 302, displays the reference image on the reference image display unit 508a, and displays the difference image as the difference image display unit 508b. To display. Furthermore, the change point detection unit 306 displays a trigger mark 540 on the seek bar 504 at a portion corresponding to the time of the frame of the detected OUT point. Other operations are the same as those described above with reference to FIG.

  In this way, the change point detection unit 306 detects the IN point and the OUT point as change points, and allows the user to select one of the two change points, so that the user can truly specify the change point. Can be selected.

  The setting screen 500 shown in FIG. 10 shows a state in which the no change point notification dialog 610 is displayed when no change point is found in the trigger area 530 specified by the trigger area specifying unit 305 by the change point detecting unit 306. Show. In the operation described with reference to FIG. 8, the change point detection unit 306 may not be able to detect a change point because the calculated difference does not exceed a predetermined threshold value. In this case, the change point detection unit 306 causes the display device 317 to display a no change point notification dialog 610 indicating that no change point has been found.

  The no change point notification dialog 610 includes an OK button 611 and a close button 612. The close button 612 is a button for closing the no change point notification dialog 610. When the user presses the OK button 611 by operating the input device 303, the change point detection unit 306 stores no information in the storage unit 302, and the image is also displayed in the reference image display unit 508 a and the difference image display unit 508 b. Is not displayed, and the no change point notification dialog 610 is closed.

(Specifying detection area and setting attribute information of detection area)
FIG. 11 is a diagram illustrating an example in which a detection area is designated in the video display unit on the setting screen displayed on the display device of the information processing device according to the first embodiment. FIG. 12 is a diagram illustrating an example in which the second detection area is designated in the video display unit on the setting screen displayed on the display device of the information processing device according to the first embodiment. FIG. 13 illustrates a state in which a trigger area set with video data different from the video data displayed on the video display unit is selected on the setting screen displayed on the display device of the information processing apparatus according to the first embodiment. It is a figure which shows the example of. FIG. 14 shows a state in which a trigger area set with video data different from the video data displayed on the video display unit is selected on the setting screen displayed on the display device of the information processing apparatus according to the first embodiment. It is a figure which shows the example of the state of. FIG. 15 illustrates detection corresponding to a trigger region set with video data different from the video data displayed on the video display unit on the setting screen displayed on the display device of the information processing apparatus according to the first embodiment. It is a figure which shows the example which designates an area | region. FIG. 16 is a diagram illustrating an example of setting the detection area attribute information designated by the video display unit on the setting screen displayed on the display device of the information processing apparatus according to the first embodiment. The detection area designation and detection area attribute information setting will be described with reference to FIGS.

  A setting screen 500 shown in FIG. 11 shows a state in which the detection area 535 is specified in the video display unit 502 in which video data is played back and displayed by the first playback unit 304. Specifically, first, during the reproduction and display of the video data on the video display unit 502, the pause button 503b is pressed by the operation of the input device 303 by the user at the timing when the part of the image for which image determination is desired is displayed. Then, the first reproduction unit 304 pauses the video data reproduced and displayed on the video display unit 502 and causes the time display unit 506 to display the imaging date and imaging time of the frame at that time. Next, when the user operates the input device 303 (for example, dragging the mouse 108), the detection area designating unit 307 designates and displays the detection area 535 on the video display unit 502. However, the detection region designation unit 307 can designate the detection region 535 because the trigger region designated by the trigger region designation unit 305 is selected in the trigger region information unit 507 (“TR6” in the example of FIG. 11). Need to be. That is, the detection area 535 designated by the detection area designation unit 307 is associated with the trigger area selected in the trigger area information unit 507.

  In addition, the setting unit 308 uses, as attribute information of the detection region 535, a name that is named for the detection region 535 designated by the detection region designation unit 307, and a predetermined sensitivity (for example, used in image determination described later) as a default value. , “50”) and a predetermined threshold (eg, “20%”), monitoring method (eg, “grayscale”), presence / absence of action (eg, “none”), and image determination method (eg, “match”) ) Is displayed on the detection area information unit 509. The setting unit 308 names “K61” as the name of the detection region 535, as shown in FIG. 11, for example, as the first region associated with the trigger region “TR6”. The setting unit 308 also delays the time of the frame at the time when the detection region 535 is specified by the detection region specifying unit 307 with respect to the time of the change point detected by the change point detection unit 306 (example in FIG. 11). Then, “2.14”) (determination time) is calculated as attribute information and displayed on the detection area information unit 509. That is, in this case, the detection region 535 named “K61” is set to perform image determination “2.14” seconds after the trigger timing. In addition, the setting unit 308 causes the storage unit 302 to store the attribute information of the detection area 535 displayed on the detection area information unit 509.

  In addition, the detection area designating unit 307 extracts an image in the detection area 535 at the time when the detection area 535 is designated as a detection reference image and stores it in the storage unit 302, and stores information on the designated detection area 535 in the storage unit 302. Remember me.

  A setting screen 500 shown in FIG. 12 shows a state in which the detection area 536 that is the second detection area is designated in the video display unit 502 in which video data is reproduced and displayed by the first reproduction unit 304. Specifically, first, during the reproduction display (pause) of the video data on the video display unit 502, the slider 505 of the seek bar 504 is slid by the operation of the input device 303 by the user, and the part of the image for which image determination is desired. Adjust to the timing when is displayed. The first reproduction unit 304 causes the time display unit 506 to display the imaging date and the imaging time of the frame of the video data temporarily stopped on the video display unit 502.

  Next, when the user operates the input device 303 (for example, dragging the mouse 108), the detection area designating unit 307 designates and displays the detection area 536, which is the second detection area, on the video display unit 502. . However, the detection area designation unit 307 can designate the detection area 536, as in the case of designation of the detection area 535, in the trigger area information part 507, the trigger area designated by the trigger area designation unit 305 is selected ( In the example of FIG. 12, “TR6” is required. That is, the detection area 536 designated by the detection area designation unit 307 is associated with the trigger area selected in the trigger area information unit 507.

  In addition, the setting unit 308 uses, as attribute information of the detection region 536, a name given to the detection region 536 designated by the detection region designation unit 307, and a predetermined sensitivity (for example, used in image determination described later) as a default value. , “50”) and a predetermined threshold (eg, “20%”), monitoring method (eg, “grayscale”), presence / absence of action (eg, “none”), and image determination method (eg, “match”) ) Is displayed on the detection area information unit 509. As illustrated in FIG. 12, the setting unit 308 names “K62” as the name of the detection region 536, for example, as the second region associated with the detection region “TR6”. The setting unit 308 also delays the time of the frame at the time when the detection region 536 is specified by the detection region specifying unit 307 with respect to the time of the change point detected by the change point detection unit 306 (example in FIG. 12). Then, “−1.86”) is calculated as attribute information and displayed on the detection area information unit 509. That is, in this case, the detection area 535 named “K62” is set to perform image determination “1.86” seconds before the trigger timing. In addition, the setting unit 308 causes the storage unit 302 to store the attribute information of the detection region 536 displayed on the detection region information unit 509.

  In addition, the detection area designating unit 307 extracts an image in the detection area 536 at the time when the detection area 536 is designated as a detection reference image and stores it in the storage unit 302, and stores information on the designated detection area 536 in the storage unit 302. Remember me.

  A setting screen 500 shown in FIG. 13 is an image of the image pickup apparatus 2 different from the image pickup apparatus 2 that picked up the image data being reproduced and displayed in the image display section 502 where the image data is reproduced and displayed by the first reproduction unit 304. The state which selects the trigger area designated in data is shown. Specifically, first, when a pull-down button 507a of the trigger area information unit 507 is pressed by an operation of the input device 303 by the user, a pull-down menu 507b appears from the “trigger area name” column. The setting unit 308 causes the specified trigger area among the video data of the plurality of imaging devices 2 to be displayed in an array on the pull-down menu 507b.

  A setting screen 500 illustrated in FIG. 14 illustrates a state in which “TR2” is selected from the trigger area displayed in the pull-down menu 507b by the user operating the input device 303. Here, the trigger area whose trigger area name is “TR2” is, for example, a trigger area designated for the video data of the imaging apparatus (imaging apparatus 2b) of “2”.

  Specifically, first, any of the trigger areas ("TR2" and "TR6" in the example of FIG. 13) displayed in an array on the pull-down menu 507b of the trigger area information unit 507 by the operation of the input device 303 by the user. The trigger area (“TR2” in the example of FIG. 14) is selected. Then, the setting unit 308 acquires the sensitivity and threshold value of the trigger region “TR2” selected by the input device 303 from the storage unit 302 and causes the trigger region information unit 507 to display it. In addition, in the video data of the imaging device “6”, the setting unit 308 has no detection region associated with the trigger region “TR2”, and thus the detection regions 535 and 536 displayed in the detection region information unit 509. Clear the display of attribute information. Further, the change point detection unit 306 acquires the reference image and difference image of the trigger region “TR2” from the storage unit 302, displays the reference image on the reference image display unit 508a, and causes the difference image to be displayed on the difference image display unit 508b. Display.

  A setting screen 500 shown in FIG. 15 shows a state in which a detection area 537, which is the third detection area, is designated in the video display section 502 where video data is played back and displayed by the first playback section 304. Specifically, first, during the reproduction display (pause) of the video data on the video display unit 502, the slider 505 of the seek bar 504 is slid by the operation of the input device 303 by the user, and the part of the image for which image determination is desired. Adjust to the timing when is displayed. The first reproduction unit 304 causes the time display unit 506 to display the imaging date and the imaging time of the frame of the video data temporarily stopped on the video display unit 502.

  Next, when the user operates the input device 303 (for example, dragging the mouse 108), the detection area designating unit 307 designates and displays the detection area 537, which is the third detection area, on the video display unit 502. . In this case, the detection region 536 designated by the detection region designation unit 307 is associated with the trigger region “TR2” selected in the trigger region information unit 507.

  In addition, the setting unit 308 uses, as attribute information of the detection region 537, a name given to the detection region 537 specified by the detection region specifying unit 307, and a predetermined sensitivity (for example, used in image determination described later) as a default value. , “50”) and a predetermined threshold (eg, “20%”), monitoring method (eg, “grayscale”), presence / absence of action (eg, “none”), and image determination method (eg, “match”) ) Is displayed on the detection area information unit 509. As illustrated in FIG. 15, the setting unit 308 names “K21” as the first region associated with the detection region “TR2”, for example, as illustrated in FIG. In addition, the setting unit 308 detects the change point of the trigger region “TR2” of the video data of the image pickup device “2” (the image pickup device 2b) detected by the change point detection unit 306. The default value of the delay time (“1.50” in the example of FIG. 15) at the timing at which the detection area 537 is designated by 307 is determined as attribute information and displayed on the detection area information unit 509. In addition, the setting unit 308 causes the storage unit 302 to store the attribute information of the detection region 537 displayed on the detection region information unit 509.

  In addition, the detection area designating unit 307 extracts an image in the detection area 537 at the time when the detection area 537 is designated as a detection reference image and stores it in the storage unit 302, and stores information on the designated detection area 537 in the storage unit 302. Remember me.

  As described above, since the detection area can be specified in association with the trigger area specified by the video data of the imaging device 2 different from the imaging device 2 related to the video data displayed on the video display unit 502, a plurality of imaging devices 2 can be specified. Can be linked, and the degree of freedom of image determination can be dramatically improved.

  The setting screen 500 shown in FIG. 16 has a detection area “K61” (attribute information displayed in the detection area information section 509 in a state where the trigger area “TR6” (trigger area 530) is selected in the trigger area information section 507). In the detection area 535) and “K62” (detection area 536), “K62” is selected. The editing of the attribute information of the detection area will be described with reference to FIG. However, the sensitivity adjustment of the detection area when the sensitivity determination button 510d of the detection area button unit 510 is pressed will be described later with reference to FIGS.

  First, as illustrated in FIG. 16, when one of detection areas whose attribute information is displayed in the detection area information unit 509 is selected by the user's operation of the input device 303, the setting unit 308 displays the selected detection area information. The display part of the attribute information of the detection area of the part 509 is highlighted. In the following description, it is assumed that the detection area 536 (detection area name “K62”) is selected as shown in FIG.

  Next, for example, when the action button 510 a is pressed by the user operating the input device 303, the setting unit 308 switches the setting value of the presence or absence of the action in the selected detection area 536. For example, when the presence / absence of an action in the detection area 536 is “none”, when the action button 510a is pressed, the setting unit 308 sets the presence / absence of an action in the detection area 536 to “present”. On the other hand, when the presence / absence of the action in the detection area 536 is “Yes”, when the action button 510a is pressed, the setting unit 308 sets the presence / absence of the action in the detection area 536 to “No”. As will be described later, when the presence / absence of an action is “Yes”, the external output unit 316 outputs an abnormal signal when the detection region determination unit 314 determines that the image determination in the detection region is abnormal.

  For example, when the image determination selection button 510b is pressed by the user operating the input device 303, the setting unit 308 switches the setting value of the screen determination method of the selected detection area 536. For example, when the screen determination method of the detection area 536 is “match”, when the image determination selection button 510b is pressed, the setting unit 308 sets the screen determination method of the detection area 536 to “mismatch”. To do. On the other hand, when the screen determination method of the detection area 536 is “mismatch”, when the image determination selection button 510b is pressed, the setting unit 308 sets the screen determination method of the detection area 536 to “match”. To do.

  As will be described later, when the screen determination method is “match”, the detection area determination unit 314 determines that the difference between the extracted image and the detection reference image is equal to or greater than the threshold value of the attribute information as abnormal. . In this way, as an example of the case where the screen judgment method is “match”, in a facility for inserting a part into a workpiece, the state where the part is inserted into the workpiece is used as a detection reference image, and the part is inserted into the workpiece. There is a case where an image is determined at the timing when the operation is completed. In this case, when the part insertion operation to the workpiece fails and the image determination is executed by the detection region determination unit 314 in a state where the part is not normally inserted, the difference from the detection reference image is equal to or greater than the threshold value of the attribute information. Thus, it is determined that there is an abnormality.

  On the other hand, when the screen determination method is “mismatch”, the detection region determination unit 314 determines that the difference is between the extracted image and the detection reference image is less than the threshold value of the attribute information. In this way, as an example of the case where the screen judgment method is “mismatch”, in the equipment for inserting a part into the workpiece, the state where the part is not inserted into the workpiece is used as a detection reference image, and the part is inserted into the workpiece. There is a case where an image is determined at the timing when the operation is completed. In this case, when the operation of inserting the component into the workpiece fails and the detection region determination unit 314 performs image determination in a state where the component is not normally inserted, the difference from the detection reference image is less than the threshold value of the attribute information. Thus, it is determined that there is an abnormality. A situation in which a state in which no part is inserted into the workpiece is used as the detection reference image includes a case where it is difficult to prepare a workpiece into which a component as a finished product is inserted.

  For example, when the user presses the delete button 510 c by operating the input device 303, the setting unit 308 deletes the attribute information of the detection area 536 displayed in the detection area information unit 509 and stores the attribute information in the storage unit 302. The attribute information of the detected area 536 is deleted. Further, the detection area designating unit 307 deletes the detection area 536 displayed on the video display unit 502 and deletes the detection reference image and the information on the detection area 536 stored in the storage unit 302.

  For example, when the sensitivity determination button 510d is pressed by the user operating the input device 303, the setting unit 308 causes the display device 317 to display a sensitivity setting preview screen 700 illustrated in FIG.

  For example, when the sensitivity increase button 510e is pressed by the user operating the input device 303, the setting unit 308 increases the sensitivity value used for image determination of the selected detection region 536 by a predetermined value. On the other hand, when the sensitivity reduction button 510f is pressed by the user operating the input device 303, the setting unit 308 decreases the sensitivity value used for image determination of the selected detection region 536 by a predetermined value.

  For example, when the threshold value increase button 510g is pressed by the user operating the input device 303, the setting unit 308 increases the threshold value used for image determination of the selected detection region 536 by a predetermined value. On the other hand, when the threshold value decrease button 510h is pressed by the user operating the input device 303, the setting unit 308 decreases the threshold value used for image determination of the selected detection region 536 by a predetermined value.

(Change of detection area and trigger area size)
FIG. 17 is a diagram illustrating an example of a state in the case where the size of the detection area already designated in the video display unit is changed on the setting screen displayed on the display device of the information processing device according to the first embodiment. . FIG. 18 is a diagram illustrating an example of a state in the case where the size of the trigger region designated in the video display unit is changed on the setting screen displayed on the display device of the information processing device according to the first embodiment. . The operation when the sizes of the detection area and the trigger area are changed will be described with reference to FIGS.

  A setting screen 500 illustrated in FIG. 17 illustrates a state in which the size of the detection area 535 designated in the video display unit 502 in which video data is reproduced and displayed by the first reproduction unit 304 is changed. Specifically, first, during the reproduction display (pause) of the video data on the video display unit 502, the slider 505 of the seek bar 504 is slid by the operation of the input device 303 by the user, and the part of the image for which image determination is desired. Readjust to the timing when is displayed. The first reproduction unit 304 causes the time display unit 506 to display the imaging date and the imaging time of the frame of the video data temporarily stopped on the video display unit 502.

  Next, when the user operates the input device 303 (for example, dragging the mouse 108), the detection area designating unit 307, for example, the size of the detection area 536 that is the second detection area designated in the video display unit 502 is displayed. This is changed to a detection area 536a. At this time, the setting unit 308 delays the time of the frame when the size is changed to the detection region 536a by the detection region specifying unit 307 with respect to the time of the change point detected by the change point detection unit 306 ( In the example of FIG. 17, “−0.86”) is recalculated as attribute information and displayed on the detection area information unit 509. In addition, the setting unit 308 causes the storage unit 302 to store the attribute information of the detection region 536 a that is updated and displayed in the detection region information unit 509.

  In addition, the detection area designating unit 307 extracts an image in the detection area 536a at the time when the size is changed to the detection area 536a as a detection reference image, stores the extracted image in the storage unit 302, and stores information on the designated detection area 536a. The data is stored in the unit 302.

  A setting screen 500 shown in FIG. 18 shows a state in which the size of the trigger area 530 is changed in the video display unit 502 where video data is played back and displayed by the first playback unit 304. Specifically, first, during playback / pause of video data on the video display unit 502, the slider 505 of the seek bar 504 is slid by the operation of the input device 303 by the user, and a desired basis for generating a trigger is generated. Readjust to the timing when the part of the image to be displayed is displayed. The first reproduction unit 304 causes the time display unit 506 to display the imaging date and the imaging time of the frame of the video data temporarily stopped on the video display unit 502.

  Next, when the user operates the input device 303 (for example, dragging the mouse 108), the trigger area specifying unit 305 changes the size of the trigger area 530 specified in the video display unit 502 to be the trigger area 530a. . When the size of the trigger region 530 designated by the trigger region designation unit 305 is changed to become the trigger region 530a, the change point detection unit 306 reproduces and displays the first reproduction unit 304 when the size is changed. A frame for a predetermined time (for example, 2 seconds before and after the imaging time) of the frame of the video data being acquired is acquired, and an image in the trigger area 530a at the time when the size is changed to the trigger area 530a is newly set as a reference image. Extract as Thereafter, the operation of calculating the difference of the acquired frame, the detection of the change point, and the extraction of the difference image by the change point detection unit 306 is the same as the operation described above with reference to FIGS.

  The change point detection unit 306 stores the newly extracted reference image and the difference image in the frame of the change point in the storage unit 302, redisplays the reference image on the reference image display unit 508a, and displays the difference image as the difference image display unit. It is displayed again at 508b. Further, the change point detection unit 306 deletes the trigger mark 540 in the seek bar 504, and displays the trigger mark 540a at a portion corresponding to the time of the frame of the newly detected change point.

  When a new change point is detected by the change point detection unit 306, the trigger area specifying unit 305 causes the storage unit 302 to store information on the trigger area 530a.

  The setting unit 308 delays the time of the frame at the time when the detection region 535 is specified by the detection region specifying unit 307 with respect to the time of the frame of the change point newly detected by the change point detection unit 306 (FIG. 18). In this example, “1.64”) is recalculated as attribute information and displayed on the detection area information unit 509. The setting unit 308 also sets the time of the frame at the time when the size of the frame of the change point newly detected by the change point detection unit 306 is changed to the detection region 536a by the detection region specifying unit 307. The delay time (“−1.36” in the example of FIG. 18) is recalculated as attribute information and displayed on the detection area information unit 509. In addition, the setting unit 308 causes the storage unit 302 to store the attribute information of the detection areas 535 and 536a updated and displayed in the detection area information unit 509.

(Sensitivity adjustment in the sensitivity setting preview screen)
FIG. 19 is a diagram illustrating an example of a state in which video data is displayed on the sensitivity setting preview screen displayed on the display device of the information processing device according to the first embodiment. FIG. 20 is a diagram illustrating an example of a determination result of the difference between the extracted images with respect to the detection reference image on the sensitivity setting preview screen displayed on the display device of the information processing apparatus according to the first embodiment. FIG. 21 is a diagram illustrating an example of a result of re-determination of the difference between the extracted images with respect to the detection reference image at different sensitivities on the sensitivity setting preview screen displayed on the display device of the information processing device according to the first embodiment. . FIG. 22 is a diagram illustrating an example in which a dialog for confirming whether to apply the changed sensitivity on the sensitivity setting preview screen displayed on the display device of the information processing apparatus according to the first embodiment is displayed. . FIG. 23 is a diagram illustrating an example of a state in which the sensitivity changed on the sensitivity setting preview screen is reflected on the setting screen displayed on the display device of the information processing apparatus according to the first embodiment. The sensitivity adjustment on the sensitivity setting preview screen 700 will be described with reference to FIGS.

  The sensitivity setting preview screen 700 is a screen for adjusting the sensitivity in image determination, which is one of the attribute information, for a selected detection area. As shown in FIG. 19, the sensitivity setting preview screen 700 includes a detection area information unit 701, a video display unit 702, a detection reference image display unit 703, an extracted image display unit 704, a difference display unit 705, and a color-specific display. Detection number display section 706, area pixel number display section 707, difference detection number display section 708, difference detection ratio display section 709, sensitivity increase button 710a, sensitivity decrease button 710b, determination button 711, and OK button 712 and a close button 713.

  The detection area information part 701 is a display part on which attribute information of the detection area selected by the setting part 308 in the detection area information part 509 of the setting screen 500 is displayed. In the example of FIG. 19, the detection area information portion 701 displays the camera name “CAM6”, the detection area name “K61”, the sensitivity “50”, and the threshold value “20%”. .

  The video display unit 702 is a display unit that reproduces and displays video data acquired from the storage unit 302 according to an operation signal from the input device 303 operated by the user by the second playback unit 309. In the example of FIG. 19, the second playback unit 309 displays the video data captured by the imaging device 2 f as the display device because the camera selection unit 501 of “6” is selected by the user operating the input device 303. 317 is displayed. The video display unit 702 reproduces and displays the video data acquired from the storage unit 302 by the second playback unit 309. However, the present invention is not limited to this, and the video reception unit 301 from the imaging device 2 is not limited thereto. Real-time video data acquired via the display may be displayed.

  The detection reference image display unit 703 is a display unit on which the detection reference image extracted by the detection region specifying unit 307 by the setting unit 308 is acquired from the storage unit 302 and displayed. The extracted image display unit 704 is a display unit on which the extracted image extracted by the image extracting unit 310 is displayed. The difference display unit 705 is a display unit that displays an image (difference feature image) indicating the difference feature between the detection reference image calculated by the difference determination unit 311 and the extracted image.

  The detection number display unit 706 for each color displays the detection number of pixels belonging to each predetermined width for the difference among the information (difference information) about the difference between the detection reference image calculated by the difference determination unit 311 and the extracted image. It is a display part. The region pixel number display unit 707 is a display unit that displays the number of pixels constituting the detection region that is the target as difference information, which is calculated by the difference determination unit 311 when calculating the difference. The difference detection number display unit 708 is a display unit that displays the number of pixels calculated as a difference information by the difference determination unit 311 and having a difference equal to or greater than a predetermined threshold. The difference detection ratio display unit 709 is calculated in the difference calculation by the difference determination unit 311, and the ratio of the number of pixels in which the difference is equal to or larger than a predetermined threshold to the number of pixels constituting the target detection area is the difference information. It is a display part displayed.

  The sensitivity increase button 710a is a button that, when pressed by the user operating the input device 303, causes the setting unit 308 to increase the sensitivity value used for image determination of the target detection area by a predetermined value. The function of the sensitivity increase button 710a is the same as the function of the sensitivity increase button 510e described above.

  The sensitivity reduction button 710b is a button that, when pressed by the user operating the input device 303, causes the setting unit 308 to decrease the sensitivity value used for image determination of the target detection area by a predetermined value. The function of the sensitivity decrease button 710b is the same as the function of the sensitivity decrease button 510f described above.

  When the determination button 711 is pressed by the operation of the input device 303 by the user, an extracted image is extracted by the image extraction unit 310, and a difference between the detection reference image and the extracted image is calculated by the difference determination unit 311. Is a button that is calculated.

  When the OK button 712 is pressed by the operation of the input device 303 by the user, the sensitivity used by the setting unit 308 to calculate the difference by the difference determination unit 311 (sensitivity displayed in the detection region information unit 701). Are buttons reflected in the detection area information part 509 of the setting screen 500.

  When the close button 713 is pressed by the user operating the input device 303, the sensitivity setting preview screen 700 is closed by the setting unit 308 and the setting screen 500 is displayed on the display device 317.

  The sensitivity setting preview screen 700 shown in FIG. 19 is displayed when the detection area 535 (detection area name “K61”) in the detection area information section 509 is selected by the user's operation of the input device 303 and the sensitivity determination button 510d is pressed. This is the screen that is displayed when Specifically, when the sensitivity determination button 510d is pressed in a state where the detection region 535 (detection region name “K61”) is selected by the operation of the input device 303 by the user, the setting unit 308 causes the setting screen 500 to be displayed. Is displayed on the display device 317. In addition, the setting unit 308 displays the attribute information of the selected detection region 535 on the detection region information unit 701, and the detection reference image corresponding to the detection region 535 extracted by the detection region specifying unit 307 is stored from the storage unit 302. Acquired and displayed on the detection reference image display unit 703. In the example of FIG. 19, the detection reference image displayed on the detection reference image display unit 703 indicates a screw hole of a work to be produced, and a screw is not yet fitted.

  Also, the second playback unit 309 acquires the video data of the imaging device 2 corresponding to the camera name displayed in the detection area information unit 701 from the storage unit 302 by the operation of the input device 303 by the user, and the video display unit 702 is displayed for reproduction.

  In the sensitivity setting preview screen 700 shown in FIG. 20, when the determination button 711 is pressed by the user operating the input device 303, the extracted image is extracted by the image extraction unit 310, and the detection reference image is extracted by the difference determination unit 311. The result of calculating the difference from the image is shown. Specifically, first, while the video data is being displayed on the video display unit 702, the determination button 711 is pressed by the user operating the input device 303 at a timing when the difference calculation is desired. Then, the image extraction unit 310 extracts, as an extracted image, an image of a portion corresponding to the detection area 535 designated by the detection area designation unit 307 in the video display unit 702 where the video data is displayed by the second reproduction unit 309. And displayed on the extracted image display unit 704. In the example of FIG. 20, the extracted image displayed on the extracted image display unit 704 is a screw hole of a work to be produced and shows a state in which a screw is fitted.

  Then, the difference determination unit 311 calculates the difference between the detection reference image and the extracted image extracted by the image extraction unit 310, and the calculated difference has a predetermined width (in the example of FIG. 20, “70 or more”, “50”). To 69 ”,“ 20 to 49 ”, and“ 19 or less ”) are displayed on the color detection number display unit 706. Further, the difference determination unit 311 calculates the number of pixels constituting the detection region 535 that is the target of the difference calculation, and causes the region pixel number display unit 707 to display it. In addition, the difference determination unit 311 calculates the number of pixels in which the difference calculated in the extracted image is equal to or greater than a predetermined threshold value, and displays the calculated number on the difference detection number display unit 708. Then, the difference determination unit 311 calculates the ratio of the number of pixels in which the difference is equal to or greater than a predetermined threshold in the extracted image to the number of pixels constituting the detection area 535 that is the target of the difference calculation, and the difference detection ratio It is displayed on the display unit 709. Here, the maximum value of the difference is, for example, “255”.

  Also, the difference determination unit 311 causes the difference display unit 705 to display a difference feature image indicating the calculated difference feature. Specifically, as shown in FIG. 20, the difference determination unit 311 color-codes the calculated difference according to which of the above-described predetermined widths, and corresponds to the difference on the pixel corresponding to the calculated difference. A difference feature image is generated by superimposing dots of the color to be used.

  The sensitivity setting preview screen 700 shown in FIG. 21 shows the result of the user performing the difference calculation again by the difference determination unit 311 with the sensitivity reduced compared to the case of FIG. 20 based on the difference result of FIG. ing. Specifically, first, the sensitivity reduction button 710b is pressed by the user operating the input device 303, and the sensitivity used for the difference calculation is decreased from “50” to “40”. Then, during the display of the video data on the video display unit 702, the determination button 711 is pressed by the user operating the input device 303 at the timing when the difference calculation is desired (here, the timing is the same as the timing in FIG. 20). Then, the image extraction unit 310 extracts, as an extracted image, an image of a portion corresponding to the detection area 535 designated by the detection area designation unit 307 in the video display unit 702 where the video data is displayed by the second reproduction unit 309. And displayed on the extracted image display unit 704.

  Here, since the determination button 711 is pressed at the timing of FIG. 20, the images displayed on the extracted image display unit 704, the difference display unit 705, the detection number display unit 706 for each color, and the area pixel number display unit 707. The information is the same as in FIG.

  However, the difference determination unit 311 calculates the number of pixels in which the difference calculated in the extracted image is equal to or greater than a predetermined threshold value, and displays the calculated number on the difference detection number display unit 708. In this case, since the sensitivity of the difference calculation performed by the difference determination unit 311 is lower than that in the case of FIG. 20, the number of pixels in which the difference calculated in the extracted image is equal to or greater than a predetermined threshold is greater than in the case of FIG. (In the example of FIG. 21, it is “10” to “16”). Then, the difference determination unit 311 calculates the ratio of the number of pixels in which the difference is equal to or greater than a predetermined threshold in the extracted image to the number of pixels constituting the detection area 535 that is the target of the difference calculation, and the difference detection ratio It is displayed on the display unit 709. In this case, since the sensitivity of the difference calculation by the difference determination unit 311 is lower than that in FIG. 20, the ratio of the number of pixels in which the difference calculated in the extracted image is equal to or greater than a predetermined threshold is shown in FIG. (In the example of FIG. 21, it is “4.4%” to “7.1%”).

  A sensitivity setting preview screen 700 shown in FIG. 22 shows a state in which a sensitivity application confirmation dialog 800 is displayed when the user presses an OK button 712 in order to apply the sensitivity adjusted in FIG. A setting screen 500 shown in FIG. 23 shows a state in which the sensitivity adjusted for the target detection region (the detection region 535 in FIGS. 22 and 23) in the sensitivity setting preview screen 700 is reflected in the detection region information unit 509. .

  Specifically, when the OK button 712 is pressed by the user operating the input device 303, the setting unit 308 applies the sensitivity adjusted in FIG. 21 (value is “40”), that is, on the setting screen 500. The display device 317 displays a sensitivity application confirmation dialog 800 for confirming whether or not to reflect the detection area information unit 509. The sensitivity application confirmation dialog 800 includes a yes button 801, a no button 802, a cancel button 803, and a close button 804. A close button 804 is a button for closing the sensitivity application confirmation dialog 800.

  When the user presses the yes button 801 by operating the input device 303, the difference determination unit 311 shows the sensitivity adjusted in FIG. 21 (sensitivity displayed in the detection area information unit 701) as shown in FIG. To be reflected in the detection area information section 509 of the setting screen 500. Then, the setting unit 308 closes the sensitivity application confirmation dialog 800 and the sensitivity setting preview screen 700 and causes the display device 317 to display the setting screen 500.

  When the user presses the no button 802 by operating the input device 303, the difference determination unit 311 does not reflect the sensitivity adjusted in FIG. 21 on the detection area information unit 509 of the setting screen 500. Then, the setting unit 308 closes the sensitivity application confirmation dialog 800 and the sensitivity setting preview screen 700 and causes the display device 317 to display the setting screen 500.

  When the user presses the cancel button 803 by operating the input device 303, the difference determination unit 311 does not reflect the sensitivity adjusted in FIG. 21 on the detection area information unit 509 of the setting screen 500. Then, the setting unit 308 closes the sensitivity application confirmation dialog 800 and keeps the sensitivity setting preview screen 700 displayed on the display device 317.

(Flow of setting operation to specify trigger area and detection area)
FIG. 24 is a flowchart illustrating an example of a setting operation for designating a trigger area and a detection area on a setting screen displayed on the display device of the information processing apparatus according to the first embodiment. With reference to FIG. 24, the flow of the setting operation for designating the trigger area and the detection area on the setting screen 500 will be generally described.

<Step S11>
The first playback unit 304 acquires video data from the storage unit 302 according to the operation of the input device 303 by the user, and causes the video display unit 502 to play back and display the video data. And then. The process proceeds to step S12.

<Step S12>
When the user presses the pause button 503b by the operation of the input device 303 at the timing when the desired image portion that is the basis of the occurrence of the trigger is displayed while the video data is being reproduced and displayed on the video display unit 502. The first playback unit 304 pauses the video data that is played back and displayed on the video display unit 502. When the user operates the input device 303 (for example, dragging the mouse 108), the trigger area designating unit 305 designates and displays the trigger area on the video display unit 502. Then, the process proceeds to step S13.

<Step S13>
When the trigger area is specified by the trigger area specifying unit 305, the change point detecting unit 306 is a predetermined time before and after the imaging time of the frame of the video data that the first playback unit 304 reproduces and displays at the specified time ( (For example, 2 seconds before and after) frames are acquired, and in the acquired frames, a difference in pixel value from the immediately preceding frame is calculated for each frame, and a change point for the image in the trigger region is calculated based on the calculated difference. Is detected. A specific change point detection method is as described above with reference to FIGS. Then, the process proceeds to step S14.

<Step S14>
When the change point is detected by the change point detection unit 306 (step S14: Yes), the process proceeds to step S15. When the change point is not detected (step S14: No), the process returns to step S11.

<Step S15>
When the change point detection unit 306 detects the IN point as the change point, the change point detection unit 306 extracts an image in the trigger region 530 of the frame at the IN point as a difference image. In addition, when the OUT point is detected as the change point, the change point detection unit 306 extracts an image in the trigger region 530 of the frame at the OUT point as a difference image. The change point detection unit 306 stores the extracted reference image and the difference image in the frame of the change point in the storage unit 302, displays the reference image on the reference image display unit 508a, and displays the difference image on the difference image display unit 508b. Let The trigger area specifying unit 305 causes the storage unit 302 to store information on the specified trigger area. Then, the process proceeds to step S16.

<Step S16>
While the video data is being reproduced and displayed on the video display unit 502, the user inputs the input device 303 at the timing when the portion of the image for which image determination is desired is displayed, i. When the pause button 503b is pressed by the above operation, the first reproduction unit 304 pauses the video data reproduced and displayed on the video display unit 502. Then, the process proceeds to step S17.

<Step S17>
When the user operates the input device 303 (for example, dragging the mouse 108), the detection area designating unit 307 designates and displays the detection area on the video display unit 502. In addition, the setting unit 308 includes, as attribute information of the detection region 535, a name that is named for the detection region specified by the detection region specifying unit 307, and a default value that includes a predetermined sensitivity and a predetermined threshold value used in image determination. The monitoring area information unit 509 displays the monitoring method, the presence / absence of an action, and the image determination method (for example, “match”). Also, the setting unit 308 calculates, as attribute information, the delay time of the frame time when the detection region is specified by the detection region specification unit 307 with respect to the time of the change point detected by the change point detection unit 306. And displayed on the detection area information unit 509. In addition, the setting unit 308 causes the storage unit 302 to store the attribute information of the detection area 535 displayed on the detection area information unit 509. Then, the process proceeds to step S18.

<Step S18>
When one of the detection areas whose attribute information is displayed in the detection area information unit 509 is selected by the user's operation of the input device 303, the setting unit 308 displays the attribute information of the detection area of the selected detection area information unit 509. Highlight the display area. Next, when the action button 510a is pressed by the user's operation of the input device 303, the setting unit 308 switches setting values for presence / absence of an action in the selected detection area. When the image determination selection button 510b is pressed by the user operating the input device 303, the setting unit 308 switches the setting value of the screen determination method for the selected detection area.

  When the sensitivity increase button 510e is pressed by the user operating the input device 303, the setting unit 308 increases the sensitivity value used for image determination of the selected detection area by a predetermined value. On the other hand, when the sensitivity reduction button 510f is pressed by the user operating the input device 303, the setting unit 308 decreases the sensitivity value used for image determination of the selected detection region by a predetermined value. Note that the sensitivity used for image determination of the selected detection area can be set on the sensitivity setting preview screen 700 as described above.

  When the threshold value increase button 510g is pressed by the user operating the input device 303, the setting unit 308 increases the threshold value used for image determination of the selected detection area by a predetermined value. On the other hand, when the threshold value decrease button 510h is pressed by the user operating the input device 303, the setting unit 308 decreases the threshold value used for image determination of the selected detection area by a predetermined value. When the setting of the attribute information of the selected detection area is completed, the process proceeds to step S19.

<Step S19>
In the video display unit 502 displaying the video data, when all the detection areas are designated (step S19: Yes), the setting operation for designating the trigger area and the detection area is finished, and all the detection areas are designated. If not (step S19: No), the process returns to step S16.

(Image judgment operation on the main screen)
FIG. 25 is a diagram illustrating an example of a state after the trigger region and the detection region are designated on the main screen displayed on the display device of the information processing device according to the first embodiment. FIG. 26 is a flowchart illustrating an example of an image determination operation performed by the information processing apparatus according to the first embodiment. FIG. 27 shows the display state of the trigger area when the trigger of the trigger area occurs on the main screen displayed on the display device of the information processing apparatus according to the first embodiment, and the detection at the detection timing of the detection area. It is a figure which shows the example of the display state of an area | region.

  First, an image determination operation on the main screen 400 will be described with reference to FIG. The main screen 400 shown in FIG. 25 displays video data received from a plurality of imaging devices 2 in real time, the trigger area “TR2” is designated by the video display section 401b, and the trigger area “TR6” is designated by the video display section 401f. In addition, the detection areas “K61”, “K62”, and “K21” are designated. In a specific image determination operation on the main screen 400, first, the video distribution unit 312 distributes and displays the real-time video data acquired from the video reception unit 301 on the video display units 401a to 401f. The trigger generation unit 313 acquires, from the storage unit 302, a difference image in the frame of the change point detected by the change point detection unit 306. Then, the trigger generation unit 313 performs the difference image and the frame of the video data in the trigger region “TR2” specified by the video display unit 401b and the trigger region “TR6” specified by the video display unit 401f, respectively. Compare the image in the trigger area. Note that the video display portions 401b and 401f here correspond to the “second display area”. The trigger generation unit 313 is a difference (second difference) in which the image of the trigger region of the video data frame is the sum of the number of pixels whose pixel values differ by a predetermined sensitivity or more compared to the difference image in the change point frame. A trigger signal is generated at a timing when becomes less than a predetermined threshold.

  It is assumed that the detection delay time is set to the value shown in FIG. 23 among the attribute information of the detection areas “K61” and “K62”. Further, it is assumed that the detection delay time is set to the value shown in FIG. 15 in the attribute information of the detection area “K21”. The detection area determination unit 314 acquires attribute information of the designated detection area from the storage unit 302. The detection area determination unit 314 detects the detection reference image and the detection area of the frame of the video data in the designated detection area before or after the set time when the trigger signal generated by the trigger generation unit 313 is received. The image is compared with the image to determine whether it is abnormal.

  Specifically, the detection region determination unit 314 receives the trigger signal corresponding to the trigger region “TR6” from the trigger generation unit 313, and then the detection delay time “1.5” seconds corresponding to the detection region “K61”. The image of the detection area of the frame of the subsequent video data is compared with the detection reference image to determine whether or not there is an abnormality. At this time, as the image determination, the detection area determination unit 314 is the total of the number of pixels whose pixel values differ by more than the sensitivity of the attribute information of the detection area “K61” in the comparison between the detection reference image and the detection area image. When the difference (third difference) is equal to or greater than the threshold value (third threshold value) of the attribute information, the image determination of the detection region “K61” is determined to be abnormal. As described above, as an example of the case where the detection region determination unit 314 determines the image at the previous timing from when the trigger signal is received, in the equipment such as a press-fitting machine, the timing after the press-fitting operation is used as a trigger. A case where the image of the press-fitted state is determined is mentioned. Specifically, the trigger generation unit 313 generates a trigger signal with a timing at which the press-fitting machine starts an operation of press-fitting the press-fitting part into the work as a trigger. Thereafter, the press-fitting machine press-fits the press-fitting parts into the work, and after the press-fitting operation, each actuator of the press-fitting machine returns to the origin position. Then, the detection area determination unit 314 performs image determination on the press-fitted state of the work after press-fitting. The timing of the image determination by the detection region determination unit 314 is the timing after the time set by the detection delay time after receiving the trigger signal from the trigger generation unit 313. The time from the start of the press-fitting operation that generates the trigger signal to the execution of press-fitting image judgment after press-fitting is a fixed time because it is an automatic operation by the press-fitting machine, and detection from the trigger timing to image judgment The delay time can be set as a fixed time.

  In addition, the detection region determination unit 314 receives the trigger signal corresponding to the trigger region “TR6” from the trigger generation unit 313, and after the detection delay time “−0 · 86” seconds corresponding to the detection region “K62”, That is, the image of the detection area of the frame of the video data “0.86” seconds ago is compared with the detection reference image to determine whether or not there is an abnormality. As described above, as an example of the case where the detection region determination unit 314 determines the image at the timing before the trigger signal is received, in an installation such as a press-fitting machine that manually press-fits the press-fitting component, There is a case where an image determination is performed on a workpiece after press-fitting in a state where the workpiece is set in a press-fitting device before being put into a finished product case, with the timing of putting the workpiece after press-fitting into the finished product case as a trigger. Specifically, first, an operator sets a work before press-fitting on a press-fitting machine. Next, the operator presses the press-fitting start button, and the press-fitting machine starts the press-fitting operation of the press-fitting parts to the workpiece. After the press-fitting, the operator picks up the press-fitted work and puts it into a finished product case. The trigger generation unit 313 generates a trigger signal triggered by the timing at which the operator puts the press-fitted work into a finished product. The detection area determination unit 314 then press-fits the trigger signal at the timing set by the detection delay time from the time when the trigger signal is received, that is, at the timing set in the press-fitting machine before being put into the finished product case. The image of the subsequent workpiece is determined. As described above, the following situation can be avoided by performing image determination on the workpiece set in the press-fitting machine at the timing before the trigger signal is generated. That is, for example, after an operator sets a work, the worker temporarily leaves the work site during a lunch break or the like, and after returning, forgets the press-fitting operation and puts the work before press-fitting into a finished product case. In this case, the worker must be the workpiece before press-fitting by determining the image at the timing set in the detection delay time from the time of the trigger using the timing when the workpiece is put in the finished product case as a trigger. It is possible to prevent the workpiece before press-fitting from being mixed into the finished product case.

  In addition, the detection region determination unit 314 receives the trigger signal corresponding to the trigger region “TR2” from the trigger generation unit 313 and the video after the detection delay time “1.5” seconds corresponding to the detection region “K21”. The image in the detection area of the data frame is compared with the detection reference image to determine whether or not there is an abnormality.

  Then, the external output unit 316 outputs an abnormal signal to the external device 10 when the detection region determination unit 314 determines abnormality.

  Next, a series of flow of the image determination operation on the main screen 400 will be generally described with reference to FIG.

<Step S31>
When the video distribution unit 312 can acquire video data from the imaging device 2 via the video reception unit 301 (step S31: Yes), the process proceeds to step S32, and when acquisition is not possible (step S31: No), a standby state is entered.

<Step S32>
The video distribution unit 312 distributes and displays the real-time video data acquired from the video reception unit 301 on the video display units 401 a to 401 f and stores the video data in the storage unit 302. Then, the process proceeds to step S33.

<Step S33>
If there is a past frame as the video data stored in the storage unit 302 before a predetermined time before the current time (step S33: Yes), the process proceeds to step S34, and if it does not exist (step S33: No), step S31. Return to.

<Step S34>
The video distribution unit 312 deletes the past frames as the video data stored in the storage unit 302 from a predetermined time before the current time. Then, the process proceeds to step S35.

<Step S35>
The trigger generation unit 313 acquires, from the storage unit 302, a difference image in the frame of the change point detected by the change point detection unit 306. Then, the trigger generation unit 313 compares the difference image with the image in the trigger area of the video data frame in the designated trigger area. Then, the process proceeds to step S36.

<Step S36>
The trigger generation unit 313 is a difference (second difference) in which the image of the trigger region of the video data frame is the sum of the number of pixels whose pixel values differ by a predetermined sensitivity or more compared to the difference image in the change point frame. Is generated at a timing when becomes less than a predetermined threshold (second threshold). When the trigger signal is generated and output by the trigger generation unit 313 (step S36: Yes), the process proceeds to step S37, and when it is not output (step S36: No), the process returns to step S31. For example, as illustrated in FIG. 27A, the trigger generation unit 313 generates a trigger signal at the timing when the trigger signal is generated and output in the trigger region “TR6” (trigger region 530a) specified by the trigger region specifying unit 305. The frame color of the area 530a is displayed in reverse. Thereby, it is possible to visually grasp when the trigger is output by the trigger generation unit 313.

<Step S37>
Among the detection areas designated by the detection area designation unit 307, there is a detection area in which image detection (detection) is performed by the detection area determination unit 314 at a past timing from the trigger timing by the trigger generation unit 313 (step S37: Yes), the process proceeds to step S38, and if not (step S37: No), the process proceeds to step S41.

<Step S38>
The detection region determination unit 314 includes an image of a detection region of a frame of video data before a predetermined time (detection delay time) corresponding to the detection region from when the trigger signal corresponding to the trigger region is received by the trigger generation unit 313, The image is determined whether or not it is abnormal by comparing with the detection reference image. Then, the process proceeds to step S39.

<Step S39>
When the screen determination method of the attribute information of the detection area is “match”, the detection area determination unit 314 compares the detection reference image with the image of the detection area and the pixel value is equal to or higher than the sensitivity of the attribute information of the detection area. When the difference that is the total number of different pixels is equal to or greater than the threshold value of the attribute information, the image determination of the detection region is determined to be abnormal (abnormal detection). When it is determined that the detection area determination unit 314 is abnormal (step S39: Yes), the process proceeds to step S40, and when it is determined normal (step S39: No), the process returns to step S37. Note that, when the screen determination method of the attribute information of the detection area is “mismatch”, the detection area determination unit 314 compares the detection reference image with the image of the detection area, and the pixel value is the attribute information of the detection area. When the difference, which is the total number of pixels that differ by more than the sensitivity, is less than the threshold value of the attribute information, the image determination of the detection area is determined to be abnormal (abnormal detection).

<Step S40>
The external output unit 316 outputs an abnormal signal to the external device 10 when the detection region determination unit 314 determines abnormality. Then, the process returns to step S37.

<Step S41>
Among the detection areas specified by the detection area specifying unit 307, the detection area is determined by the detection area determination unit 314 at the future timing from the trigger timing by the trigger generation unit 313. If the determination (detection) timing is reached (step S41: Yes), the process proceeds to step S42. If the timing is not reached (step S41: No), the process proceeds to step S45.

<Step S42>
The detection area determination unit 314 includes an image of a detection area of a frame of video data after a predetermined time (detection delay time) corresponding to the detection area from the time when the trigger generation unit 313 receives the trigger signal corresponding to the trigger area, The image is determined whether or not it is abnormal by comparing with the detection reference image. For example, as illustrated in FIG. 27B, the detection region determination unit 314 performs the detection region 535 at the timing when the image determination is performed in the detection region “K61” (detection region 535) designated by the detection region designation unit 307. Highlight the frame color. Thereby, it is possible to visually grasp when the image is determined by the detection area determination unit 314. Then, the process proceeds to step S43.

<Step S43>
When the screen determination method of the attribute information of the detection area is “match”, the detection area determination unit 314 compares the detection reference image with the image of the detection area and the pixel value is equal to or higher than the sensitivity of the attribute information of the detection area. When the difference that is the total number of different pixels is equal to or greater than the threshold value of the attribute information, the image determination of the detection region is determined to be abnormal (abnormal detection). When it is determined that the detection area determination unit 314 is abnormal (step S43: Yes), the process proceeds to step S44, and when it is determined normal (step S43: No), the process returns to step S41. Note that, when the screen determination method of the attribute information of the detection area is “mismatch”, the detection area determination unit 314 compares the detection reference image with the image of the detection area, and the pixel value is the attribute information of the detection area. When the difference, which is the total number of pixels that differ by more than the sensitivity, is less than the threshold value of the attribute information, the image determination of the detection area is determined to be abnormal (abnormal detection).

<Step S44>
The external output unit 316 outputs an abnormal signal to the external device 10 when the detection region determination unit 314 determines abnormality. Then, the process returns to step S41.

<Step S45>
Among the detection areas designated by the detection area designation unit 307, the detection area is determined by the detection area determination unit 314 at the future timing from the trigger timing by the trigger generation unit 313, and the image determination is still performed. If there is a detection area that has not been subjected to (step S45: Yes), the process returns to step S41, and if not (step S45: No), the image determination operation ends.

  As described above, when the user operates the input device 303, the trigger area designation unit 305 designates and displays the trigger area on the video display unit 502, and the change point detection unit 306 performs the first reproduction at the designated time point. A frame for a predetermined time before and after the imaging time of the frame of the video data reproduced and displayed by the unit 304 is acquired, and a reference image that is an image extracted in the trigger area at the specified time is acquired The difference between the pixel values of the image in the trigger region of the frame is calculated. Furthermore, the change point detection unit 306 has a graph showing the calculated differences in time series, and the number of pixels having pixel values different from each other by a predetermined sensitivity or more with respect to an image in the trigger region of the first frame of the acquired frame. The sum is taken as a difference, and a point in time when the difference is equal to or greater than a predetermined threshold and there is almost no difference compared to the reference image is detected as a change point (IN point). In addition, the change point detection unit 306 is an image in the trigger region of the first frame of the acquired frame in the graph showing the calculated difference in time series, and the image has almost no difference compared to the reference image. If there is a difference from the image in the trigger area of the first frame, the sum of the number of pixels whose pixel values differ by a predetermined sensitivity or more is set as a difference, and the time when the difference becomes a predetermined threshold or more is a change point (OUT point) Detect as. Then, when the change point detected by the change point detection unit 306 is the OUT point, the trigger generation unit 313 has a state in which the image of the trigger region of the frame of the video data has almost no difference compared to the reference image. A trigger signal is generated at a timing when the difference, which is the sum of the number of pixels having pixel values different from each other by a predetermined sensitivity or more, becomes a predetermined threshold or more. In addition, when the change point is the IN point, the trigger generation unit 313 is a difference that is the sum of the number of pixels whose pixel values differ by a predetermined sensitivity or more with respect to the image (difference image) of the trigger region of the video data frame The trigger signal is generated at a timing when becomes equal to or greater than a predetermined threshold and there is almost no difference compared to the reference image. As a result, the trigger timing that is the reference for the timing of the image determination by the detection region determination unit 314 can be freely generated.

  In addition, during the playback and display of the video data on the video display unit 502, the pause button 503b is pressed by the user's operation of the input device 303 at the timing when the portion of the image for which image determination is desired is displayed, and the first playback is performed. The unit 304 pauses the video data being reproduced and displayed on the video display unit 502. And the detection area designation | designated part 307 designates and displays a detection area in the video display part 502 by operation of the input device 303 by a user. Also, the setting unit 308 calculates, as attribute information, the delay time of the frame time when the detection region is specified by the detection region specification unit 307 with respect to the time of the change point detected by the change point detection unit 306. . Then, the detection region determination unit 314 receives the trigger signal generated by the trigger generation unit 313, and before or after the delay time set by the setting unit 308, in the specified detection region, the detection reference image And the image in the detection area of the frame of the video data are compared to determine whether the image is abnormal. This makes it possible not only to perform image determination on the image of the detection area of the frame after a predetermined time, but also to the image of the detection area of the past frame of the predetermined time before, based on the generated trigger timing. The image can be determined. Therefore, the degree of freedom in designing the image determination timing can be dramatically improved.

  In the present embodiment, on the setting screen 500, a trigger area and a detection area are specified in the video display unit 502, and on the main screen 400, a trigger signal is generated based on an image displayed on the video display unit 401. Although generated and image determination is performed, the present invention is not limited to this. That is, the functions of both the setting screen 500 and the main screen 400 are unified, for example, the video display unit 502 and the video display unit 401 are set as the same display unit, the trigger area and the detection area are specified, and the trigger signal. Generation and image determination may be executed. In this case, when the video display unit 502 and the video display unit 401 are the same display unit, the display unit corresponds to both the “first display unit” and the “second display unit”.

  The trigger signal generated by the trigger generation unit 313 is used as a reference for the timing of image determination by the detection region determination unit 314 in the above-described embodiment, but is not limited to this. For example, the trigger signal may be used as a reference for timing at which the production facility performs predetermined processing. In this case, the trigger signal is transmitted to the production facility when the trigger signal is generated. It may be.

[Second Embodiment]
The information processing apparatus 3 according to the present embodiment will be described focusing on differences from the information processing apparatus 3 according to the first embodiment. In particular, the difference between the information processing device 3 of the first embodiment and the information processing device 3 of the first embodiment will be described in detail with respect to the operation of detecting a change point in the trigger region of the information processing device 3 of the present embodiment. The overall configuration of the image processing system 1 of the present embodiment, the hardware configuration and block configuration of the information processing device 3, the configuration of the main screen 400, the configuration of the setting screen 500, the designation of the detection area, and the attribute information of the detection area The setting operation, the sensitivity adjustment operation on the sensitivity setting preview screen 700, and the image determination operation on the main screen 400 are the same as those in the first embodiment.

(Block configuration of information processing device)
As described above, the block configuration of the information processing device 3 of the present embodiment is the same as the block configuration of the information processing device 3 of the first embodiment shown in FIG. explain.

  In the present embodiment, when the trigger region is designated by the trigger region designation unit 305, the change point detection unit 306 captures the imaging time of the frame of the video data reproduced and displayed by the first reproduction unit 304 at the designated time point. Frames from the previous frame to a predetermined time (for example, 3 seconds), and in the acquired frames, the difference between the previous frame and the pixel value is calculated for each frame, and the trigger area 530 is calculated based on the calculated difference. It is a function part which detects the reference | standard image about these images. In addition, the change point detection unit 306 acquires a frame from the time of the detected reference image frame (reference frame) to a predetermined time (for example, 6 seconds) later. A difference (first difference) for the pixel value is calculated, and a change point for the image in the trigger region is detected based on the calculated difference. The changing points in the present embodiment will be described later. As a method of calculating the above-mentioned difference, for example, pixel values in the trigger area of two frames are compared, and the total number of pixels whose pixel values differ by a predetermined sensitivity or more, or among the pixels of the image in the trigger area, pixels A ratio (area ratio) of the number of pixels whose values differ by a predetermined sensitivity or more is calculated as a difference. Also, when the frame is a grayscale image, the pixel value matches the luminance value, so the luminance values in the trigger area of the two frames are compared, and the number of pixels whose luminance values differ by more than a predetermined sensitivity is calculated. It is good. When the frame is an image composed of RGB pixel values, the pixel values of R (red), G (green), and B (blue) are compared, and the pixel value is the same as described above. The number of pixels that differ by more than a predetermined sensitivity may be calculated. Alternatively, edge detection processing may be performed on the frame image, and the sum of the edge portions may be calculated as a difference. The change point detection unit 306 is realized by a program executed by the CPU 101 shown in FIG.

(Specify trigger area and detect change point)
FIG. 28 is a diagram illustrating an example of a dialog displayed when a reference image in the trigger area is not found on the setting screen displayed on the display device of the information processing device according to the second embodiment. FIG. 29 is a diagram illustrating an operation of detecting a change point in the trigger area according to the second embodiment. With reference to FIGS. 28 and 29 and the drawings in the first embodiment, specification of a trigger region, detection of a reference image, and detection of a change point will be described.

  First, as shown in FIG. 6, during the reproduction and display of video data on the video display unit 502, the user operates the input device 303 at a timing when a desired image portion that is the basis of generation of a trigger is displayed. When the pause button 503b is pressed, the first reproduction unit 304 pauses the video data reproduced and displayed on the video display unit 502, and the time display unit 506 displays the imaging date and time of the current frame. Display. Next, when the user operates the input device 303 (for example, dragging the mouse 108), the trigger area specifying unit 305 specifies and displays the trigger area 530 on the video display unit 502 (first display area).

  Next, when the trigger area 530 is designated by the trigger area designation unit 305, the change point detection unit 306 is predetermined from the imaging time of the frame of the video data that is reproduced and displayed by the first reproduction unit 304 at the designated time. Frames up to a time (for example, 3 seconds) (first predetermined time) are acquired. Then, the change point detection unit 306 calculates, for each acquired frame, a difference in pixel value from the immediately preceding frame for each frame, and detects a reference image for an image in the trigger region 530 based on the calculated difference. To do. Specifically, in the acquired frame, the change point detection unit 306 has a difference between the pixel value of the image in the trigger area 530 of the target frame and the pixel value of the image in the trigger area 530 of the frame immediately before the frame. The ratio (area ratio) of the number of pixels that differ by a predetermined sensitivity (for example, “30”) or more is calculated as a difference, and when the difference falls below a predetermined threshold (for example, “5%”), The image of the trigger area 530 is detected as a reference image.

  Note that the specific values of the predetermined time, the predetermined sensitivity, and the predetermined threshold used for detecting the above-described reference image are examples, and other values may be used, or they may be stored in the storage unit 302 as a setting file and edited. It may be possible. Further, as described above, the change point detection unit 306 acquires frames from the imaging time of the frame of the video data reproduced and displayed by the first reproduction unit 304 to a predetermined time before, but is not limited thereto. Is not to be done. For example, the change point detection unit 306 acquires a frame for a predetermined time before and after the imaging time of a frame of video data reproduced and displayed by the first playback unit 304 or a frame from the imaging time to a predetermined time later. May be.

  In the setting screen 500 shown in FIG. 28, when the reference image is not detected in the trigger area 530 specified by the trigger area specifying unit 305 by the change point detection unit 306, a notification dialog 620 indicating no reference image is displayed. Is shown. In the above-described reference image detection operation, the change point detection unit 306 may not be able to detect the reference image because the calculated difference does not exceed a predetermined threshold value. In this case, the change point detection unit 306 causes the display device 317 to display a reference image absence notification dialog 620 indicating that the reference image could not be detected.

  The reference image absence notification dialog 620 includes an OK button 621 and a close button 622. The OK button 621 is a button for closing the reference image absence notification dialog 620. When the user presses the OK button 621 by operating the input device 303, the change point detection unit 306 does not store any information in the storage unit 302, and the image is also displayed in the reference image display unit 508a and the difference image display unit 508b. Is not displayed, and the reference image absence notification dialog 620 is closed.

  Next, when detecting the reference image, the change point detection unit 306 acquires frames from the imaging time of the reference frame that is a frame of the reference image to a predetermined time (for example, 6 seconds) (second predetermined time). . Then, the change point detection unit 306 calculates a difference (first difference) about the pixel value with respect to the reference frame for each frame in the acquired frame, and changes about the image in the trigger region based on the calculated difference. Detect points. Here, as a difference calculation method, for example, in the frame acquired by the change point detection unit 306, the pixel value of the image in the trigger area 530 of the target frame and the pixel value of the image in the trigger area 530 of the reference frame are calculated. The difference is the ratio (area ratio) of the number of pixels that differ by at least a predetermined sensitivity (eg, “50”). Hereinafter, a specific method for detecting change points by the change point detection unit 306 will be described.

  As shown in FIG. 29A, the change point detection unit 306 acquires 13 frames (frame (0) to frame (12)) as frames for a predetermined time from the imaging time of the reference frame. To do. In this case, the change point detection unit 306 acquires a frame for a predetermined time from the frame (0) as a reference frame. In addition, in order to simplify the description, the image of the part corresponding to the trigger region 530 in each frame is also referred to as frame (0) to frame (12), respectively. Then, the change point detection unit 306 calculates a difference in pixel values from the reference frame for each frame in the acquired frame. The difference calculation method is as described above. The graph shown in FIG. 29B is a graph in which the change point detection unit 306 extracts the frames exemplified in FIG. 29A and expresses the differences calculated for each frame in time series. In FIG. 29B, for example, the difference between the frame (2) indicates the difference between the frame (2) and the frame (0) that is the reference frame. Here, the IN point and the OUT point as changing points of the present embodiment are defined as follows. That is, the IN point is a point in time when the image in the trigger area 530 is changed by a predetermined degree in the consecutive frames as compared with the image in the trigger area 530 of the reference frame. The OUT point is a point in time when an image in the trigger area 530 approximates an image in the trigger area 530 of the reference frame in successive frames from the IN point.

  The change point detection unit 306 finds an IN point candidate and an OUT point candidate based on the difference calculated for each frame. Specifically, the change point detection unit 306 responds to a difference exceeding the threshold when the calculated difference exceeds the threshold from a state equal to or less than a predetermined threshold (for example, 20%) (first threshold). The frame to be used is the IN point candidate frame. In the example of FIG. 29B, the difference between the frames (1) to (3) is less than the threshold, and the difference exceeds the threshold when the frame (4) is reached. Become. Similarly, the frame (10) is also a candidate frame for the IN point.

  On the other hand, the change point detection unit 306, when the calculated difference falls below the threshold value from a state larger than the predetermined threshold value (first threshold value), the frame corresponding to the difference below the threshold value is selected as an OUT point candidate. Frame. In the example of FIG. 29 (b), the difference between the frames (4) to (6) is larger than the threshold, and when the frame (7) is reached, the difference is equal to or less than the threshold. Become. Similarly, the frame (11) is also a candidate frame for the OUT point.

  As described above, in the example of FIG. 29, the change point detection unit 306 identifies the frames (4) and (10) as the IN point candidate frames and the frames (7) and (10) as the OUT point candidate frames. 11) is specified. As described above, when a plurality of IN point candidate frames are specified, the change point detection unit 306 determines the frame closest to the reference frame, that is, the frame (4) as the IN point frame. Similarly, when a plurality of OUT point candidate frames are specified, the change point detection unit 306 determines the frame closest to the reference frame, that is, the frame (7) as the OUT point frame. As described above, the change point detection unit 306 detects the change points (IN point and OUT point) by determining the IN point frame and the OUT point frame from the acquired frames for a predetermined time. When the change point is detected, the change point detection unit 306 detects a moving object in the trigger region 530 using a frame before the change point frame, a frame after the change point frame, or a frame before and after the change point frame. Vector information indicating whether a moving object has entered from the storage unit 302 may be stored in the storage unit 302. In this case, the trigger generation unit 313 uses the vector information stored in the storage unit 302 to check whether the moving object has moved from the same direction as the vector information, and triggers based on the check result. A signal may be generated.

  FIG. 29 shows an example in which both the IN point and OUT point frames are found. However, the change point detection unit 306 may find only the IN point frame or only the OUT point frame. In this case, these IN point or OUT point is detected as a change point. In this case, the change point detection unit 306 extracts an image in the trigger region 530 of the detected change point frame as a difference image (first image). The change point detection unit 306 stores the extracted reference image and the difference image in the frame of the change point in the storage unit 302, displays the reference image on the reference image display unit 508a, and displays the difference image on the difference image display unit 508b. Let Furthermore, the change point detection unit 306 displays a trigger mark 540 on the seek bar 504 at a portion corresponding to the time of the detected change point frame. When a change point is detected by the change point detection unit 306, the trigger area specifying unit 305 causes the storage unit 302 to store information on the specified trigger area 530. In addition, the setting unit 308 has a name given to the trigger region 530 specified by the trigger region specifying unit 305, a predetermined sensitivity (for example, “50”) used for detection of a trigger by the trigger generation unit 313, and a predetermined A threshold (for example, “20%”) (default value) is displayed on the trigger area information unit 507. As shown in FIG. 7 described above, the setting unit 308 assumes that the trigger area 530 is a trigger area designated for the video data of the imaging device (imaging device 2f) of “6”, for example. It is named “TR6”.

  The specific values of the predetermined time, the predetermined sensitivity, and the predetermined threshold used for detecting the above-described change point are examples, and may be other values, or may be stored in the storage unit 302 as a setting file and edited. It may be possible.

  In addition, when the change point detection unit 306 detects two change points (IN point and OUT point) in the trigger region 530 designated by the trigger region designation unit 305 as described above, the change point detection unit 306 is illustrated in FIG. 9 described above. The operation is the same as that of the change point selection dialog 600. In this way, the change point detection unit 306 detects the IN point and the OUT point as change points, and allows the user to select one of the two change points, so that the user can truly specify the change point. Can be selected.

  Further, when the change point detection unit 306 does not find a change point in the trigger region 530 specified by the trigger region specifying unit 305, the operation is the same as that performed by the above-described no change point notification dialog 610 shown in FIG. .

(Flow of setting operation to specify trigger area and detection area)
FIG. 30 is a flowchart illustrating an example of a setting operation for designating a trigger area and a detection area on a setting screen displayed on the display device of the information processing apparatus according to the second embodiment. With reference to FIG. 30, the flow of the setting operation for designating the trigger area and the detection area on the setting screen 500 will be generally described.

<Step S51>
The first playback unit 304 acquires video data from the storage unit 302 according to the operation of the input device 303 by the user, and causes the video display unit 502 to play back and display the video data. Then, the process proceeds to step S52.

<Step S52>
When the user presses the pause button 503b by the operation of the input device 303 at the timing when the desired image portion that is the basis of the occurrence of the trigger is displayed while the video data is being reproduced and displayed on the video display unit 502. The first playback unit 304 pauses the video data that is played back and displayed on the video display unit 502. When the user operates the input device 303 (for example, dragging the mouse 108), the trigger area designating unit 305 designates and displays the trigger area on the video display unit 502. Then, the process proceeds to step S53.

<Step S53>
When the trigger area 530 is designated by the trigger area designation unit 305, the change point detection unit 306 is a predetermined time (for example, from the imaging time of the frame of the video data reproduced and displayed by the first reproduction unit 304 at the designated time point) 3 seconds) The previous frame is acquired. Then, the change point detection unit 306 calculates, for each acquired frame, a difference in pixel value from the immediately preceding frame for each frame, and detects a reference image for an image in the trigger region 530 based on the calculated difference. To do. A specific reference image detection method is as described above. Then, the process proceeds to step S54.

<Step S54>
When the reference image is detected by the change point detection unit 306 (step S54: Yes), the process proceeds to step S55, and when the reference image is not detected (step S54: No), the process returns to step S51.

<Step S55>
When detecting the reference image, the change point detection unit 306 acquires frames from the imaging time of the reference frame to a predetermined time (for example, 6 seconds). Then, the change point detection unit 306 calculates a difference for the pixel value from the reference frame for each frame in the acquired frame, and detects a change point for the image in the trigger region based on the calculated difference. The specific change point detection method is as described above with reference to FIGS. Then, the process proceeds to step S56.

<Step S56>
When the change point is detected by the change point detection unit 306 (step S56: Yes), the process proceeds to step S57, and when the change point is not detected (step S56: No), the process returns to step S51.

<Steps S57 to 61>
The processing in steps S57 to 61 is the same as the processing in steps S15 to 19 shown in FIG. 24 in the first embodiment.

  As described above, the change point detection unit 306 first detects the reference image, and when the reference image is detected, acquires a frame from the imaging time of the reference frame to a predetermined time later, and the reference frame for each frame. And a change point for an image in the trigger region is detected based on the calculated difference. Based on the change point detected by such a method, the trigger generation unit 313 generates a trigger signal. As a result, the trigger timing that is the reference for the timing of the image determination by the detection region determination unit 314 can be freely generated. In addition, the same effects as those of the first embodiment can be obtained.

  In addition, the first reproduction unit 304, the trigger region designation unit 305, the change point detection unit 306, the detection region designation unit 307, the setting unit 308, the second reproduction unit 309, the image extraction unit 310, and the difference determination of each embodiment described above. When at least one of the unit 311, the video distribution unit 312, the trigger generation unit 313, the detection area determination unit 314, and the display control unit 315 is realized by executing a program, the program is provided in advance in a ROM or the like Is done. The program executed by the information processing apparatus 3 according to the present embodiment is an installable or executable file and is read by a computer such as a CD-ROM, flexible disk (FD), CD-R, or DVD. You may comprise so that it may record on a possible recording medium and provide. The program executed by the information processing apparatus 3 according to the present embodiment may be configured to be stored by being stored on a computer connected to a network such as the Internet and downloaded via the network. Further, the program executed by the information processing apparatus 3 of the present embodiment may be configured to be provided or distributed via a network such as the Internet. The program executed by the information processing apparatus 3 according to the present embodiment includes the first reproduction unit 304, the trigger region designation unit 305, the change point detection unit 306, the detection region designation unit 307, the setting unit 308, and the second reproduction unit described above. 309, an image extraction unit 310, a difference determination unit 311, a video distribution unit 312, a trigger generation unit 313, a detection region determination unit 314, and a display control unit 315. As described above, the CPU reads out the program from the ROM and executes the program, so that each unit described above is loaded on the main storage device and generated.

DESCRIPTION OF SYMBOLS 1 Image processing system 2, 2a-2f Image pick-up device 3 Information processing device 4 Network 10 External apparatus 101 CPU
102 ROM
103 RAM
104 External storage device 105 Display 106 Network I / F
107 keyboard 108 mouse 109 DVD drive 110 DVD
111 External device I / F
DESCRIPTION OF SYMBOLS 112 Bus 301 Image | video receiving part 302 Memory | storage part 303 Input device 304 1st reproduction | regeneration part 305 Trigger area | region designation | designated part 306 Change point detection part 307 Detection area | region designation | designated part 308 Setting part 309 2nd reproduction | regeneration part 310 Image extraction part 311 Difference determination part 312 Image | video Distribution unit 313 Trigger generation unit 314 Detection area determination unit 315 Display control unit 316 External output unit 317 Display device 400 Main screen 401, 401a to 401f Video display unit 411 Monitoring start button 412 Monitoring stop button 413 Monitoring setting button 500 Setting screen 501 Camera selection unit 502 Video display unit 503a Play button 503b Pause button 503c Fast forward button 503d Fast reverse button 503e Stop button 504 Seek bar 505 Slider 506 Time display unit 507 Trigger area information unit 50 7a Pull-down button 507b Pull-down menu 508a Reference image display section 508b Difference image display section 509 Detection area information section 510 Detection area button section 510a Action button 510b Image determination selection button 510c Delete button 510d Sensitivity determination button 510e Sensitivity increase button 510f Sensitivity decrease button 510g Threshold increase button 510h Threshold decrease button 511 Setting read button 512 Setting write button 513 Setting reflection button 514 Close button 530, 530a Trigger area 535, 536, 536a, 537 Detection area 540, 540a Trigger mark 600 Change point selection dialog 601 Yes button 602 No button 603 Close button 610 No change point notification dialog 611 OK button 612 Close button 62 Reference image absence notification dialog 621 OK button 622 Close button 700 Sensitivity setting preview screen 701 Detection area information section 702 Video display section 703 Detection reference image display section 704 Extracted image display section 705 Difference display section 706 Color detection number display section 707 Area pixel Number display part 708 Difference detection number display part 709 Difference detection ratio display part 710a Sensitivity increase button 710b Sensitivity decrease button 711 Determination button 712 OK button 713 Close button 800 Sensitivity application confirmation dialog 801 Yes button 802 No button 803 Cancel button 804 Close button

JP 07-248214 A

Claims (22)

Display means;
Reproduction means for displaying the video data imaged by the imaging means on the display means;
Trigger area specifying means for specifying a trigger area in the first display area of the display means on which the video data is displayed;
A change point in the image in the trigger area is detected from the video data for a first predetermined time with reference to a time point when the trigger area is specified by the trigger area specifying means, and the trigger area of the frame of the change point is detected. Detecting means for extracting the image inside as the first image;
The image in the area corresponding to the trigger area in the second display area of the display means on which the video data imaged by the imaging means is displayed is compared with the first image, and a trigger is generated according to the comparison result Generating means to
An image processing apparatus. The detection means includes
Obtaining a frame of the video data for the first predetermined time, calculating a difference from the previous frame for each of the obtained frames, and detecting a reference image in the trigger region based on the difference;
A frame of the video data from a time point of a frame including the reference image to a second predetermined time later is acquired, a first difference from the frame including the reference image is calculated for each of the acquired frames, and the first The image processing apparatus according to claim 1, wherein the change point in the image in the trigger region is detected based on the difference.   The detection means may determine a time point of a frame when the first difference exceeds a first threshold value, or a time point of a frame where the first difference falls below the first threshold value from a time point of a frame where the first difference exceeds the first threshold value. The image processing apparatus according to claim 2, wherein the image processing apparatus detects the change point. The detecting means acquires frames of the video data for the first predetermined time before and after the time when the trigger area is specified by the trigger area specifying means, and for each frame, a first frame with a previous frame is acquired. The image processing apparatus according to claim 1, wherein a difference is calculated, and the change point in the image in the trigger region is detected based on the first difference.   The image processing apparatus according to claim 4, wherein the detection unit detects the change point based on a comparison result between the first difference and a first threshold value.   The image processing apparatus according to claim 1, wherein the generation unit generates the trigger that serves as a reference for image determination in the second display region. The generating means calculates a second difference between an image in an area corresponding to the trigger area in the second display area and the first image, and the second difference is less than a second threshold value. The image processing apparatus according to claim 1, wherein the trigger is generated. In the first display region, a detection region designation means for designating a detection region to be determined images,
Detection area determination means for performing the image determination on an area corresponding to the detection area in the second display area ;
A setting unit for setting a determination time for determining a time point at which the image determination is performed by the detection region determination unit with respect to a time point when the generation unit generates the trigger in an area corresponding to the detection region;
The image processing apparatus according to claim 1, further comprising: The said setting means sets the said determination time so that the said image determination may be performed with respect to the area | region corresponding to the said detection area | region of the flame | frame of the video data before the time when the said production | generation means produced | generated the said trigger. The image processing apparatus according to 8. The said setting means sets the said determination time so that the said image determination may be performed with respect to the area | region corresponding to the said detection area of the flame | frame of the video data after the time when the said production | generation means produced | generated the said trigger. The image processing apparatus according to 8. The trigger area designating unit designates a trigger area different from the trigger area in the first display area for video data captured by an imaging unit different from the imaging unit,
The detection area determination unit performs the image determination on an area corresponding to the detection area in the second display area based on a time point when the trigger for the other trigger area is generated by the generation unit. The image processing apparatus according to any one of 8 to 10.   The image processing apparatus according to any one of claims 8 to 11, wherein the detection area designating unit designates a plurality of the detection areas in the first display area. The detection area determination means includes a second image that is an image in the detection area when specified by the detection area specification means, and an image of an area corresponding to the detection area of the frame at the determination time. The image processing apparatus according to claim 8, wherein three image differences are calculated, and an abnormality is determined when the third difference is equal to or greater than a third threshold value. The detection area determination means includes a second image that is an image in the detection area when specified by the detection area specification means, and an image of an area corresponding to the detection area of the frame at the determination time. The image processing apparatus according to any one of claims 8 to 12, wherein three image differences are calculated, and an abnormality is determined when the third difference is less than a third threshold value. The detection area determination means calculates the third difference based on the number of pixels in which pixel values of the second image and an image of an area corresponding to the detection area of the frame at the determination time differ by a predetermined sensitivity or more. Operate,
The image processing apparatus according to claim 13, wherein the setting unit sets the predetermined sensitivity.   The image processing apparatus according to claim 8, wherein the detection area designating unit designates the detection area based on the video data displayed by the reproduction unit.   The image processing apparatus according to any one of claims 8 to 16, further comprising an external output unit that outputs an abnormality to an external device when the detection region determination unit determines that an abnormality has occurred in the image determination. . Said detecting means, as the first differential, the image processing apparatus according to any one of claims 2 to 5 for calculating a difference between the pixel values relating to the luminance value or a predetermined color.   The image processing apparatus according to any one of claims 1 to 18, wherein the trigger area designating unit designates the trigger area based on the video data displayed by the reproduction unit. One or more imaging means;
An image processing device according to any one of claims 1 to 19,
An image processing system. A reproduction step of causing the display means to display the video data imaged by the imaging means;
A trigger area designating step for designating a trigger area in the first display area of the display means on which the video data is displayed;
A change point in the image in the trigger area is detected from the video data for a predetermined time with reference to the time point when the trigger area is specified, and an image in the trigger area of the frame of the change point is extracted as a difference image. A detection step;
An image in an area corresponding to the trigger area in the second display area of the display means on which video data captured by the imaging means is displayed is compared with the difference image, and a trigger is generated according to the comparison result. Generation step;
An image processing method. On the computer,
A reproduction process for causing the display means to display the video data imaged by the imaging means;
A trigger area designating process for designating a trigger area in the first display area of the display means on which the video data is displayed;
A change point in the image in the trigger area is detected from the video data for a predetermined time with reference to the time point when the trigger area is specified, and an image in the trigger area of the frame of the change point is extracted as a difference image. Detection process,
An image in an area corresponding to the trigger area in the second display area of the display means on which video data captured by the imaging means is displayed is compared with the difference image, and a trigger is generated according to the comparison result. Generation process,
A program that executes