CN111191656A - Behavior identification method and system based on multispectral image information - Google Patents
Behavior identification method and system based on multispectral image information Download PDFInfo
- Publication number
- CN111191656A CN111191656A CN201911318478.8A CN201911318478A CN111191656A CN 111191656 A CN111191656 A CN 111191656A CN 201911318478 A CN201911318478 A CN 201911318478A CN 111191656 A CN111191656 A CN 111191656A
- Authority
- CN
- China
- Prior art keywords
- image
- infrared
- candidate
- resolution processing
- infrared image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 56
- 238000002329 infrared spectrum Methods 0.000 claims abstract description 64
- 238000012545 processing Methods 0.000 claims abstract description 62
- 230000003595 spectral effect Effects 0.000 claims abstract description 55
- 238000001429 visible spectrum Methods 0.000 claims abstract description 27
- 231100001267 hazard identification Toxicity 0.000 claims abstract description 7
- 230000004044 response Effects 0.000 claims abstract description 4
- 238000001914 filtration Methods 0.000 claims description 18
- 230000008569 process Effects 0.000 claims description 7
- 230000008859 change Effects 0.000 claims description 6
- 230000006870 function Effects 0.000 description 37
- 230000006399 behavior Effects 0.000 description 15
- 238000012544 monitoring process Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 8
- 238000007689 inspection Methods 0.000 description 8
- 238000004590 computer program Methods 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 4
- 239000000725 suspension Substances 0.000 description 4
- 230000007547 defect Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 239000003245 coal Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000009529 body temperature measurement Methods 0.000 description 1
- 230000007717 exclusion Effects 0.000 description 1
- 238000010438 heat treatment Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000003331 infrared imaging Methods 0.000 description 1
- 239000012212 insulator Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000005855 radiation Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000011282 treatment Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/403—Edge-driven scaling; Edge-based scaling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Psychiatry (AREA)
- Social Psychology (AREA)
- Human Computer Interaction (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a behavior identification method based on multispectral image information, which comprises the following steps: acquiring a visible spectrum image and an infrared spectrum image by a local terminal; the local terminal performs high-resolution processing on the infrared spectrum image and sends the infrared spectrum image subjected to the high-resolution processing to the server, or the local terminal sends the infrared spectrum image to the server so that the server performs the high-resolution processing on the infrared spectrum image; identifying, by the server, the object in the image based on the high-resolution processed infrared spectral image and the visible spectral image; performing, by the server, hazard identification based on the identified object; when a dangerous condition occurs, the server sends out early warning to an alarm center; in response to receiving the early warning, the warning center gives an alarm; wherein the high resolution processing of the infrared spectral image comprises at least the step of comparing the visible spectral image with the interpolated infrared spectral image.
Description
Technical Field
The invention relates to the technical field of infrared image processing and identification, in particular to a behavior identification method and system based on multispectral image information.
Background
The infrared imaging technology is a high and new technology with a wide prospect. Electromagnetic waves longer than 0.78 microns are outside the red color of the visible spectrum and are called infrared, also known as infrared radiation. It means an electromagnetic wave having a wavelength of 0.78 to 1000 μm, wherein a portion having a wavelength of 0.78 to 2.0 μm is called near infrared and a portion having a wavelength of 2.0 to 1000 μm is called thermal infrared.
The prior art CN109034272A discloses the technical field of power inspection, in particular relates to the infrared temperature measurement compensation technology of power equipment, and specifically relates to an automatic identification method for a heating component of a power transmission line. The identification method comprises the following steps: selecting a real-time infrared video sequence shot by helicopter power inspection, carrying out Hough transformation on the real-time infrared video sequence, and detecting a power transmission line to identify an insulator in an infrared image by SIFT feature matching; and segmenting the hot spot region in the infrared image by adopting an Otsu self-adaptive threshold algorithm, extracting a defect region, and classifying and grading the defect region.
The prior art CN107043000B discloses a belt conveyor safety intelligent guarantee system based on machine vision, which comprises a giant foreign matter identification system, a fixed suspension type robot monitoring system consisting of a fixed suspension type robot and a foreign matter identification module; the coal flow monitoring system adopts a suspension type track inspection robot monitoring system consisting of a suspension type track inspection robot and a coal flow monitoring module; the carrier roller temperature monitoring system adopts a track inspection robot monitoring system consisting of a carrier roller temperature monitoring module based on infrared thermal imagery and a track inspection robot; the belt monitoring system adopts a belt damage monitoring module based on machine vision and a suspended track inspection robot monitoring system formed by a suspended track inspection robot.
The information disclosed in this background section is only for enhancement of understanding of the general background of the invention and should not be taken as an acknowledgement or any form of suggestion that this information forms the prior art already known to a person skilled in the art.
Disclosure of Invention
The invention aims to provide a behavior identification method and system based on multispectral image information, which can overcome the defects of the prior art.
In order to achieve the above object, the present invention provides a behavior recognition method based on multispectral image information, which is characterized in that: the behavior identification method based on the multispectral image information comprises the following steps:
acquiring a visible spectrum image and an infrared spectrum image by a local terminal;
the local terminal performs high-resolution processing on the infrared spectrum image and sends the infrared spectrum image subjected to the high-resolution processing to the server, or the local terminal sends the infrared spectrum image to the server so that the server performs the high-resolution processing on the infrared spectrum image;
identifying, by the server, the object in the image based on the high-resolution processed infrared spectral image and the visible spectral image;
performing, by the server, hazard identification based on the identified object;
when a dangerous condition occurs, the server sends out early warning to an alarm center;
in response to receiving the early warning, the warning center gives an alarm;
wherein the high resolution processing of the infrared spectral image comprises at least the step of comparing the visible spectral image with the interpolated infrared spectral image.
In a preferred embodiment, the high resolution processing of the infrared spectral image comprises the following steps:
utilizing a bicubic interpolation method to perform upsampling on the infrared spectrum image so as to enable the infrared spectrum image and the visible spectrum image to have the same size;
taking the up-sampled infrared spectrum image as an initial infrared image;
s1: establishing a cost function based on the initial infrared image or the candidate infrared image, wherein the cost function is as follows:
wherein,
is a function of the cost of the received signal,
is a custom constant value, d is a candidate infrared image,
is the infrared image resulting from the current iteration, and wherein, when i =0,
is the initial infrared image.
In a preferred embodiment, the high resolution processing of the infrared spectral image comprises the following steps:
s2: obtaining an image of a red channel in the visible spectrum image based on the visible spectrum image;
s3: setting a correlation threshold value;
s4: in a given region of the image, calculating the cross correlation between a plurality of pixel points of the image of the red channel and a plurality of pixel points of the initial infrared image or a plurality of pixel points of the candidate infrared image;
s5: and if the cross correlation between the multiple pixel points of the image of the red channel and the multiple pixel points of the initial infrared image or the multiple pixel points of the candidate infrared image is greater than the correlation threshold value, determining that the given area of the image is a cross correlation area.
In a preferred embodiment, the high resolution processing of the infrared spectral image comprises the following steps:
s6: filtering a cost function in the cross-correlation, wherein filtering the cost function in the cross-correlation is based on the following formula:
wherein,
is a function of the cost of performing the filtering process,
is a custom window, wherein the size of the custom window is smaller than the size of the cross-correlation region,
is an image of the red channel, where,
and
defined by the following formula;
wherein,
is the initial infrared image or the candidate infrared image
The average value of (a) is,
is the initial infrared image or the candidate infrared image
The variance in (a) is greater than or equal to,
is a constant value that is self-defined,
is defined by the following equation:
wherein,
is the total number of pixel points in the user-defined window;
s7: by making
And minimizing the function to obtain the updated candidate infrared image d.
In a preferred embodiment, the high resolution processing of the infrared spectral image comprises the following steps:
s8: performing sub-pixel level interpolation on the updated candidate infrared image d to obtain a finished candidate infrared image
The sub-pixel-level interpolation of the updated candidate infrared image d is specifically performed by the following method:
wherein,
，
and wherein the first and second end portions of the first and second,
s9: to complete candidate infrared images
Replacing the initial infrared image or the candidate infrared image;
and (5) iterating steps S1-S9 until the change of the cost function reaches a convergence standard, thereby obtaining the infrared image processed by high resolution.
The invention provides a behavior recognition system based on multispectral image information, which is characterized in that: wherein the behavior recognition system based on the multispectral image information is completed at the server, wherein the behavior recognition system based on the multispectral image information comprises:
means for identifying an object in an image based on a high resolution processed infrared spectral image and a visible spectral image, wherein the visible spectral image and the infrared spectral image are acquired by a local terminal, and wherein the local terminal is configured to: carrying out high-resolution processing on the infrared spectrum image and sending the infrared spectrum image subjected to the high-resolution processing to a server, or sending the infrared spectrum image to the server so as to carry out the high-resolution processing on the infrared spectrum image by the server;
means for performing hazard identification based on the identified object;
the module is used for sending out early warning to the alarm center when a dangerous condition occurs;
wherein the high resolution processing of the infrared spectral image comprises at least the step of comparing the visible spectral image with the interpolated infrared spectral image.
In a preferred embodiment, the high resolution processing of the infrared spectral image comprises the following steps:
utilizing a bicubic interpolation method to perform upsampling on the infrared spectrum image so as to enable the infrared spectrum image and the visible spectrum image to have the same size;
taking the up-sampled infrared spectrum image as an initial infrared image;
s1: establishing a cost function based on the initial infrared image or the candidate infrared image, wherein the cost function is as follows:
wherein,
is a function of the cost of the received signal,
is a custom constant value, d is a candidate infrared image,
the infrared image resulting from the current iteration, and wherein, when i =0,
is the initial infrared image.
In a preferred embodiment, the high resolution processing of the infrared spectral image comprises the following steps:
s2: obtaining an image of a red channel in the visible spectrum image based on the visible spectrum image;
s3: setting a correlation threshold value;
s4: in a given region of the image, calculating the cross correlation between a plurality of pixel points of the image of the red channel and a plurality of pixel points of the initial infrared image or a plurality of pixel points of the candidate infrared image;
s5: and if the cross correlation between the multiple pixel points of the image of the red channel and the multiple pixel points of the initial infrared image or the multiple pixel points of the candidate infrared image is greater than the correlation threshold value, determining that the given area of the image is a cross correlation area.
In a preferred embodiment, the high resolution processing of the infrared spectral image comprises the following steps:
s6: filtering a cost function in the cross-correlation, wherein filtering the cost function in the cross-correlation is based on the following formula:
wherein,
is a function of the cost of performing the filtering process,
is a custom window, wherein the size of the custom window is smaller than the size of the cross-correlation region,
is an image of the red channel, where,
and
defined by the following formula;
wherein,
is the initial infrared image or the candidate infrared image
The average value of (a) is,
is the initial infrared image or the candidate infrared image
The variance in (a) is greater than or equal to,
is a custom constant value，
Is defined by the following equation:
wherein,
is the total number of pixel points in the user-defined window;
s7: by making
And minimizing the function to obtain the updated candidate infrared image d.
In a preferred embodiment, the high resolution processing of the infrared spectral image comprises the following steps:
s8: performing sub-pixel level interpolation on the updated candidate infrared image d to obtain a finished candidate infrared image
The sub-pixel-level interpolation of the updated candidate infrared image d is specifically performed by the following method:
wherein,
，
and wherein the first and second end portions of the first and second,
s9: to finishCandidate infrared image
Replacing the initial infrared image or the candidate infrared image;
and (5) iterating steps S1-S9 until the change of the cost function reaches a convergence standard, thereby obtaining the infrared image processed by high resolution.
Compared with the prior art, the invention has the following advantages that due to the limitation of hardware technology, the size of the sensor of most current infrared photographic devices is significantly smaller than that of the sensor of visible photographic equipment, which means that the resolution of the visible photographic equipment can be 3-4 times higher than that of the infrared equipment at the same price, and no matter whether the visible photographic equipment is identified by human eyes or applied to machine vision, the high resolution and definition of images are prerequisites, and the problem of information loss in low-resolution images cannot be well solved in the world. At present, high-resolution infrared images can be obtained by increasing the purchase cost of equipment, but the equipment cost is strictly controlled in many application occasions, and the compromise between the cost and the image quality is a technical problem. The method for simply increasing the nominal resolution of the image (i.e. although the resolution of the image is increased, the amount of information carried by the image is not changed) includes an interpolation method, and this method cannot obtain an image with a larger amount of information, and the unchanged amount of information means that the image recognition effect is not changed, so the method for simply increasing the nominal resolution of the image is not a method for really increasing the resolution of the image. The invention provides a behavior identification method based on multispectral image information, and the method integrates an interpolation method and a cost function method through a mode of comparing the difference value of a low-resolution infrared image and a high-resolution visible light image, and obtains the high-resolution infrared image with clearer edges and sharper images.
Drawings
Fig. 1 is a flowchart of a method of behavior recognition based on multispectral image information according to an embodiment of the present invention.
FIG. 2 is a flow chart of a method for high resolution processing of infrared spectral images in accordance with an embodiment of the present invention.
FIG. 3 is a photograph of an untreated visible spectrum according to one embodiment of the present invention.
FIG. 4 is a photograph of an infrared spectrum having undergone high resolution processing in accordance with one embodiment of the present invention.
FIG. 5 is a photograph of an infrared spectrum subjected to bicubic interpolation according to an embodiment of the present invention.
FIG. 6 is a photograph of an untreated visible spectrum according to one embodiment of the present invention.
FIG. 7 is a photograph of an infrared spectrum having undergone high resolution processing in accordance with one embodiment of the present invention.
FIG. 8 is a photograph of an infrared spectrum subjected to bicubic interpolation according to an embodiment of the present invention.
Detailed Description
The following detailed description of the present invention is provided in conjunction with the accompanying drawings, but it should be understood that the scope of the present invention is not limited to the specific embodiments.
Throughout the specification and claims, unless explicitly stated otherwise, the word "comprise", or variations such as "comprises" or "comprising", will be understood to imply the inclusion of a stated element or component but not the exclusion of any other element or component.
Fig. 1 is a flowchart of a method of behavior recognition based on multispectral image information according to an embodiment of the present invention. As shown in the figure, the behavior identification method based on multispectral image information includes the following steps:
step 101: acquiring a visible spectrum image and an infrared spectrum image by a local terminal;
step 102: the local terminal performs high-resolution processing on the infrared spectrum image and sends the infrared spectrum image subjected to the high-resolution processing to the server, or the local terminal sends the infrared spectrum image to the server so that the server performs the high-resolution processing on the infrared spectrum image;
step 103: identifying, by the server, the object in the image based on the high-resolution processed infrared spectral image and the visible spectral image;
step 104: performing, by the server, hazard identification based on the identified object;
step 105: when a dangerous condition occurs, the server sends out early warning to an alarm center;
step 106: in response to receiving the early warning, the warning center gives an alarm;
the high-resolution processing of the infrared spectrum image at least comprises a step of comparing the visible spectrum image with the infrared spectrum image subjected to interpolation, wherein the risk identification method adopted by the invention can be a method in the prior art, and the method in the prior art can be, for example, a method adopted by a document in the background art of the application, and the details are not repeated in the application.
In a preferred embodiment, the high resolution processing of the infrared spectral image comprises the following steps:
up-sampling the infrared spectrum image by using a bicubic interpolation (bicubic interpolation) method so as to enable the infrared spectrum image and the visible spectrum image to have the same size;
taking the up-sampled infrared spectrum image as an initial infrared image;
s1: establishing a cost function based on the initial infrared image or the candidate infrared image, wherein the cost function is as follows:
wherein,
is a function of the cost of the received signal,
is a custom constant value (the size of the value can be determined by trial and error, which is not described in detail in this application), d is a candidate infrared image,
is the infrared image resulting from the current iteration, and wherein, when i =0,
is the initial infrared image.
In a preferred embodiment, the high resolution processing of the infrared spectral image comprises the following steps:
s2: obtaining an image of a red channel in the visible spectrum image based on the visible spectrum image;
s3: setting a correlation threshold value;
s4: in a given region of the image, calculating the cross correlation between a plurality of pixel points of the image of the red channel and a plurality of pixel points of the initial infrared image or a plurality of pixel points of the candidate infrared image;
s5: and if the cross correlation between the multiple pixel points of the image of the red channel and the multiple pixel points of the initial infrared image or the multiple pixel points of the candidate infrared image is greater than the correlation threshold value, determining that the given area of the image is a cross correlation area.
In a preferred embodiment, the high resolution processing of the infrared spectral image comprises the following steps:
s6: filtering a cost function in the cross-correlation, wherein filtering the cost function in the cross-correlation is based on the following formula:
wherein,
is a function of the cost of performing the filtering process,
is a self-defined window (the window size can be determined by trial and error, if the window is large, the algorithm is runThe calculation is faster, but the result is coarser, if the window is smaller, the result is finer, but the calculation speed is slower, and the improvement degree of the processing effect is weakened as the window size is reduced, wherein the size of the self-defined window is smaller than the size of the cross-correlation area,
is an image of the red channel, where,
and
defined by the following formula;
wherein,
is the initial infrared image or the candidate infrared image
The average value of (a) is,
is the initial infrared image or the candidate infrared image
The variance in (a) is greater than or equal to,
is a constant value that is self-defined,
is defined by the following equation:
wherein,
is the total number of pixel points in the user-defined window;
s7: by making
And minimizing the function to obtain the updated candidate infrared image d.
In a preferred embodiment, the high resolution processing of the infrared spectral image comprises the following steps:
s8: performing sub-pixel level interpolation on the updated candidate infrared image d to obtain a finished candidate infrared image
The sub-pixel-level interpolation of the updated candidate infrared image d is specifically performed by the following method:
wherein,
，
and wherein the first and second end portions of the first and second,
s9: to complete candidate infrared images
Replacing the initial infrared image or the candidate infrared image;
and (5) iterating steps S1-S9 until the change of the cost function reaches a convergence standard, thereby obtaining the infrared image processed by high resolution.
The invention provides a behavior recognition system based on multispectral image information, which is characterized in that: wherein the behavior recognition system based on the multispectral image information is completed at the server, wherein the behavior recognition system based on the multispectral image information comprises:
means for identifying an object in an image based on a high resolution processed infrared spectral image and a visible spectral image, wherein the visible spectral image and the infrared spectral image are acquired by a local terminal, and wherein the local terminal is configured to: carrying out high-resolution processing on the infrared spectrum image and sending the infrared spectrum image subjected to the high-resolution processing to a server, or sending the infrared spectrum image to the server so as to carry out the high-resolution processing on the infrared spectrum image by the server;
means for performing hazard identification based on the identified object;
the module is used for sending out early warning to the alarm center when a dangerous condition occurs;
wherein the high resolution processing of the infrared spectral image comprises at least the step of comparing the visible spectral image with the interpolated infrared spectral image.
In a preferred embodiment, the high resolution processing of the infrared spectral image comprises the following steps:
utilizing a bicubic interpolation method to perform upsampling on the infrared spectrum image so as to enable the infrared spectrum image and the visible spectrum image to have the same size;
taking the up-sampled infrared spectrum image as an initial infrared image;
s1: establishing a cost function based on the initial infrared image or the candidate infrared image, wherein the cost function is as follows:
wherein,
is a function of the cost of the received signal,
is a custom constant value, d is a candidate infrared image,
the infrared image resulting from the current iteration, and wherein, when i =0,
is the initial infrared image.
In a preferred embodiment, the high resolution processing of the infrared spectral image comprises the following steps:
s2: obtaining an image of a red channel in the visible spectrum image based on the visible spectrum image;
s3: setting a correlation threshold value;
s4: in a given region of the image, calculating the cross correlation between a plurality of pixel points of the image of the red channel and a plurality of pixel points of the initial infrared image or a plurality of pixel points of the candidate infrared image;
s5: and if the cross correlation between the multiple pixel points of the image of the red channel and the multiple pixel points of the initial infrared image or the multiple pixel points of the candidate infrared image is greater than the correlation threshold value, determining that the given area of the image is a cross correlation area.
In a preferred embodiment, the high resolution processing of the infrared spectral image comprises the following steps:
s6: filtering a cost function in the cross-correlation, wherein filtering the cost function in the cross-correlation is based on the following formula:
wherein,
is a function of the cost of performing the filtering process,
is a custom window, wherein the size of the custom window is smaller than the size of the cross-correlation region,
is an image of the red channel, where,
and
defined by the following formula;
wherein,
is the initial infrared image or the candidate infrared image
The average value of (a) is,
is the initial infrared image or the candidate infrared image
The variance in (a) is greater than or equal to,
is a constant value that is self-defined,
is defined by the following equation:
wherein,
is the total number of pixel points in the user-defined window;
s7: by making
And minimizing the function to obtain the updated candidate infrared image d.
In a preferred embodiment, the high resolution processing of the infrared spectral image comprises the following steps:
s8: performing sub-pixel level interpolation on the updated candidate infrared image d to obtain a finished candidate infrared image
The sub-pixel-level interpolation of the updated candidate infrared image d is specifically performed by the following method:
wherein,
，
and wherein the first and second end portions of the first and second,
s9: to complete candidate infrared images
Replacing the initial infrared image or the candidate infrared image;
and (5) iterating steps S1-S9 until the change of the cost function reaches a convergence standard, thereby obtaining the infrared image processed by high resolution.
FIG. 3 is a photograph of an untreated visible spectrum according to one embodiment of the present invention. FIG. 4 is a photograph of an infrared spectrum having undergone high resolution processing in accordance with one embodiment of the present invention. FIG. 5 is a photograph of an infrared spectrum subjected to bicubic interpolation according to an embodiment of the present invention. The basic technical problem of the present invention is how to obtain a high resolution infrared image without changing the hardware of the infrared image capturing device, the simplest method is to actually interpolate the original infrared image and "force up" the nominal resolution of the image, and fig. 5 shows the result of this processing. By comparing the results of this simple method of fig. 5 with those of the present application (fig. 4), it can be seen that the method of the present application has clearer and smoother edges. The results of the other set of treatments of fig. 6-8 also show the same effect.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The foregoing descriptions of specific exemplary embodiments of the present invention have been presented for purposes of illustration and description. It is not intended to limit the invention to the precise form disclosed, and obviously many modifications and variations are possible in light of the above teaching. The exemplary embodiments were chosen and described in order to explain certain principles of the invention and its practical application to enable one skilled in the art to make and use various exemplary embodiments of the invention and various alternatives and modifications as are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the claims and their equivalents.
Claims (10)
1. A behavior identification method based on multispectral image information is characterized in that: the behavior identification method based on the multispectral image information comprises the following steps:
acquiring a visible spectrum image and an infrared spectrum image by a local terminal;
the infrared spectrum image is subjected to high-resolution processing by the local terminal and is sent to the server, or the infrared spectrum image is sent to the server by the local terminal so as to be subjected to high-resolution processing by the server;
identifying, by a server, an object in an image based on the high-resolution processed infrared spectral image and the visible spectral image;
performing, by the server, hazard identification based on the identified object;
when a dangerous condition occurs, the server sends out early warning to an alarm center;
in response to receiving the early warning, an alarm is made by an alarm center;
wherein the high resolution processing of the infrared spectral image comprises at least the step of comparing the visible spectral image with the interpolated infrared spectral image.
2. The method for behavior recognition based on multispectral image information according to claim 1, wherein: the high-resolution processing of the infrared spectrum image comprises the following steps:
upsampling the infrared spectrum image using a bicubic interpolation method so as to make the infrared spectrum image and the visible spectrum image have the same size;
taking the up-sampled infrared spectrum image as an initial infrared image;
s1: establishing a cost function based on the initial infrared image or the candidate infrared image, wherein the cost function is as follows:
wherein,
is a function of the cost of the received signal,
is a custom constant value, d is a candidate infrared image,
is the infrared image resulting from the current iteration, and wherein, when i =0,
is the initial infrared image.
3. The method for behavior recognition based on multispectral image information according to claim 2, wherein: the high-resolution processing of the infrared spectrum image comprises the following steps:
s2: obtaining an image of a red channel in the visible spectrum image based on the visible spectrum image;
s3: setting a correlation threshold value;
s4: in a given region of an image, calculating the cross correlation between a plurality of pixel points of the image of the red channel and a plurality of pixel points of the initial infrared image or a plurality of pixel points of the candidate infrared image;
s5: and if the cross correlation between the multiple pixel points of the image of the red channel and the multiple pixel points of the initial infrared image or the multiple pixel points of the candidate infrared image is greater than the correlation threshold value, determining that the given area of the image is a cross correlation area.
4. The method for behavior recognition based on multispectral image information according to claim 3, wherein: the high-resolution processing of the infrared spectrum image comprises the following steps:
s6: filtering a cost function in the cross-correlation, wherein filtering the cost function in the cross-correlation is based on the following formula:
wherein,
is a function of the cost of performing the filtering process,
is a custom window, wherein the size of the custom window is smaller than the size of the cross-correlation region,
is an image of the red channel, wherein,
and
defined by the following formula;
wherein,
is the initial infrared image or the candidate infrared image
Middle panelThe average value of the average value is calculated,
is the initial infrared image or the candidate infrared image
The variance in (a) is greater than or equal to,
is a constant value that is self-defined,
is defined by the following equation:
wherein,
is the total number of pixel points in the user-defined window;
s7: by making the
And minimizing the function to obtain the updated candidate infrared image d.
5. The method for behavior recognition based on multispectral image information as recited in claim 4, wherein: the high-resolution processing of the infrared spectrum image comprises the following steps:
s8: performing sub-pixel level interpolation on the updated candidate infrared image d to obtain a finished candidate infrared image
Performing sub-pixel-level interpolation on the updated candidate infrared image d specifically by the following method:
wherein,
，
and wherein the first and second end portions of the first and second,
s9: to complete candidate infrared images
Replacing the initial infrared image or the candidate infrared image;
and (5) iterating steps S1-S9 until the change of the cost function reaches a convergence standard, thereby obtaining the infrared image processed by high resolution.
6. A behavior recognition system based on multispectral image information is characterized in that: wherein the multispectral image information based behavior identification system is completed at a server, wherein the multispectral image information based behavior identification system comprises:
means for identifying an object in an image based on a high resolution processed infrared spectral image and a visible spectral image, wherein the visible spectral image and the infrared spectral image are acquired by a local terminal, and wherein the local terminal is configured to: carrying out high-resolution processing on the infrared spectrum image and sending the infrared spectrum image subjected to the high-resolution processing to a server, or sending the infrared spectrum image to the server so as to carry out the high-resolution processing on the infrared spectrum image by the server;
means for performing hazard identification based on the identified object;
the module is used for sending out early warning to the alarm center when a dangerous condition occurs;
wherein the high resolution processing of the infrared spectral image comprises at least the step of comparing the visible spectral image with the interpolated infrared spectral image.
7. The multispectral image information-based behavior recognition system of claim 6, wherein: the high-resolution processing of the infrared spectrum image comprises the following steps:
upsampling the infrared spectrum image using a bicubic interpolation method so as to make the infrared spectrum image and the visible spectrum image have the same size;
taking the up-sampled infrared spectrum image as an initial infrared image;
s1: establishing a cost function based on the initial infrared image or the candidate infrared image, wherein the cost function is as follows:
wherein,
is a function of the cost of the received signal,
is a custom constant value, d is a candidate infrared image,
the infrared image resulting from the current iteration, and wherein, when i =0,
is the initial infrared image.
8. The multispectral image information-based behavior recognition system of claim 7, wherein: the high-resolution processing of the infrared spectrum image comprises the following steps:
s2: obtaining an image of a red channel in the visible spectrum image based on the visible spectrum image;
s3: setting a correlation threshold value;
s4: in a given region of an image, calculating the cross correlation between a plurality of pixel points of the image of the red channel and a plurality of pixel points of the initial infrared image or a plurality of pixel points of the candidate infrared image;
s5: and if the cross correlation between the multiple pixel points of the image of the red channel and the multiple pixel points of the initial infrared image or the multiple pixel points of the candidate infrared image is greater than the correlation threshold value, determining that the given area of the image is a cross correlation area.
9. The multispectral image information-based behavior recognition system of claim 8, wherein: the high-resolution processing of the infrared spectrum image comprises the following steps:
s6: filtering a cost function in the cross-correlation, wherein filtering the cost function in the cross-correlation is based on the following formula:
wherein,
is a function of the cost of performing the filtering process,
is a custom window, wherein the size of the custom window is smaller than the size of the cross-correlation region,
is an image of the red channel, wherein,
and
defined by the following formula;
wherein,
is the initial infrared image or the candidate infrared image
The average value of (a) is,
is the initial infrared image or the candidate infrared image
The variance in (a) is greater than or equal to,
is a constant value that is self-defined,
is defined by the following equation:
wherein,
is the total number of pixel points in the user-defined window;
s7: by making the
And minimizing the function to obtain the updated candidate infrared image d.
10. The multispectral image information-based behavior recognition system of claim 9, wherein: the high-resolution processing of the infrared spectrum image comprises the following steps:
s8: performing sub-pixel level interpolation on the updated candidate infrared image d to obtain a finished candidate infrared image
Performing sub-pixel-level interpolation on the updated candidate infrared image d specifically by the following method:
wherein,
，
and wherein the first and second end portions of the first and second,
s9: to complete candidate infrared images
Replacing the initial infrared image or the candidate infrared image;
and (5) iterating steps S1-S9 until the change of the cost function reaches a convergence standard, thereby obtaining the infrared image processed by high resolution.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911318478.8A CN111191656A (en) | 2019-12-20 | 2019-12-20 | Behavior identification method and system based on multispectral image information |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911318478.8A CN111191656A (en) | 2019-12-20 | 2019-12-20 | Behavior identification method and system based on multispectral image information |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111191656A true CN111191656A (en) | 2020-05-22 |
Family
ID=70707375
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911318478.8A Pending CN111191656A (en) | 2019-12-20 | 2019-12-20 | Behavior identification method and system based on multispectral image information |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111191656A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130201342A1 (en) * | 2012-02-02 | 2013-08-08 | Xerox Corporation | Estimating a visible vector representation for pixels in an infrared image |
| CN106447630A (en) * | 2016-09-18 | 2017-02-22 | 清华大学 | High-spectral image sharpening method based on probability matrix decomposition |
| CN107918748A (en) * | 2017-10-27 | 2018-04-17 | 南京理工大学 | A kind of multispectral two-dimension code recognition device and method |
| CN110533620A (en) * | 2019-07-19 | 2019-12-03 | 西安电子科技大学 | The EO-1 hyperion and panchromatic image fusion method of space characteristics are extracted based on AAE |
| US20200304732A1 (en) * | 2019-03-20 | 2020-09-24 | Apple Inc. | Multispectral image decorrelation method and system |
-
2019
- 2019-12-20 CN CN201911318478.8A patent/CN111191656A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130201342A1 (en) * | 2012-02-02 | 2013-08-08 | Xerox Corporation | Estimating a visible vector representation for pixels in an infrared image |
| CN106447630A (en) * | 2016-09-18 | 2017-02-22 | 清华大学 | High-spectral image sharpening method based on probability matrix decomposition |
| CN107918748A (en) * | 2017-10-27 | 2018-04-17 | 南京理工大学 | A kind of multispectral two-dimension code recognition device and method |
| US20200304732A1 (en) * | 2019-03-20 | 2020-09-24 | Apple Inc. | Multispectral image decorrelation method and system |
| CN110533620A (en) * | 2019-07-19 | 2019-12-03 | 西安电子科技大学 | The EO-1 hyperion and panchromatic image fusion method of space characteristics are extracted based on AAE |
Non-Patent Citations (2)
| Title |
|---|
| 李旭;高雅楠;SHIGANG YUE;: "一种尺度感知型遥感图像融合新方法", 宇航学报 * |
| 胡永利;王亮;刘蓉;张丽;段福庆;: "一种从粗到精的红外和可见光卫星图像配准方法", 光谱学与光谱分析 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Eigen et al. | Restoring an image taken through a window covered with dirt or rain | |
| You et al. | Adherent raindrop modeling, detection and removal in video | |
| CN106780486B (en) | Steel plate surface defect image extraction method | |
| US9030559B2 (en) | Constrained parametric curve detection using clustering on Hough curves over a sequence of images | |
| TWI607901B (en) | Image inpainting system area and method using the same | |
| US10043090B2 (en) | Information processing device, information processing method, computer-readable recording medium, and inspection system | |
| KR20180109665A (en) | A method and apparatus of image processing for object detection | |
| EP1892666A1 (en) | A method, an apparatus and a computer-readable medium for processing an image dataset | |
| CN108710910A (en) | A kind of target identification method and system based on convolutional neural networks | |
| Paul et al. | Application of HDR algorithms to solve direct sunlight problems when autonomous vehicles using machine vision systems are driving into sun | |
| CN118195902B (en) | Super-resolution image processing method and processing system based on interpolation algorithm | |
| EP3113107A1 (en) | Static soiling detection and correction | |
| CN109544535B (en) | Peeping camera detection method and system based on optical filtering characteristics of infrared cut-off filter | |
| CN108399617B (en) | Method and device for detecting animal health condition | |
| Shit et al. | An encoder‐decoder based CNN architecture using end to end dehaze and detection network for proper image visualization and detection | |
| CN106657816A (en) | ORB algorithm based multipath rapid video splicing algorithm with image registering and image fusion in parallel | |
| CN109410160B (en) | Infrared polarization image fusion method based on multi-feature and feature difference driving | |
| CN114332655A (en) | Vehicle self-adaptive fusion detection method and system | |
| CN102609710A (en) | Smoke and fire object segmentation method aiming at smog covering scene in fire disaster image video | |
| CN111191656A (en) | Behavior identification method and system based on multispectral image information | |
| Thepade et al. | Improved haze removal method using proportionate fusion of color attenuation prior and edge preserving | |
| Lin et al. | Face detection based on skin color segmentation and SVM classification | |
| Kristanto et al. | Forensic analysis of faces on low-quality images using detection and recognition methods | |
| Jayanthi et al. | Underwater haze removal using contrast boosted grayscale image | |
| Sasikala et al. | An adaptive edge detecting method for satellite imagery based on canny edge algorithm |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20200522 |