CN115984124A - A neuromorphic pulse signal denoising and super-resolution method and device - Google Patents

Abstract

The invention discloses a neuromorphic pulse signal denoising and super-resolution method and device. By setting the same video with different resolutions in the display screen and using a pulse camera to shoot the display screen, real pulse data pairs with different resolutions can be obtained. , using the real shot data set as the training set avoids the problem that the trained network is not compatible with the real data due to the large gap between the simulated data and the real data, and solves the problem that the pulse signal simulator cannot accurately generate event data. At the same time, using the deep learning method, the 3D-UNet network model is used to learn the end-to-end mapping model of pulse signal denoising and super-resolution reconstruction. When the input is only the pulse sequence, it can effectively realize the denoising and super-resolution of the event. The super-resolution task avoids the dependence of existing methods on video frames and IMU information, saves the process of solving optical flow information, saves a lot of running time, and greatly improves the processing speed.

Description

A neuromorphic pulse signal denoising and super-resolution method and device

technical field

The invention relates to the technical field of computer vision, in particular to a neuromorphic pulse signal denoising and super-resolution method and device.

Background technique

With the development of computer technology, computer computing power is gradually strengthened, machine learning and deep learning technology are advancing rapidly, and computer vision-related technologies are gradually applied to various scenarios, such as face detection of mobile phone cameras, retouching and beautifying pictures, and taking pictures at night. Pedestrian detection and road recognition in human driving, face recognition for mobile payment and station identity detection, or simultaneous positioning and mapping tasks for robots, etc. With the advent of the era of big data and intelligence, more and more application scenarios need the support of computational vision technology. Massive video and image data need to be processed urgently, which highlights the importance of underlying visual tasks. Therefore, the irreplaceability of the underlying image processing technology and its significance for higher semantic level tasks have attracted widespread attention from the society. Imaging with low noise, low blur, high spatial resolution, high temporal resolution, and high dynamic range is the basic task of computational photography, and its development is extremely important for other computer vision technologies.

However, after decades of development, traditional digital cameras have entered various fields of people's lives. With the advent of the research boom of artificial intelligence in recent years, traditional digital cameras are powerless to solve the visual problems in the application fields such as automatic driving, drone control, and intelligent robots. The reason is that these emerging applications have high requirements for the capture of high-speed motion, while the sampling method of fixed frame rate of traditional digital cameras can only produce blurred images or videos in the face of high-speed motion. In recent years, the neuromorphic pulse sensor imitating the principle of biological retinal imaging has gradually become popular. With its advantages of high dynamic range and high temporal resolution, it has entered many visual analysis applications. However, the shortcomings of high noise and low resolution restrict the application of pulse cameras in the field of industrial vision.

Compared with traditional digital cameras, pulse cameras abandon the concept of "frame" and "exposure". Each pixel perceives and integrates the light gun independently. Passed out, 0 means that the pixel has no pulse at this moment, and 1 means that the pixel outputs a pulse at this moment. Pulse trains are generated continuously as the light intensity varies. Unlike traditional 2D pictures or video sequences, the triggered pulse time series is presented in the form of a 3D spatiotemporal point cloud. Due to the special imaging principle of the pulse camera and the limitation of the existing sensor manufacturing process level, the current pulse camera has problems such as high noise and low spatial resolution, which restrict the application of the pulse camera in the field of industrial vision. When impulsive cameras are used for tasks such as target tracking and object detection, features may be distorted or missing, leading to large degradation in results. When the pulse camera is used for the tasks of high frame rate image generation, image deblurring, and image high dynamic range restoration, problems such as loss of detail texture and poor visual experience may occur.

For the denoising problem of pulse signal, there is no related method at present. The signal denoising problem of the event camera, which belongs to the same neuromorphic camera as the pulse camera, currently has the following three methods to solve: Method 1) Remove noise events based on the spatio-temporal correlation of event signals in local spatio-temporal blocks, such as Super Resolve Dynamic Scene fromContinuous Spike Streams. Method 2) By using the video frame and camera motion information recorded synchronously in DVS to predict the probability of whether the event in the local spatio-temporal block is noise or not, so as to mark the training samples, and then learn the event noise classification network based on the neural network, and then analyze the event signal Noise removal, such as Event Probability Mask (EPM) and Event Denoising Convolutional Neural Network (EDnCNN) for NeuromorphicCameras). Method 3) Build a hybrid camera system of an event camera and a traditional camera, and establish the relationship between the image signal and the event signal by calculating the spatio-temporal gradient, so as to use the low-noise and high-resolution features of the image signal and improve the accuracy of the event signal through guided filtering. Quality (Joint Filtering of Intensity Images and Neuromorphic Events for High-Resolution Noise-Robust Imaging). However, these three methods have their own problems. Method 1) and method 2) cannot handle the complex event signals of the captured motion scenes, cannot achieve super-resolution processing of events, and can only eliminate events marked as noise. Untriggered events cannot be restored; the performance of method 3) depends on the quality of the image signal, and the optical flow information of the local space-time block needs to be calculated, and the processing speed is relatively slow.

For the above-mentioned super-resolution problem of pulse signal, there are currently two main methods to solve it: Method 4) Based on the spatio-temporal correlation between image intensity and pulse signal, establish a low-resolution pulse signal to high-resolution image guided by moving optical flow. Super-resolution algorithms for high-resolution images. Method 5) Use a pulse signal simulator to form a data set, and learn a mapping network from a low-resolution pulse signal to a high-resolution image based on a deep learning network. However, method 4) is extremely slow for super-resolution due to the process of optical flow estimation and pixel-by-pixel estimation. Method 5) Since the existing pulse signal simulators are difficult to simulate the noise and high time accuracy of real pulse signals, the trained network lacks compatibility with real pulse signals.

Contents of the invention

Aiming at the defects of the prior art, the present invention proposes a neuromorphic pulse signal denoising and super-resolution method based on real sample collection and deep learning, and obtains images for network training by synchronously shooting the same scene with different resolutions with a pulse camera A large number of real data sets solve the problem that the pulse signal simulator cannot accurately generate event data; at the same time, the 3D-UNet network model is used to learn the end-to-end mapping model of pulse signal denoising and super-resolution reconstruction, avoiding the dependence of existing methods Video frame and IMU information saves the process of solving optical flow information and saves a lot of running time.

In order to achieve the above object, the present invention provides the following technical solutions:

In one aspect, the present invention provides a neuromorphic pulse signal denoising and super-resolution method, comprising the following steps:

S1. Training data collection: The real training data set is obtained by synchronously shooting the same scene with different spatial resolutions with the pulse camera, and the display screen is used to synchronously display motion videos with different resolutions, and then intercept from the data captured by the pulse camera Generate pulse data with different resolutions, and finally form a complete RGB frame + multi-resolution pulse data set;

S2. Pulse data conversion: use 3D convolutional neural network to perform Encoder-Decoder processing on event information;

S3. Pulse denoising and spatial upsampling: The convolutional neural network based on the L2 norm obtains the denoising model of the event signal during learning, obtains the optimal solution of the denoising model, and outputs denoising and upsampling in the form of 3D tensor The reconstructed image afterwards;

S4. Pulse signal redistribution: Redistribute the pulses of the reconstructed image in the form of 3D tensor by distributing time stamps at equal intervals to restore high-resolution pulse signals.

Further, the information contained in each group of captured pulse data in step S1 is combined into {RGB frame, S1A, S1B, S2, time stamp sequence}, and the information combination of {RGB frame, S1A, S2, time stamp sequence} is used to Complete the training of the 2-fold super-resolution network, and the information combination of {RGB frame, S1A, S1B, time stamp sequence} is used to complete the training of the denoising network.

Further, the original pulse data in step S1 contains 40,000 H×W 0-1 matrices per second, and records whether there is a pulse on each pixel with a time accuracy of 25 μs. 0 means no pulse, and 1 means there is a pulse.

Further, in step S2, before the Encoder-Decoder process, the original pulse signal is pre-processed for image reconstruction using a pulse interval-based method.

Further, the preprocessing process is: use the time interval of two adjacent pulses on each pixel to represent the reciprocal of the light intensity at that moment, so as to form a preliminary reconstructed image at each moment, that is, reconstruct 40,000 frames of preliminary images per second. image, as the input of the subsequent network.

Further, the optimal solution of the denoising model in step S3 is expressed as:

where S is the input and output training data contaminated by noise, and Ω is the obtained denoising model.

Further, step S3 utilizes the structure of 3D UNet to achieve denoising and super-resolution tasks at the same time. In the 2x super-resolution network, 3D deconvolution layers are added to each level of 3D UNet for cross-level feature fusion to achieve resolution rate magnification.

Further, during the training period, 24,000 LR-HR pulse pairs were generated from the collected data as the training set; the benchmark size was set to 8, and 100 epochs were trained; the optimizer was ADAM, and the loss function loss was determined by the weight ratio It is a combination of Charbonnier loss and TV loss of 1:0.005; the initial learning rate is 0.001, and the decay is 0.5 times every 50 cycles.

On the other hand, the present invention also provides a neuromorphic pulse signal denoising and super-resolution device, including a display screen and a pulse camera, and the following modules to realize the method described in any one of the above:

Training data acquisition module: used to capture the same scene with different spatial resolutions synchronously through the pulse camera to obtain a real training data set, and use the display screen to display motion videos with different resolutions synchronously, and intercept different data from the data captured by the pulse camera resolution pulse data, and finally form a complete RGB frame + multi-resolution pulse data set;

Pulse data conversion module: used for Encoder-Decoder processing of event information using 3D convolutional neural network;

Pulse denoising and spatial upsampling module: used to obtain the denoising model of the event signal and obtain the optimal solution of the denoising model, and output the reconstructed image in the form of 3D tensor after denoising and upsampling;

Pulse signal redistribution module: It is used to redistribute the pulses of the reconstructed image in the form of 3D tensor by distributing time stamps at equal intervals, and restore high-resolution pulse signals.

In another aspect, the present invention also provides a device, including a processor, a communication interface, a memory, and a communication bus, and the processor, the communication interface, and the memory complete mutual communication through the communication bus; wherein :

The memory is used to store computer programs;

The processor is configured to implement the method described in any one of the above when executing the program stored in the memory.

Compared with prior art, the beneficial effect of the present invention is:

In the neuromorphic pulse signal denoising and super-resolution method and device of the present invention, the same video with different resolutions is set in the display screen, and the pulse camera is used to shoot the display screen, thereby obtaining real pulse data pairs with different resolutions. Taking the data set as the training set avoids the problem that the trained network is not compatible with the real data due to the large gap between the simulation data and the real data, and solves the problem that the pulse signal simulator cannot accurately generate event data. At the same time, using the method of deep learning, the 3D-UNet network model is used to learn the end-to-end mapping model of pulse signal denoising and super-resolution reconstruction. When the input is only the pulse sequence, the denoising and event denoising can be effectively realized. The super-resolution task avoids the dependence of existing methods on video frames and IMU information, saves the process of solving optical flow information, saves a lot of running time, and greatly improves the processing speed.

Description of drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings required in the embodiments. Apparently, the drawings in the following description are only some embodiments described in the present invention, and those skilled in the art can also obtain other drawings according to these drawings.

Fig. 1 is a flowchart of a neuromorphic pulse signal denoising and super-resolution method provided by an embodiment of the present invention.

Fig. 2 is a schematic diagram of a training data collection device provided by an embodiment of the present invention.

Fig. 3 is a schematic diagram of a shooting system provided by an embodiment of the present invention.

Fig. 4 shows three viewing angles of windows in the display provided by the embodiment of the present invention.

Detailed ways

In order to better understand the technical solution, the method of the present invention will be described in detail below in conjunction with the accompanying drawings.

The neuromorphic pulse signal denoising and super-resolution method of the present invention, as shown in Figure 1, includes training data acquisition, pulse data conversion, pulse signal space upsampling and pulse signal redistribution steps, specifically as follows:

S1. Training data collection

Since the existing emulators cannot accurately simulate the distribution model of event data, the present invention proposes to use a pulse camera (such as Vidar) to simultaneously capture the same scene with different spatial resolutions, thereby obtaining a real "low resolution-high resolution Rate" training data set, using the display screen to display motion videos of different resolutions synchronously, and then intercepting pulse data of different resolutions from the data captured by the pulse camera, and finally forming a complete RGB frame + multi-resolution pulse data set. The combination of information contained in each group of captured pulse data is {RGB frame, S1A, S1B, S2, time stamp sequence}, and the information combination of {RGB frame, S1A, S2, time stamp sequence} is used to complete the 2 times super-resolution network The training of {RGB frame, S1A, S1B, timestamp sequence} information combination is used to complete the training of the denoising network.

S2, pulse data conversion

Since the present invention uses a 3D convolutional neural network to perform Encoder-Decoder processing on event information, in order to include more spatial image information in the input information, the present invention first preprocesses the original pulse signal. In the original pulse data, each second contains 40,000 H×W 0-1 matrices, and records whether there is a pulse on each pixel with a time accuracy of 25 μs (0 means no pulse, 1 means there is a pulse). The present invention utilizes the TFI image reconstruction method to represent the reciprocal of the light intensity at this moment by using the time interval of two adjacent pulses before and after each pixel (as shown in (a) in Figure 4), thereby forming a preliminary Reconstruct the image, that is, reconstruct 40,000 frames of preliminary images per second as the input of the subsequent network. The high-resolution pulse sequence as the ground truth also needs to do this preprocessing operation.

S3, pulse denoising and spatial upsampling

Although the low-resolution and high-resolution data collected in step S1 are polluted by noise, since the noise of the pulse signal basically conforms to the law of Gaussian distribution, the present invention adopts the convolutional neural network based on the L2 norm in the The denoising model of the event signal is obtained during the learning, the optimal solution of the denoising model is obtained, and the reconstructed image after denoising and upsampling in the form of 3D tensor is output; the optimal solution solution formula is as follows:

where S is the input and output training data contaminated by noise, and Ω is the obtained denoising model.

Therefore, the present invention can train a denoising network model only by using real noisy data. As shown in Figure 4, the present invention utilizes the structure of 3D UNet to simultaneously achieve denoising and super-resolution tasks.

S4. Pulse signal redistribution

Since the denoising and upsampling events of the network output are output in the form of a 3D tensor, the number in the tensor must be converted into a pulse expression to finally realize the function that the input is an event and the output is also an event. Since the present invention initially reconstructs the image based on the pulse interval during preprocessing, the value of each pixel of the output image represents the interval of two pulses before and after the time point of the pixel on the time axis. Based on this principle, the output result can be Restores the pulse signal in binary mode. Specifically, the present invention redistributes the pulses of the reconstructed images in the form of 3D tensor by distributing time stamps at equal intervals to restore high-resolution pulse signals.

The present invention adopts the real data of different resolutions captured in step S1 to train the neural network, and the specific training process is as follows:

(1) Shooting training data set

a) Download the public high-speed slow-motion video dataset from the Internet.

b) Re-synthesize a new video, each frame of the video contains multiple local windows with the same time synchronization content but different spatial resolutions, corresponding to 1 times and 2 times the resolution (if there is a pulse camera with a larger resolution in the future, also can be increased to 4 times and 8 times).

c) Build a data acquisition device as shown in the figure below, as shown in Figure 2, including a display and a neuromorphic pulse camera (or time camera), and may also include a level and aiming device, as shown in Figure 3, to ensure event shooting The viewing angle is directly and parallel to the display.

d) Shooting data: When shooting starts, except for the display screen, all indoor ambient light sources are turned off to reduce the impact of the external environment on data shooting. Then take the processed video.

e) Processing pulse data: For each group of captured pulse data, intercept local areas in turn to form independent pulse data with different resolutions, corresponding to pulse data with different resolutions, and finally form a complete RGB frame + multiple Datasets of resolution pulses, each group contains information combined as {RGB frame, S1A, S1B, S2, timestamp sequence}. In the following network training, the present invention uses the combination of {RGB frame, S1A, S2, time stamp sequence} to complete the training of the 2 times super-resolution network, and completes it with the combination of {RGB frame, S1A, S1B, time stamp sequence} Training of the denoising network.

(2) Training of neural network

a) Event information preprocessing: When training denoising and upsampling networks, both LR (low dynamic range) and HR (high dynamic range) events are merged into a 32-channel event tensor to complete supervised training. The events in that time interval are summed per pixel in each channel. We also experimented with different channel numbers and found that 32 channels gave the best performance.

b) The main module of the entire network is 3D UNet. In the 2x super-resolution network, the network adds a 3D deconvolution layer to the skip connection of each level of 3D UNet, and cross-level feature fusion to achieve resolution amplification. In the denoising network, there is no need to add the above-mentioned deconvolution network. The input and output pulse signals are pre-processed for preliminary image reconstruction based on the pulse interval method, and the output tensor is rounded to an integer value to obtain the reconstructed image after super-resolution and denoising, and then distribute time stamps at equal intervals. The pulse redistribution is carried out in this way, and the high-resolution pulse signal can be restored.

c) During training, we generated 24,000 LR-HR pulse pairs from the collected data as a training set. Benchsize is set to 8 and trained for 100 epochs. The optimizer is ADAM, the loss is a loss function composed of Charbonnier loss and TV loss with a weight ratio of 1:0.005, the initial learning rate is 0.001, and the decay is 0.5 times every 50 cycles. It took a total of about 12 hours using PyTorch 1.6 and an NVIDIA 2080Ti GPU.

d) During the test, denoising and super-resolution can be performed only by inputting the real shot pulse sequence.

Corresponding to the method provided by the above-mentioned embodiments of the present invention, the present invention provides a neuromorphic pulse signal denoising and super-resolution device, including a display screen, a pulse camera, and the following modules to implement the method described in any one of the above-mentioned embodiments :

Training data acquisition module: used to capture the same scene with different spatial resolutions synchronously through the pulse camera to obtain a real training data set, and use the display screen to display motion videos with different resolutions synchronously, and intercept different data from the data captured by the pulse camera resolution pulse data, and finally form a complete RGB frame + multi-resolution pulse data set;

Pulse data conversion module: used for Encoder-Decoder processing of event information using 3D convolutional neural network;

Pulse denoising and spatial upsampling module: used to obtain the denoising model of the event signal and obtain the optimal solution of the denoising model, and output the reconstructed image in the form of 3D tensor after denoising and upsampling;

Pulse signal redistribution module: It is used to redistribute the pulses of the reconstructed image in the form of 3D tensor by distributing time stamps at equal intervals, and restore high-resolution pulse signals.

When the method or device of the present invention is applied, the following steps can be adopted:

a) Download the public high-speed slow-motion video dataset from the Internet, which contains 45 video sequences corresponding to all color frames, the frame rate is adjusted to 30fps, and the spatial resolution of each frame is 1280×720.

b) As shown in Figure 4 as an example, 45 corresponding new videos are re-synthesized, the frame rate of each video is adjusted to 360fps, and the resolution is 1280×720. Each frame of the video contains three local windows with the same time synchronization content but different spatial resolutions. The resolution of the largest window is 720×720, and the resolution of the smallest two windows is 360×360. In order to allow enough time to play the video and start the camera capture when shooting, the beginning and end frames of each video are paused for two seconds.

c) Building the data collection device shown in Figure 2: In this embodiment, the model of the display screen is ASUSPG259QNR, the resolution is 1920×1080, and the refresh rate is 360 Hz. The model is VidarOne, the resolution is 400×250, and the event camera with F/1.4 lens is placed horizontally about 180cm in front of the display screen. In order to ensure that the shooting angle is facing and parallel to the monitor.

d) Registration of the camera angle of view and the display area of the display screen: as shown in Figure 3, during registration, a cross bullseye is set at the center of the display, an aiming point is placed directly in front of the camera (limited by the breadboard), and the three-point-one-line method is used to Finally, make sure that the camera plane and the display plane are parallel, and the line connecting the center point is perpendicular to the two planes. And limit the horizontal rotation angle of the camera and the display to be the same by placing a spirit level on the camera and the display. This ensures perfect registration of the camera viewing angle and the display area of the display. The angle of view of the event camera after registration is shown in Figure 4. The angle of view corresponds to the area of the three windows in the display. The largest window corresponds to the resolution of the pulse camera at 240×240, and the smallest two windows correspond to the resolution of 120×120.

e) When shooting starts, except for the display screen, all indoor ambient light sources are turned off to reduce the impact of the external environment on data shooting. Then 45 processed videos were taken sequentially.

f) Process pulse data: realize the time registration of pulse data and color video. Start and end timing alignment is done using the markers reserved during the shoot. For each group of captured event data, according to the spatial coordinate position in the above figure, the local area is sequentially intercepted to form independent event data with different resolutions, which correspond to three data of S1A, S1B, and S2 respectively. The present invention finally forms a complete data set of 45 groups of RGB frames + multi-resolution pulses, and the information contained in each group is combined into {RGB frame, S1A, S1B, S2, time stamp sequence}. In the following network training, the present invention uses the combination of {S1A, S2, time stamp sequence} to complete the training of the 2-fold super-resolution network, and uses the combination of {S1A, S1B, time stamp sequence} to complete the training of the denoising network.

To sum up, compared with the prior art, the present invention first uses the real-shot data set as the training set, which avoids the problem that the trained network is not compatible with the real data due to the large gap between the simulation data and the real data. And by setting the same video with different resolutions in the display screen, and using the pulse camera to shoot the display screen, real pulse data pairs with different resolutions can be obtained. At the same time, using the method of deep learning, when the input is only the pulse sequence, the task of denoising and super-resolution of events can be effectively realized, and the processing speed is greatly improved.

Corresponding to the methods provided by the above-mentioned embodiments of the present invention, the embodiments of the present invention also provide an electronic device, including: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete the mutual communication via the communication bus. communication between

memory for storing computer programs;

The processor is configured to implement the flow of the method provided by the foregoing embodiments of the present invention when executing the program stored in the memory.

The communication bus mentioned in the above control device may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus or the like. The communication bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

The communication interface is used for communication between the electronic device and other devices.

The memory may include a random access memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-Volatile Memory, NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located far away from the aforementioned processor.

The above-mentioned processor can be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; it can also be a digital signal processor (Digital Signal Processing, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

In yet another embodiment provided by the present invention, a computer-readable storage medium is also provided, and a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the above-mentioned embodiment of the present invention is implemented. steps in any of the methods.

In yet another embodiment provided by the present invention, a computer program product containing instructions is also provided, and when it is run on a computer, it causes the computer to execute the steps of any method provided by the above-mentioned embodiments of the present invention.

In the above embodiments, all or part of them may be implemented by software, hardware, firmware or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the processes or functions according to the embodiments of the present invention will be generated in whole or in part. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website, computer, server or data center Transmission to another website site, computer, server, or data center by wired (eg, coaxial cable, optical fiber, digital terminal equipment line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server or a data center integrated with one or more available media. The available medium may be a magnetic medium (for example, a floppy disk, a hard disk, or a magnetic tape), an optical medium (for example, DVD), or a semiconductor medium (for example, a Solid State Disk (SSD)).

It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is a relationship between these entities or operations. There is no such actual relationship or order between them. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or device. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

Each embodiment in this specification is described in a related manner, the same and similar parts of each embodiment can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for apparatus embodiments, electronic device embodiments, computer-readable storage medium embodiments, and computer program product embodiments, since they are basically similar to method embodiments, the description is relatively simple. For related information, refer to method embodiments Part of the description is sufficient.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present invention are included in the protection scope of the present invention.

Claims (10)

1. a neuromorphic pulse signal denoising and super-resolution method, is characterized in that, comprises the following steps: S1. Training data collection: The real training data set is obtained by synchronously shooting the same scene with different spatial resolutions with the pulse camera, and the display screen is used to synchronously display motion videos with different resolutions, and then intercept from the data captured by the pulse camera Generate pulse data with different resolutions, and finally form a complete RGB frame + multi-resolution pulse data set; S2. Pulse data conversion: use 3D convolutional neural network to perform Encoder-Decoder processing on event information; S3. Pulse denoising and spatial upsampling: The convolutional neural network based on the L2 norm obtains the denoising model of the event signal during learning, obtains the optimal solution of the denoising model, and outputs denoising and upsampling in the form of 3D tensor The reconstructed image afterwards; S4. Pulse signal redistribution: Redistribute the pulses of the reconstructed image in the form of 3D tensor by distributing time stamps at equal intervals to restore high-resolution pulse signals. 2. The neuromorphic pulse signal denoising and super-resolution method according to claim 1, wherein the information contained in the pulse data captured by each group of step S1 is combined into {RGB frame, S1A, S1B, S2, time Stamp sequence}, {RGB frame, S1A, S2, time stamp sequence} information combination is used to complete the training of 2 times super-resolution network, and the information combination of {RGB frame, S1A, S1B, time stamp sequence} is used to complete denoising Network training. 3. The neuromorphic pulse signal denoising and super-resolution method according to claim 1, characterized in that, in the original pulse data of step S1, each second contains 40000 H×W 0-1 matrices, with a time of 25 μs Time accuracy records whether there is a pulse on each pixel, 0 means no pulse, 1 means there is a pulse. 4. The neuromorphic pulse signal denoising and super-resolution method according to claim 3, characterized in that, in step S2, before the Encoder-Decoder process, the original pulse signal is used to perform image reconstruction preprocessing based on the pulse interval method . 5. neuromorphic pulse signal denoising and super-resolution method according to claim 4, is characterized in that, pretreatment process is: utilize the time interval of two adjacent pulses before and after each pixel to represent the light intensity at this moment Count down to form a preliminary reconstructed image at each moment, that is, reconstruct 40,000 frames of preliminary images per second as the input of the subsequent network. 6. neuromorphic pulse signal denoising and super-resolution method according to claim 1, is characterized in that, in the step S3, the optimal solution of the denoising model is expressed as:

where S is the input and output training data contaminated by noise, and Ω is the obtained denoising model. 7. neuromorphic pulse signal denoising and super-resolution method according to claim 1, is characterized in that, step S3 utilizes the structure of 3D UNet to realize denoising and super-resolution task simultaneously, in 2 times super-resolution network, 3D Each level of UNet adds a 3D deconvolution layer for cross-level feature fusion to achieve resolution amplification. 8. neuromorphic pulse signal denoising and super-resolution method according to claim 1, is characterized in that, during training, has generated 24000 LR-HR pulse pairs as training set from the data gathered; Benchsize setting is 8, and trained for 100 epochs; the optimizer is ADAM, and the loss function loss is composed of Charbonnierloss and TVloss with a weight ratio of 1:0.005; the initial learning rate is 0.001, and the decay is 0.5 times every 50 cycles. 9. A neuromorphic pulse signal denoising and super-resolution device, characterized in that it includes a display screen and a pulse camera, and the following modules to realize the method according to any one of claims 1-8: Training data acquisition module: used to capture the same scene with different spatial resolutions synchronously through the pulse camera to obtain a real training data set, and use the display screen to display motion videos with different resolutions synchronously, and intercept different data from the data captured by the pulse camera resolution pulse data, and finally form a complete RGB frame + multi-resolution pulse data set; Pulse data conversion module: used for Encoder-Decoder processing of event information using 3D convolutional neural network; Pulse denoising and spatial upsampling module: used to obtain the denoising model of the event signal and obtain the optimal solution of the denoising model, and output the reconstructed image in the form of 3D tensor after denoising and upsampling; Pulse signal redistribution module: It is used to redistribute the pulses of the reconstructed image in the form of 3D tensor by distributing time stamps at equal intervals, and restore high-resolution pulse signals. 10. A kind of equipment, it is characterized in that, comprises processor, communication interface, memory and communication bus, described processor, described communication interface, described memory complete mutual communication through described communication bus; It is characterized in that, The memory is used to store computer programs; The processor is configured to implement the method according to any one of claims 1-8 when executing the program stored in the memory.

Images