CN115758225B - Fault prediction method and device based on multi-mode data fusion and storage medium - Google Patents
Fault prediction method and device based on multi-mode data fusion and storage medium Download PDFInfo
- Publication number
- CN115758225B CN115758225B CN202310017912.9A CN202310017912A CN115758225B CN 115758225 B CN115758225 B CN 115758225B CN 202310017912 A CN202310017912 A CN 202310017912A CN 115758225 B CN115758225 B CN 115758225B
- Authority
- CN
- China
- Prior art keywords
- equipment
- information
- production line
- fault
- type
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Testing And Monitoring For Control Systems (AREA)
Abstract
The invention discloses a fault prediction method, a device and a storage medium based on multi-mode data fusion, wherein the method comprises the following steps: acquiring functional information of production line equipment, and classifying the production line equipment based on the functional information to obtain a plurality of equipment category information; determining a fault analysis scheme corresponding to the equipment category information according to the equipment category information, and acquiring operation state data of the production line equipment based on the fault analysis scheme, wherein the operation state data comprises data of different modes; training a preset BERT-converter model based on the running state data to obtain a fault prediction model, and predicting the fault of the production line equipment based on the fault prediction model. The invention can adopt different fault analysis modes to carry out fault analysis on different production line equipment, thereby realizing customized accurate fault monitoring and ensuring the effect of fault analysis.
Description
Technical Field
The present invention relates to the field of equipment fault monitoring technologies, and in particular, to a fault prediction method, device and storage medium based on multi-mode data fusion.
Background
For large factories, a lot of processes are involved in the production line, and different processes need to use different production line equipment, and the failure modes of different production line equipment are different. When fault monitoring is performed on production line equipment, equipment operation data needs to be analyzed, and the modes of fault analysis are different because the expression forms of the equipment operation data are different from production line equipment to production line equipment. In the prior art, the fault analysis is basically carried out on different production line equipment in the same mode, the customized accurate fault monitoring cannot be carried out, and the effect of the fault analysis is affected.
Accordingly, there is a need for improvement and advancement in the art.
Disclosure of Invention
The invention aims to solve the technical problems that the prior art cannot be subjected to customized accurate fault monitoring and the effect of fault analysis is affected.
In order to solve the technical problems, the technical scheme adopted by the invention is as follows:
in a first aspect, the present invention provides a fault prediction method based on multi-mode data fusion, which is characterized in that the method includes:
acquiring functional information of production line equipment, and classifying the production line equipment based on the functional information to obtain a plurality of equipment category information;
determining a fault analysis scheme corresponding to the equipment category information according to the equipment category information, and acquiring operation state data of the production line equipment based on the fault analysis scheme, wherein the operation state data comprises data of different modes;
training a preset BERT-converter model based on the running state data to obtain a fault prediction model, and predicting the fault of the production line equipment based on the fault prediction model.
In one implementation, the acquiring the function information of the production line device includes:
acquiring production line process information, wherein the production line process information comprises process introduction information and processing flow information of each production line;
and acquiring production line equipment corresponding to each production line process information, and determining the function information of the production line equipment, wherein the function information is used for reflecting the processing effect of the production line equipment on a processing object.
In one implementation manner, the classifying the production line device based on the function information to obtain a plurality of device class information includes:
determining a technological process of the production line equipment for realizing corresponding function information based on the function information;
and determining the processing difficulty level of the process flow based on the process flow, classifying the production line equipment based on the processing difficulty level, and obtaining equipment type information, wherein the equipment type information comprises first type equipment and second type equipment, and the number of equipment parts of the first type equipment is greater than that of the second type equipment.
In one implementation manner, the determining, according to the device class information, a fault analysis scheme corresponding to the device class information includes:
if the equipment type information is first type equipment, acquiring a first fault analysis scheme corresponding to the first type equipment, wherein the first fault analysis scheme is a fault analysis scheme based on computer vision;
and if the equipment type information is the second type equipment, acquiring a second fault analysis scheme corresponding to the second type equipment, wherein the second fault analysis scheme is a fault analysis scheme based on audio and video information or infrared thermal information.
In one implementation, the acquiring the operation state data of the production line device based on the fault analysis scheme includes:
if the production line equipment is first type equipment, acquiring image information of the first type equipment in the running process based on the first fault analysis scheme, and determining running state data of the first type equipment based on the image information, wherein the image information comprises a motion track image and a processing process image;
if the production line equipment is second type equipment, acquiring audio and video information or infrared thermal sensation information of the second type equipment in the operation process based on the first fault analysis scheme, and determining operation state data of the second type equipment based on the audio and video information or the infrared thermal sensation information, wherein the audio and video information comprises working noise and equipment prompt sound of the second type equipment in the operation process.
In one implementation manner, the training the preset BERT-Transformer model based on the operation state data to obtain a fault prediction model includes:
inputting the image information for reflecting the running state data into a preset convolutional neural network for feature extraction to obtain image features, and denoising the image features to obtain denoised image features;
carrying out Kalman filtering processing on the infrared thermal sensation information for reflecting the running state data to obtain a noise-reduced infrared thermal sensation signal;
fourier transform and high-dimensional projection are used for the audio and video information for reflecting the running state data, so that an audio and video signal after noise reduction is obtained;
vectorizing the image characteristics after noise reduction, the audio and video signals after noise reduction and the infrared thermal signals after noise reduction respectively to obtain signals to be trained, wherein the signals to be trained comprise three vector signals with different dimensionalities;
and training the BERT-transducer model according to the signal to be trained to obtain a fault prediction model.
In one implementation manner, the training the BERT-Transformer model according to the signal to be trained to obtain a fault prediction model includes:
inputting the signal to be trained into the BERT-transducer model, wherein the BERT-transducer model comprises an encoder and a decoder;
the encoder based on the BERT-transducer model receives the signals to be trained, splices the signals to be trained, and outputs a single vector result after a multi-layer attention mechanism;
the single vector result is subjected to DNN structure and sigmoid activation function of a deep neural network trained based on manual annotation data, and a binary vector result is output, wherein the binary vector result is used for reflecting the probability of failure of the production line equipment;
and training the BERT-converter model based on the binary vector result and manually calibrated fault history data to obtain the fault prediction model.
In a second aspect, an embodiment of the present invention further provides a fault prediction device based on multi-mode data fusion, where the device includes:
the equipment classification module is used for acquiring the functional information of the production line equipment, classifying the production line equipment based on the functional information and obtaining a plurality of equipment category information;
the scheme analysis module is used for determining a fault analysis scheme corresponding to the equipment category information according to the equipment category information and acquiring the running state data of the production line equipment based on the fault analysis scheme, wherein the running state data comprise data of different modes;
the fault monitoring module is used for training a preset BERT-converter model based on the running state data to obtain a fault prediction model, and predicting the fault of the production line equipment based on the fault prediction model.
In a third aspect, an embodiment of the present invention further provides a terminal device, where the terminal device includes a memory, a processor, and a fault prediction program based on multi-mode data fusion stored in the memory and capable of running on the processor, and when the processor executes the fault prediction program based on multi-mode data fusion, the processor implements the steps of the fault prediction method based on multi-mode data fusion according to any one of the above schemes.
In a fourth aspect, an embodiment of the present invention further provides a computer readable storage medium, where the computer readable storage medium stores a fault prediction program based on multi-mode data fusion, where the step of the fault prediction method based on multi-mode data fusion according to any one of the above schemes is implemented when the fault prediction program based on multi-mode data fusion is executed by a processor.
The beneficial effects are that: compared with the prior art, the invention provides a fault prediction method based on multi-mode data fusion. Then, according to the equipment category information, determining a fault analysis scheme corresponding to the equipment category information, and acquiring operation state data of the production line equipment based on the fault analysis scheme, wherein the operation state data comprise data of different modes. And finally, training a preset BERT-converter model based on the running state data to obtain a fault prediction model, and predicting the fault of the production line equipment based on the fault prediction model. The invention can adopt different fault analysis modes to carry out fault analysis on different production line equipment, thereby realizing customized accurate fault monitoring and ensuring the effect of fault analysis.
Drawings
Fig. 1 is a schematic flow chart of a specific implementation of a fault prediction method based on multi-mode data fusion according to an embodiment of the present invention.
Fig. 2 is a schematic block diagram of a fault prediction device based on multi-mode data fusion according to an embodiment of the present invention.
Fig. 3 is a schematic diagram of a terminal device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and effects of the present invention clearer and more specific, the present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
The embodiment provides a fault prediction method based on multi-mode data fusion, and in specific implementation, the embodiment firstly obtains functional information of production line equipment, and classifies the production line equipment based on the functional information to obtain a plurality of equipment category information. Then, according to the equipment category information, determining a fault analysis scheme corresponding to the equipment category information, and acquiring operation state data of the production line equipment based on the fault analysis scheme, wherein the operation state data comprise data of different modes. And finally, training a preset BERT-converter model based on the running state data to obtain a fault prediction model, and predicting the fault of the production line equipment based on the fault prediction model. According to the embodiment, different fault analysis modes can be adopted to carry out fault analysis on different production line equipment, so that customized accurate fault monitoring is realized, and the effect of fault analysis is ensured.
Exemplary method
The fault prediction method based on multi-mode data fusion can be applied to terminal equipment, and the terminal equipment can be intelligent product terminals such as computers and mobile phones. Specifically, as shown in fig. 1, the fault prediction method based on multi-mode data fusion of the present embodiment includes the following steps:
step S100, obtaining functional information of production line equipment, and classifying the production line equipment based on the functional information to obtain a plurality of equipment category information.
The method includes the steps that firstly, functional information of production line equipment is obtained, and the functional information is used for reflecting the processing effect of the production line equipment on a processing object. For example, when the production line device is a polisher, the corresponding function information is polishing the workpiece. After the function information is obtained, the present embodiment can classify the production line devices based on the function information, so that different device class information can be obtained. The equipment category information of the embodiment can identify the production line equipment so as to realize fault analysis on different types of production line equipment.
Specifically, in this embodiment, first, production line process information is obtained, where the production line process information includes process introduction information and processing flow information of each production line. Because the manufacturer has a plurality of production line process information, the embodiment can respectively acquire the production line equipment corresponding to each production line process information. And then determining the function information corresponding to each production line device. In this embodiment, the determination of the function information may be determined based on the execution step corresponding to each production line process information, and the final processing effect of the production line device may be determined based on the final execution step, so that the function information of the production line device may be obtained.
Because the function information in the embodiment is determined based on the line process information in which the line equipment is located, after the function information is determined, the embodiment can determine the process flow corresponding to the line equipment when the line process information is executed based on the function information. The process flow includes all processing steps and corresponding processing parameters and processing requirements of each processing step, and processing difficulties of different processing parameters and processing requirements are different. And classifying the production line equipment according to the processing difficulty level to obtain equipment type information, wherein the equipment type information in the embodiment comprises first type equipment and second type equipment. That is, the present embodiment may determine the processing difficulty level of each processing flow according to the processing flow of each line device when executing the line process information, and then divide the line device into the first type device and the second type device according to the processing difficulty level, if the processing difficulty level is high, divide the line device into the first type device, and if the processing difficulty level is low, divide the line device into the second type device. In this embodiment, the number of device components of the first type of device is greater than the number of device components of the second type of device. For example, the present embodiment may preset to be defined as a first type of equipment when the number of equipment components of the line equipment exceeds a number threshold, and to be defined as a second type of equipment when the number of equipment components of the line equipment is less than the number threshold.
Step 200, determining a fault analysis scheme corresponding to the equipment category information according to the equipment category information, and acquiring operation state data of the production line equipment based on the fault analysis scheme, wherein the operation state data comprise data of different modes.
After obtaining the equipment category information, the embodiment can determine a fault analysis scheme corresponding to the equipment category information based on the equipment category information. For production line equipment with different equipment category information, the processing flow is different, and the fault conditions are different, so in order to perform fault analysis on the production line equipment in a customized manner, the embodiment sets a corresponding fault analysis scheme for each equipment category information. Therefore, the embodiment can determine the corresponding fault analysis scheme after obtaining the equipment category information.
Specifically, if the equipment type information is first type equipment, a first fault analysis scheme corresponding to the first type equipment is obtained. At this time, since the first type of equipment is basically a process flow with high processing difficulty, when the first type of equipment executes the process flow with high processing difficulty, the change degree of the running state data corresponding to the equipment is also relatively large, so that in order to analyze the running state data information of the equipment more accurately, the first fault analysis scheme in the embodiment is a fault analysis scheme based on computer vision. Computer vision is the process of processing a first type of device based on image recognition techniques. And if the equipment type information is the second type equipment, acquiring a second fault analysis scheme corresponding to the first type equipment. Because the second type equipment has small change degree of running state data corresponding to the equipment when executing a low-difficulty process flow, in order to accurately acquire the running state data, fault analysis is convenient to carry out on the processing process of the production line equipment, and if the fault analysis is carried out by adopting an image recognition technology, the second type equipment has small volume, small change degree during processing and strong anti-interference item of the equipment, so that the image processing effect can be influenced. Therefore, the second fault analysis scheme of the present embodiment is a fault analysis scheme based on audio/video information or infrared thermal information. The audio and video information and the infrared thermal information can collect the running state data more conveniently, so that fault analysis can be accurately carried out.
When the operation state data is collected, if the production line equipment is first type equipment, image information of the first type equipment in the operation process is collected based on the first fault analysis scheme, and the image information reflects an image of the whole flow of the first type equipment in the whole processing process, so that the state of the first type equipment when each processing step is executed is recorded. Specifically, the image information includes a motion trajectory image, a machining process image. When the image information is obtained, the present embodiment determines the operation state data based on the image information. The operational status data reflects the operational status of the first type of device for each period of time. And if the production line equipment is second type equipment, acquiring audio and video information or infrared thermal sensation information of the second type equipment in the running process based on the first fault analysis scheme. The audio and video information acquired in the embodiment can reflect the working noise, the equipment prompt tone, the operation prompt tone and other sounds generated in the processing process of the second type equipment. The infrared thermal information may reflect temperature changes of the second type of device during processing. The audio-video information or the infrared thermal information can reflect the processing condition of the second type of equipment in the processing process, so that the embodiment can determine the running state data of the second type of equipment based on the audio-video information or the infrared thermal information. Therefore, after the production line equipment comprises the first type equipment and the second type equipment, the embodiment can acquire information of different modes based on the first fault analysis scheme and the second fault analysis scheme respectively, and then analyze the acquired information of different modes, so that the running state data of the first type equipment and the running state data of the second type equipment are obtained respectively, and the customized fault analysis of different equipment is facilitated.
And step S300, training a preset BERT-converter model based on the running state data to obtain a fault prediction model, and predicting the fault of the production line equipment based on the fault prediction model.
After the operation state data is obtained, the embodiment trains a preset BERT-Transformer model based on the operation state data to obtain a fault prediction model. In this embodiment, the BERT-Transformer model is a model trained based on a BERT (language processing model) model and a Transformer model, and since the BERT (language processing model) model applies an encoder and the Transformer model applies a decoder, the BERT-Transformer model includes an encoder and a decoder, specifically. Because the operation state data comprises fault data, when the BERT-converter model is based on the operation state data, operations such as semantic identification, feature extraction and the like can be automatically performed on the operation state data, and the obtained fault prediction model has the capability of automatically analyzing the fault data from the operation state data which is automatically identified. Thus, a fault of the production line equipment is monitored based on the fault prediction model.
Specifically, when the fault prediction model is trained and corresponding image information is acquired for the first type of equipment, the embodiment firstly inputs the image information for reflecting the running state data into a preset convolutional neural network to perform feature extraction, and image features are obtained. The image features extracted at this time are image features when the first type of equipment fails in the machining process. Then, the embodiment performs noise reduction on the image features to obtain noise-reduced image features. And when corresponding audio and video information or infrared thermal information is acquired aiming at the second type of equipment, the embodiment carries out Kalman filtering processing on the infrared thermal information for reflecting the running state data to obtain an infrared thermal signal after noise reduction. The infrared thermal signal after noise reduction reflects temperature change information when the second type of equipment fails. The embodiment also uses fourier transform and high-dimensional projection to the audio and video information for reflecting the running state data, so as to obtain the audio and video signal after noise reduction. The audio/video signal after noise reduction reflects the sound signal such as noise generated when the second type of equipment fails. Then, in this embodiment, the image features after noise reduction, the audio/video signals after noise reduction and the infrared thermal signals after noise reduction are respectively vectorized to obtain the signals to be trained, and since the various signals (such as the image features after noise reduction, the audio/video signals after noise reduction and the infrared thermal signals after noise reduction) are collected in this embodiment, the signals to be trained obtained after quantization includes vector signals of three different dimensions. In order to facilitate subsequent processing, the embodiment may perform vectorization processing on the image feature after noise reduction, the audio/video signal after noise reduction, and the infrared thermal signal after noise reduction to obtain three word vector signals with different dimensions, and then train the BERT-Transformer model according to the signal to be trained to obtain a fault prediction model.
Specifically, the signal to be trained is input into the BERT-Transformer model, which includes an encoder and a decoder. Then, an encoder based on the BERT-transducer model receives the signal to be trained, and the signal to be trained is a word vector signal, so that
Firstly, carrying out semantic recognition on the signals to be trained based on a BERT model, then splicing to obtain spliced vectors, and then outputting a single vector result after a multi-layer attention mechanism of a transducer model, wherein the single vector result is the vector obtained by fusing the vectors with different dimensions, and the single vector result can reflect the running condition (namely normal condition or fault condition) of the running state data of the production line equipment. Then, in this embodiment, the single vector result is subjected to a DNN (Deep Neural Networks, deep neural network) structure and a sigmoid activation function of a deep neural network trained based on the manual labeling data, and a binary vector result is output, and since the deep neural network is trained based on the manual labeling data, the binary vector result output through the deep neural network can reflect the probability of the failure of the production line equipment. Then, the embodiment trains the BERT-converter model based on the binary vector result and manually calibrated fault history data to obtain the fault prediction model. In addition, the embodiment can also carry out gradient descent according to the difference between the predicted faults and the actual faults of the fault prediction model, so that the parameters of the BERT-converter model are updated, the prediction of the fault prediction model is more accurate, and the identification of the current time point of the fault of the customized equipment at the production line level and the prediction of the fault possibility in the future (next time step) are realized.
In summary, the present embodiment first obtains function information of production line equipment, and classifies the production line equipment based on the function information to obtain a plurality of equipment category information. Then, according to the equipment category information, determining a fault analysis scheme corresponding to the equipment category information, and acquiring operation state data of the production line equipment based on the fault analysis scheme, wherein the operation state data comprise data of different modes. And finally, training a preset BERT-converter model based on the running state data to obtain a fault prediction model, and predicting the fault of the production line equipment based on the fault prediction model. Therefore, when a manufacturer is provided with a plurality of production line processes, the production line equipment is designed to be a plurality of production line equipment, customized fault analysis can be performed on the production line equipment with different equipment category information respectively, data of different modes are collected, different fault analysis modes are adopted to perform fault analysis on different production line equipment, customized accurate fault monitoring is achieved, and the effect of fault analysis is guaranteed.
Exemplary apparatus
Based on the above embodiment, the present invention further provides a fault prediction device based on multi-mode data fusion, specifically, as shown in fig. 2, the device of this embodiment includes: a device classification module 10, a scenario analysis module 20, and a fault monitoring module 30. Specifically, the device classification module 10 is configured to obtain functional information of the production line device, and classify the production line device based on the functional information to obtain a plurality of device class information. The solution analysis module 20 is configured to determine, according to the equipment category information, a fault analysis solution corresponding to the equipment category information, and obtain, based on the fault analysis solution, operation state data of the production line equipment, where the operation state data includes data of different modalities. The fault monitoring module 20 is configured to train a preset BERT-Transformer model based on the running state data to obtain a fault prediction model, and predict a fault of the production line equipment based on the fault prediction model.
In one implementation, the device classification module 10 includes:
the process determining unit is used for obtaining production line process information, wherein the production line process information comprises process introduction information and processing flow information of each production line;
the function determining unit is used for acquiring production line equipment corresponding to each production line process information, determining the function information of the production line equipment, and reflecting the processing effect of the production line equipment on the processing object.
In one implementation, the device classification module 10 includes:
the flow determining unit is used for determining the technological flow of the production line equipment for realizing the corresponding function information based on the function information;
the equipment classification unit is used for determining the processing difficulty level of the process flow based on the process flow, classifying the production line equipment based on the processing difficulty level and obtaining the equipment type information, wherein the equipment type information comprises first type equipment and second type equipment, and the number of equipment parts of the first type equipment is greater than that of the second type equipment.
In one implementation, the solution analysis module 10 includes:
the first scheme analysis module is used for acquiring a first fault analysis scheme corresponding to the first type equipment if the equipment type information is the first type equipment, wherein the first fault analysis scheme is a fault analysis scheme based on computer vision;
and the second scheme analysis module is used for acquiring a second fault analysis scheme corresponding to the second type equipment if the equipment type information is the second type equipment, wherein the second fault analysis scheme is a fault analysis scheme based on audio and video information or infrared thermal information.
In one implementation, the solution analysis module 10 includes:
the first information acquisition module is used for acquiring image information of the first type equipment in the running process based on the first fault analysis scheme if the production line equipment is the first type equipment, and determining the running state data based on the image information, wherein the image information comprises a motion track image and a machining process image;
and the second information acquisition module is used for acquiring audio and video information or infrared thermal sensation information of the second type equipment in the operation process based on the first fault analysis scheme if the production line equipment is the second type equipment, and determining the operation state data of the second type equipment based on the audio and video information or the infrared thermal sensation information, wherein the audio and video information comprises working noise and equipment prompt sound of the second type equipment in the operation process.
In one implementation, the fault monitoring module 10 includes:
the feature extraction unit is used for inputting the image information for reflecting the running state data into a preset convolutional neural network to perform feature extraction to obtain image features, and performing noise reduction on the image features to obtain noise-reduced image features;
the filtering processing unit is used for carrying out Kalman filtering processing on the infrared thermal sensation information for reflecting the running state data to obtain an infrared thermal sensation signal after noise reduction;
fourier transform and high-dimensional projection are used for the audio and video information for reflecting the running state data, so that an audio and video signal after noise reduction is obtained;
the audio and video processing unit is used for respectively carrying out vectorization processing on the noise-reduced image characteristics, the noise-reduced audio and video signals and the noise-reduced infrared thermal signals to obtain signals to be trained, wherein the signals to be trained comprise three vector signals with different dimensionalities;
and the model training unit is used for training the BERT-converter model according to the signal to be trained to obtain a fault prediction model.
In one implementation, the model training unit includes:
a signal input subunit, configured to input the signal to be trained into the BERT-Transformer model, where the BERT-Transformer model includes an encoder and a decoder;
the signal splicing subunit is used for receiving the signal to be trained based on the encoder of the BERT-transducer model, splicing the signal to be trained, and outputting a single vector result after a multi-layer attention mechanism;
the vector processing subunit is used for outputting a binary vector result through a DNN structure and a sigmoid activation function of the deep neural network, and the binary vector result is used for reflecting the probability of the fault of the production line equipment;
and the model training subunit is used for training the BERT-converter model based on the binary vector result and manually calibrated fault history data to obtain the fault prediction model.
The working principle of each module in the fault prediction device based on multi-mode data fusion in this embodiment is the same as the principle of each step in the above method embodiment, and will not be described here again.
Based on the above embodiment, the present invention also provides a terminal device, and a schematic block diagram of the terminal device may be shown in fig. 3. The terminal device may include one or more processors 100 (only one shown in fig. 3), a memory 101, and a computer program 102 stored in the memory 101 and executable on the one or more processors 100, e.g., a program based on fault prediction for multimodal data fusion. The one or more processors 100, when executing the computer program 102, may implement the various steps in the method embodiments of APP theme scene control. Alternatively, the functions of the modules/units in the apparatus embodiments of the present invention based on multi-modal data fusion may be implemented by one or more processors 100 when executing computer program 102, without limitation.
In one embodiment, the processor 100 may be a central processing unit (Central Processing Unit, CPU), but may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
In one embodiment, the memory 101 may be an internal storage unit of the electronic device, such as a hard disk or a memory of the electronic device. The memory 101 may also be an external storage device of the electronic device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash card (flash card) or the like, which are provided on the electronic device. Further, the memory 101 may also include both an internal storage unit and an external storage device of the electronic device. The memory 101 is used to store computer programs and other programs and data required by the terminal device. The memory 101 may also be used to temporarily store data that has been output or is to be output.
It will be appreciated by persons skilled in the art that the functional block diagram shown in fig. 3 is merely a block diagram of some of the structures associated with the present inventive arrangements and is not limiting of the terminal device to which the present inventive arrangements are applied, and that a particular terminal device may include more or fewer components than shown, or may combine some of the components, or may have a different arrangement of components.
Those skilled in the art will appreciate that implementing all or part of the above-described methods may be accomplished by way of a computer program, which may be stored on a non-transitory computer readable storage medium and which, when executed, may comprise the steps of the above-described embodiments of the methods. Any reference to memory, storage, operational database, or other medium used in embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual operation data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (8)
1. A method for predicting faults based on multi-modal data fusion, the method comprising:
acquiring functional information of production line equipment, and classifying the production line equipment based on the functional information to obtain a plurality of equipment category information;
determining a fault analysis scheme corresponding to the equipment category information according to the equipment category information, and acquiring operation state data of the production line equipment based on the fault analysis scheme, wherein the operation state data comprises data of different modes;
training a preset BERT-converter model based on the running state data to obtain a fault prediction model, and predicting the fault of the production line equipment based on the fault prediction model;
the classifying the production line equipment based on the function information to obtain a plurality of equipment category information comprises:
determining a technological process of the production line equipment for realizing corresponding function information based on the function information;
determining the processing difficulty level of the process flow based on the process flow, and classifying the production line equipment based on the processing difficulty level to obtain equipment type information, wherein the equipment type information comprises first type equipment and second type equipment, and the number of equipment parts of the first type equipment is greater than that of the second type equipment;
the determining a fault analysis scheme corresponding to the equipment category information according to the equipment category information comprises the following steps:
if the equipment type information is first type equipment, acquiring a first fault analysis scheme corresponding to the first type equipment, wherein the first fault analysis scheme is a fault analysis scheme based on computer vision;
if the equipment type information is the second type equipment, a second fault analysis scheme corresponding to the second type equipment is obtained, wherein the second fault analysis scheme is a fault analysis scheme based on audio-video information or infrared thermal information, and the audio-video information comprises working noise and equipment prompt sound of the second type equipment in the operation process.
2. The method for predicting faults based on multi-modal data fusion as claimed in claim 1, wherein the obtaining functional information of the production line equipment includes:
acquiring production line process information, wherein the production line process information comprises process introduction information and processing flow information of each production line;
and acquiring production line equipment corresponding to each production line process information, and determining the function information of the production line equipment, wherein the function information is used for reflecting the processing effect of the production line equipment on a processing object.
3. The method for predicting faults based on multi-modal data fusion according to claim 1, wherein the obtaining the operating state data of the production line equipment based on the fault analysis scheme comprises:
if the production line equipment is first type equipment, acquiring image information of the first type equipment in the running process based on the first fault analysis scheme, and determining running state data of the first type equipment based on the image information, wherein the image information comprises a motion track image and a processing process image;
if the production line equipment is second type equipment, acquiring audio and video information or infrared thermal sensation information of the second type equipment in the operation process based on the first fault analysis scheme, and determining the operation state data of the second type equipment based on the audio and video information or the infrared thermal sensation information.
4. The method for predicting faults based on multi-modal data fusion as claimed in claim 3, wherein training a preset BERT-Transformer model based on the running state data to obtain a fault prediction model includes:
inputting the image information for reflecting the running state data into a preset convolutional neural network for feature extraction to obtain image features, and denoising the image features to obtain denoised image features;
carrying out Kalman filtering processing on the infrared thermal sensation information for reflecting the running state data to obtain a noise-reduced infrared thermal sensation signal;
fourier transform and high-dimensional projection are used for the audio and video information for reflecting the running state data, so that an audio and video signal after noise reduction is obtained;
vectorizing the image characteristics after noise reduction, the audio and video signals after noise reduction and the infrared thermal signals after noise reduction respectively to obtain signals to be trained, wherein the signals to be trained comprise three vector signals with different dimensionalities;
and training the BERT-transducer model according to the signal to be trained to obtain a fault prediction model.
5. The method for predicting faults based on multi-modal data fusion as claimed in claim 4, wherein the training the BERT-Transformer model according to the signal to be trained to obtain a fault prediction model includes:
inputting the signal to be trained into the BERT-transducer model, wherein the BERT-transducer model comprises an encoder and a decoder;
the encoder based on the BERT-transducer model receives the signals to be trained, splices the signals to be trained, and outputs a single vector result after a multi-layer attention mechanism;
the single vector result is subjected to DNN structure and sigmoid activation function of a deep neural network trained based on manual annotation data, and a binary vector result is output, wherein the binary vector result is used for reflecting the probability of failure of the production line equipment;
and training the BERT-converter model based on the binary vector result and manually calibrated fault history data to obtain the fault prediction model.
6. A multi-modal data fusion-based fault prediction apparatus, the apparatus comprising:
the equipment classification module is used for acquiring the functional information of the production line equipment, classifying the production line equipment based on the functional information and obtaining a plurality of equipment category information;
the scheme analysis module is used for determining a fault analysis scheme corresponding to the equipment category information according to the equipment category information and acquiring the running state data of the production line equipment based on the fault analysis scheme, wherein the running state data comprise data of different modes;
the fault monitoring module is used for training a preset BERT-converter model based on the running state data to obtain a fault prediction model, and predicting the fault of the production line equipment based on the fault prediction model;
the device classification module comprises:
the flow determining unit is used for determining the technological flow of the production line equipment for realizing the corresponding function information based on the function information;
the equipment classification unit is used for determining the processing difficulty level of the process flow based on the process flow, classifying the production line equipment based on the processing difficulty level and obtaining equipment type information, wherein the equipment type information comprises first type equipment and second type equipment, and the number of equipment parts of the first type equipment is greater than that of the second type equipment;
the solution analysis module includes:
the first scheme analysis module is used for acquiring a first fault analysis scheme corresponding to the first type equipment if the equipment type information is the first type equipment, wherein the first fault analysis scheme is a fault analysis scheme based on computer vision;
the second scheme analysis module is used for acquiring a second fault analysis scheme corresponding to the second type equipment if the equipment type information is the second type equipment, wherein the second fault analysis scheme is a fault analysis scheme based on audio and video information or infrared thermal information, and the audio and video information comprises working noise and equipment prompt tones of the second type equipment in the running process.
7. A terminal device, characterized in that it comprises a memory, a processor and a multi-modal data fusion based fault prediction program stored in the memory and executable on the processor, said processor implementing the steps of the multi-modal data fusion based fault prediction method according to any one of claims 1-5 when executing said multi-modal data fusion based fault prediction program.
8. A computer readable storage medium, wherein a fault prediction program based on multi-modal data fusion is stored on the computer readable storage medium, and when the fault prediction program based on multi-modal data fusion is executed by a processor, the steps of the fault prediction method based on multi-modal data fusion according to any one of claims 1-5 are implemented.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310017912.9A CN115758225B (en) | 2023-01-06 | 2023-01-06 | Fault prediction method and device based on multi-mode data fusion and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310017912.9A CN115758225B (en) | 2023-01-06 | 2023-01-06 | Fault prediction method and device based on multi-mode data fusion and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115758225A CN115758225A (en) | 2023-03-07 |
| CN115758225B true CN115758225B (en) | 2023-08-29 |
Family
ID=85348379
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310017912.9A Active CN115758225B (en) | 2023-01-06 | 2023-01-06 | Fault prediction method and device based on multi-mode data fusion and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115758225B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116402219A (en) * | 2023-03-29 | 2023-07-07 | 中科航迈数控软件(深圳)有限公司 | Full life cycle operation and maintenance strategy method and device based on prediction model |
| CN116184988B (en) * | 2023-05-04 | 2023-07-21 | 中科航迈数控软件(深圳)有限公司 | Multi-mode data-based fault prediction method, device, equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110366031A (en) * | 2019-07-24 | 2019-10-22 | 长春融成智能设备制造股份有限公司 | A kind of monitoring abnormal state and method for diagnosing faults digitizing workshop MES system view-based access control model |
| CN111858265A (en) * | 2020-06-30 | 2020-10-30 | 苏州浪潮智能科技有限公司 | Storage fault prediction method, system and device of storage system |
| CN112465237A (en) * | 2020-12-02 | 2021-03-09 | 浙江正泰电器股份有限公司 | Fault prediction method, device, equipment and storage medium based on big data analysis |
| CN113568815A (en) * | 2021-08-13 | 2021-10-29 | 太仓美克斯机械设备有限公司 | Industrial manufacturing equipment fault diagnosis method and device based on big data and electronic equipment |
| CN115329876A (en) * | 2022-08-17 | 2022-11-11 | 平安科技（深圳）有限公司 | Equipment fault processing method and device, computer equipment and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10867250B2 (en) * | 2018-12-27 | 2020-12-15 | Utopus Insights, Inc. | System and method for fault detection of components using information fusion technique |
-
2023
- 2023-01-06 CN CN202310017912.9A patent/CN115758225B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110366031A (en) * | 2019-07-24 | 2019-10-22 | 长春融成智能设备制造股份有限公司 | A kind of monitoring abnormal state and method for diagnosing faults digitizing workshop MES system view-based access control model |
| CN111858265A (en) * | 2020-06-30 | 2020-10-30 | 苏州浪潮智能科技有限公司 | Storage fault prediction method, system and device of storage system |
| CN112465237A (en) * | 2020-12-02 | 2021-03-09 | 浙江正泰电器股份有限公司 | Fault prediction method, device, equipment and storage medium based on big data analysis |
| CN113568815A (en) * | 2021-08-13 | 2021-10-29 | 太仓美克斯机械设备有限公司 | Industrial manufacturing equipment fault diagnosis method and device based on big data and electronic equipment |
| CN115329876A (en) * | 2022-08-17 | 2022-11-11 | 平安科技（深圳）有限公司 | Equipment fault processing method and device, computer equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 基于多源数据的牵引供电设备故障 预测与健康运行维护管理平台;张雷等;《电力信息与通信技术》;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115758225A (en) | 2023-03-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115758225B (en) | Fault prediction method and device based on multi-mode data fusion and storage medium | |
| JP6980119B2 (en) | Speech recognition methods and their devices, devices, storage media and programs | |
| CN111477250B (en) | Audio scene recognition method, training method and device for audio scene recognition model | |
| US20180032790A1 (en) | Method for improving a fingerprint template, device and terminal thereof | |
| CN111429887B (en) | Speech keyword recognition method, device and equipment based on end-to-end | |
| EP3664084B1 (en) | Electronic device and control method therefor | |
| CN111931929A (en) | Training method and device of multi-task model and storage medium | |
| CN109559735A (en) | A kind of audio recognition method neural network based, terminal device and medium | |
| US11769492B2 (en) | Voice conversation analysis method and apparatus using artificial intelligence | |
| CN113095202A (en) | Data segmentation method and device in double-record data quality inspection | |
| CN111192576A (en) | Decoding method, speech recognition device and system | |
| US20210312290A1 (en) | Method for recommending object, computing device and computer-readable storage medium | |
| US11133022B2 (en) | Method and device for audio recognition using sample audio and a voting matrix | |
| CN112149754B (en) | Information classification method, device, equipment and storage medium | |
| US11830493B2 (en) | Method and apparatus with speech processing | |
| CN114360520A (en) | Training method, device and equipment of voice classification model and storage medium | |
| CN117523218A (en) | Label generation, training of image classification model and image classification method and device | |
| CN113436650B (en) | Baby cry identification method and device, electronic equipment and storage medium | |
| JP2024533267A (en) | Method and system for detecting content in a media stream | |
| CN112908315B (en) | Question and answer intention judging method based on sound characteristics and voice recognition | |
| CN117237359B (en) | Conveyor belt tearing detection method and device, storage medium and electronic equipment | |
| KR102539256B1 (en) | Technique for training a model to distinguish a voice of a specific speaker | |
| CN116741155A (en) | Speech recognition method, training method, device and equipment of speech recognition model | |
| US20220301550A1 (en) | Method and apparatus with keyword detection | |
| CN111883109B (en) | Voice information processing and verification model training method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |