CN115393252A

CN115393252A - Defect detection method and device for display panel, electronic equipment and storage medium

Info

Publication number: CN115393252A
Application number: CN202110572435.3A
Authority: CN
Inventors: 汪浩; 刘阳兴
Original assignee: Wuhan TCL Group Industrial Research Institute Co Ltd
Current assignee: Wuhan TCL Group Industrial Research Institute Co Ltd
Priority date: 2021-05-25
Filing date: 2021-05-25
Publication date: 2022-11-25

Abstract

The application discloses a defect detection method and device of a display panel, electronic equipment and a storage medium, wherein a panel image of the display panel to be detected is obtained, the defect positioning is carried out on the panel image through a defect positioning model, a defect area with defects in the panel image is cut out correspondingly, then the defect classification is carried out on the defect area through a defect classification model, and the defect type of the defect area is determined. On the one hand, replace traditional artifical visual detection through adopting the defect detection mode based on artificial intelligence, can avoid artificial subjective judgement to promote the accuracy of defect testing result. On the other hand, the defect detection is divided into two parts, namely defect positioning and defect classification, the defect area is firstly positioned, and then the defect area is cut out for defect classification, so that the influence of the image content outside the defect area on the defect classification can be avoided, and the accuracy of the defect detection result can be further improved.

Description

Defect detection method and device for display panel, electronic equipment and storage medium

Technical Field

The application relates to the technical field of artificial intelligence, in particular to a method and a device for detecting defects of a display panel, electronic equipment and a storage medium.

Background

As an important expression of intellectualization, the display panel is widely used in a plurality of electronic devices such as a mobile phone, a tablet computer, a television, a vehicle-mounted computer, and the like. In order to ensure that the display panel can normally display, the display panel needs to be subjected to defect detection. In the related art, a method of manual visual inspection is usually adopted to inspect possible defects of the display panel, however, the accuracy of the defect inspection result is affected by factors such as manual subjective judgment.

Disclosure of Invention

The embodiment of the application provides a method and a device for detecting defects of a display panel, an electronic device and a storage medium, which can improve the accuracy of the defect detection of the display panel.

In a first aspect, the present application provides a method for detecting defects of a display panel, including:

acquiring a panel image of a display panel to be detected;

carrying out defect positioning on the panel image through a defect positioning model, and determining a defect area with defects in the panel image;

cropping the defect area from the panel image;

and carrying out defect classification on the defect area through a defect classification model, and determining the defect type of the defect area.

In a second aspect, the present application provides a defect detecting apparatus for a display panel, including:

the image acquisition module is used for acquiring a panel image of the display panel to be detected;

the defect positioning module is used for carrying out defect positioning on the panel image through a defect positioning model and determining a defect area with defects in the panel image;

the image cutting module is used for cutting the defect area from the panel image;

and the defect classification module is used for classifying the defects of the defect area through a defect classification model and determining the defect type of the defect area.

In a third aspect, the present application provides an electronic device, including a memory and a processor, where the memory stores a computer program executable on the processor, and the processor implements the steps in the defect detection method of the display panel provided in the present application when executing the computer program.

In a fourth aspect, the present application provides a storage medium having a computer program stored thereon, where the computer program is executed by a processor to implement the steps in the method for detecting defects of a display panel provided by the present application.

According to the method and the device, the panel image of the display panel to be detected is obtained, the defect positioning is carried out on the panel image through the defect positioning model, the defect area with the defects in the panel image is determined, the defect area is cut out from the panel image, the defect classification is carried out on the defect area through the defect classification model, and the defect category of the defect area is determined. On the one hand, replace traditional artifical visual detection through adopting the defect detection mode based on artificial intelligence, can avoid artificial subjective judgement to promote the accuracy of defect testing result. On the other hand, the defect detection is divided into two parts of defect positioning and defect classification, the defect area is firstly positioned, then the defect area is cut out for defect classification, the influence of the image content outside the defect area on the defect classification can be avoided, and the accuracy of the defect detection result can be further improved.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the description below are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

FIG. 1 is a schematic view of a defect detection system for a display panel according to an embodiment of the present disclosure;

FIG. 2 is a schematic flowchart illustrating a method for detecting defects of a display panel according to an embodiment of the present disclosure;

FIG. 3 is a schematic structural diagram of a PP-YOLO model in an embodiment of the present application;

FIG. 4 is a detailed flowchart of S120 in FIG. 2;

FIG. 5 is a schematic structural diagram of a ResNet50-VD network in an embodiment of the present application;

FIG. 6 is a schematic structural diagram of a feature pyramid network in an embodiment of the present application;

FIG. 7 is a schematic diagram of a refined structure of the convolution module of FIG. 6;

FIG. 8 is a schematic diagram of the structure of the up-sampling module of FIG. 6;

FIG. 9 is a schematic view of a detailed flow of S1240 in FIG. 4;

FIG. 10 is a schematic diagram of cutting out a defect area from a panel image according to an embodiment of the present application;

fig. 11 is a schematic structural diagram of a defect detection apparatus for a display panel according to an embodiment of the present application;

fig. 12 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

It is to be appreciated that the principles of the present application are illustrated as being implemented in a suitable computing environment. The following description is based on illustrated embodiments of the application and should not be taken as limiting the application with respect to other embodiments that are not detailed herein.

Relational terms such as first and second, and the like may be used solely to distinguish one object or operation from another object or operation without necessarily limiting the actual sequential relationship between the objects or operations.

Artificial Intelligence (AI) is a theory, method, technique and application system that utilizes a digital computer or a machine controlled by a digital computer to simulate, extend and expand human Intelligence, perceive the environment, acquire knowledge and use the knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technique of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a manner similar to human intelligence. Artificial intelligence is the research of the design principle and the implementation method of various intelligent machines, so that the machines have the functions of perception, reasoning and decision making.

The artificial intelligence technology is a comprehensive subject and relates to the field of extensive technology, namely the technology of a hardware level and the technology of a software level. The artificial intelligence infrastructure generally includes technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly includes Machine Learning (ML) technology, in which Deep Learning (DL) is a new research direction in Machine Learning, and is introduced into Machine Learning to make it closer to the original target, i.e., artificial intelligence. At present, deep learning is mainly applied in the fields of computer vision, natural language processing and the like.

Deep learning is the intrinsic regularity and expression hierarchy of learning sample data, and the information obtained in these learning processes is of great help to the interpretation of data such as text, images and sound. By utilizing the deep learning technology and the corresponding training data set, network models realizing different functions can be obtained through training, for example, a gender classification model for gender classification can be obtained through training based on one training data set, an image optimization model for image optimization can be obtained through training based on another training data set, and the like.

In order to improve the efficiency and accuracy of the defect detection of the display panel, the application introduces a deep learning technology into the defect detection of the display panel, and correspondingly provides a defect detection method of the display panel, a defect detection device of the display panel, an electronic device and a storage medium. The defect detection method of the display panel can be executed by the defect detection device of the display panel or an electronic device integrated with the defect detection device of the display panel.

The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only some embodiments of the present application, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

Referring to fig. 1, the present application further provides a defect detection system for a display panel, as shown in fig. 1, the defect detection system includes an electronic device 100, and the defect detection apparatus for a display panel provided by the present application is integrated in the electronic device 100. For example, when the electronic device 100 is further configured with a camera, the configured camera may be used to directly shoot the display panel to be detected, so as to obtain a panel image of the display panel to be detected, then the defect positioning model is used to perform defect positioning on the panel image, determine a defect area with a defect in the panel image, further cut out the defect area with the defect from the panel image, and finally perform defect classification on the cut defect area through the defect classification model, so as to determine a defect category of the defect area.

The electronic device 100 may be any device equipped with a processor and having processing capability, such as a mobile electronic device with a processor, such as a smart phone, a tablet computer, a palm computer, and a notebook computer, or a stationary electronic device with a processor, such as a desktop computer, a television, and a server.

In addition, as shown in fig. 1, the defect detection system of the display panel may further include a memory 200 for storing data, for example, the electronic device 100 stores the acquired panel image, the defect area of the panel image and the corresponding defect type in the memory 200.

It should be noted that the scene schematic diagram of the defect detecting system of the display panel shown in fig. 1 is merely an example, and the defect detecting system of the display panel and the scene described in the embodiment of the present application are for more clearly illustrating the technical solution of the embodiment of the present application, and do not form a limitation on the technical solution provided in the embodiment of the present application, and as a person having ordinary skill in the art knows, with the evolution of the defect detecting system of the display panel and the occurrence of a new service scene, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

The following are detailed descriptions. The numbers in the following examples are not intended to limit the order of preference of the examples.

Referring to fig. 2, fig. 2 is a schematic flowchart illustrating a defect detection method for a display panel according to an embodiment of the present disclosure, and as shown in fig. 2, the flow of the defect detection method for a display panel according to the present disclosure is as follows:

in S110, a panel image of the display panel to be detected is acquired.

It should be noted that the display panel to be detected refers to a display panel that needs to be subjected to defect detection, and the defect detection includes detecting an area of the display panel where a defect exists and a corresponding defect type. The defect category includes visual defects such as breakage defect and scratch defect, and functional defects such as bright point defect and line defect.

In this embodiment, the panel image may be obtained by shooting the display panel to be detected through image acquisition equipment such as a camera, or may be obtained by obtaining a pre-shot panel image of the display panel to be detected, where the obtained panel image is used in subsequent defect detection of the display panel to be detected.

In S120, the defect localization model is used to localize the defect of the panel image, and the defect area with the defect in the panel image is determined.

It should be noted that the defect localization model is a model for performing defect localization on an input panel image, and the defect localization is a model for identifying a defect region in the panel image where a defect exists. The model structure and the training mode of the defect localization model are not particularly limited, and can be selected by those skilled in the art according to actual needs.

In this embodiment, after the panel image of the display panel to be detected is acquired, the acquired panel image may be input into the defect positioning model, and the defect positioning is performed on the acquired panel image through the defect positioning model, so as to determine a defect area in which a defect exists in the panel image.

In some embodiments, the defect localization model is obtained by training an L1 loss function and an intersection-to-parallel ratio loss function based on a PP-YOLO model, referring to fig. 3, the PP-YOLO model includes a backbone network, a neck network, and a detection head network, referring to fig. 4, the defect localization is performed on the panel image through the defect localization model, and the determining of the defect area having the defect in the panel image includes:

s1210, performing feature coding on the panel image through a backbone network to obtain a plurality of feature maps with different scales;

s1220, fusing a plurality of feature maps with different scales through a neck network to obtain a fused feature map;

s1230, performing feature decoding on the fusion feature map through the detection head network to obtain a positioning result for describing the position of the defect area;

and S1240, determining a defect area with defects in the panel image according to the positioning result.

The PP-YOLO model is an improved YOLOV3 model, and the recognition accuracy and the recognition efficiency of the PP-YOLO model are superior to those of a YOLOV4 model. As shown in fig. 3, the PP-YOLO model consists of three parts, namely a backbone network, a neck network and a detector network. The main network is a basis of the model and is used for carrying out feature coding on a plurality of different scales, and correspondingly coding to obtain a plurality of feature maps of different scales, wherein the feature maps include but are not limited to features of shape edges, features of light and dark colors and the like; the neck network is connected with the backbone network and the detection head network and is used for carrying out feature fusion on a plurality of feature maps with different scales obtained by encoding the backbone network and taking the fused feature maps obtained by fusion as the input of the detection head network; the detection head network is used for carrying out feature decoding on the input fusion feature map, and correspondingly obtaining a positioning result for describing the position of the defect area.

In this embodiment, the PP-YOLO model is trained on a pre-obtained training set by using an L1 loss function and an intersection-to-parallel ratio loss function until a preset stop condition is met, so as to obtain a defect localization model. The preset stopping condition may be configured to enable the iteration number of the parameters of the PP-YOLO model in the training process to reach a preset number, or configured to enable the L1 loss function and the intersection ratio loss function to converge.

Where the L1 penalty function is also referred to as the minimum absolute deviation or minimum absolute error, the effect is to minimize the sum of the absolute differences of the tag value and the predicted value.

The intersection ratio loss function directly uses the intersection ratio of the real defect frame and the predicted defect frame as the loss function.

Correspondingly, in this embodiment, when the defect location is performed on the panel image through the defect location model and the defect area with the defect in the panel image is determined, the feature coding is performed on the panel image through the backbone network to obtain a plurality of feature maps with different scales, then the plurality of feature maps with different scales are fused through the neck network to obtain a fused feature map correspondingly, and finally the feature decoding is performed on the fused feature map through the detection head network to obtain the location result for describing the position of the defect area.

In some embodiments, the backbone network comprises a ResNet50-VD network, the neck network comprises a feature pyramid network, and the detector head network comprises a YOLOV3 detector head network.

It should be noted that the ResNet50-VD network refers to a ResNet-D network having 50 convolutional layers. ResNet-D networks can provide improved accuracy without substantially increasing the computational effort compared to ResNet-B and ResNet-C.

For example, referring to fig. 5, the resnet50-VD network can be divided into 5 feature extraction layers with different scales, namely, a C1 feature extraction layer, a C2 feature extraction layer, a C3 feature extraction layer, a C4 feature extraction layer, and a C5 feature extraction layer. The C1 feature extraction layer, the C2 feature extraction layer, the C3 feature extraction layer, the C4 feature extraction layer and the C5 feature extraction layer are used for sequentially carrying out feature coding on the defective image to correspondingly obtain 5 feature maps with different scales.

Referring to fig. 6, the feature pyramid network may be divided into 3 layers, which are a first feature pyramid layer formed by the first convolution layer and the two convolution modules, a second feature pyramid layer formed by the second convolution layer and the two convolution modules, a third pyramid layer formed by the third convolution layer and the two convolution modules, and an upsampling module and a Concat module (formed by the upsampling module and the Concat module in fig. 6) for horizontally connecting the first feature pyramid layer and the second feature pyramid layer

Representation) for connecting the second feature pyramid layer and the third feature pyramid layer laterally.

It should be noted that, in fig. 6, C5 indicates that the first feature pyramid layer is connected to the C5 feature extraction layer, C4 indicates that the second feature pyramid layer is connected to the C4 feature extraction layer, C3 indicates that the third feature pyramid layer is connected to the C3 feature extraction layer, P5 indicates that the feature map extracted by the C5 feature extraction layer is processed by the first feature pyramid layer to obtain the feature map, P4 indicates that the feature map extracted by the C4 feature extraction layer is processed by the second feature pyramid layer to obtain the feature map, and P3 indicates that the feature map extracted by the C3 feature extraction layer is processed by the third feature pyramid layer to obtain the feature map. In addition, as shown in fig. 6, the feature map Concat extracted by the P5 and C4 feature extraction layer after up-sampling by the up-sampling module is used as the input of the second feature pyramid layer, and the feature map Concat extracted by the P4 and C3 feature extraction layer after up-sampling by the up-sampling module is used as the input of the third feature pyramid layer.

Wherein the first convolutional layer comprises 512 channels of 1x1 convolution, the second convolutional layer comprises 512 256 channels of 1x1 convolution, and the third convolutional layer comprises 256 channels of 128 1x1 convolution.

In addition, referring to fig. 7, the convolution module includes 3x3 convolution layers and 1x1 convolution layers, where the number of 3x3 convolution kernels in the 3x3 convolution layers is the same as the number of input channels, the number of 3x3 convolution kernel channels is twice of the number of the input channels, the number of 1x1 convolution kernels in the 1x1 convolution layers is twice of the number of the input channels, and the number of 1x1 convolution kernel channels is half of the number of the input channels.

Referring to fig. 8, the upsampling module includes a 1x1 convolutional layer and a 2 times upsampling layer, where the number of 1x1 convolutional cores in the 1x1 convolutional layer is the same as the number of input channels, and the number of 1x1 convolutional core channels is half of the number of the input channels.

As can be seen from the above, in this embodiment, the panel image is sequentially feature-coded by the C1 feature extraction layer, the C2 feature extraction layer, the C3 feature extraction layer, the C4 feature extraction layer, and the C5 feature extraction layer, so as to obtain 5 kinds of feature maps with different scales correspondingly. And then, further selecting the feature graphs output by the C3 feature extraction layer, the C4 feature extraction layer and the C5 feature extraction layer through a feature pyramid network to perform feature fusion, and correspondingly obtaining a feature graph pyramid (namely a fusion feature graph) consisting of 3 feature graphs with different scales.

In addition, the YOLOV3 detector head network comprises 3 detector heads corresponding to different scale characteristic diagrams, and the detector heads have the same structure and comprise a 3x3 convolutional layer and a 1x1 convolutional layer. The 3 detection heads are respectively used for carrying out feature decoding on the feature maps of 3 different scales in the feature map pyramid, so that the positioning result of describing the positions of the defect regions at different scales is obtained. Wherein the positioning result comprises a defect frame for describing the position of the defect area.

In this embodiment, a ResNet50-VD network is constructed as a backbone network of the PP-YOLO model, a feature pyramid network is constructed as a neck network of the PP-YOLO model, and a YOLOV3 detector network is constructed as a detector network of the PP-YOLO model. For example, a paddlepaddleplatform may be used to implement the building of each aforementioned network part.

In some embodiments, some or all of the convolutional layers in the ResNet50-VD network are replaced with deformable convolutional layers.

It should be noted that the deformable convolution layer is characterized in that the convolution kernel thereof is additionally added with a learnable offset parameter on each element. The convolution kernel can adjust the receptive field of convolution in the learning process, so that the image features can be better coded.

Therefore, in this embodiment, all convolutional layers in the ResNet50-VD network can be replaced with deformable convolutional layers to obtain the best feature coding effect; or

Partial convolutional layers in a ResNet50-VD network can be replaced with deformable convolutional layers to balance the effect of feature coding against computational overhead.

Exemplarily, in this embodiment, the convolutional layer in the C5 feature extraction layer of the ResNet50-VD network is replaced by a deformable convolutional layer, so as to achieve the purposes of introducing less computation overhead and improving the encoding effect.

In some embodiments, the 1x1 convolutional layers in the feature pyramid network and the first layer convolutional layers in the YOLOV3 detector head network are replaced with coordinate convolutional layers.

It can be understood that the convolution operation has a translation and other degeneration, so that a uniform convolution kernel parameter is shared at different positions of the object image of the convolution operation, but the coordinates of the current feature in the object image cannot be perceived. The coordinate convolution layer represents the coordinates of the pixel points of the characteristic diagram by newly adding a corresponding channel in the object image of the convolution operation, so that the coordinate is sensed to a certain extent to improve the operation precision.

Therefore, in this embodiment, only the first layer of the convolutional layer in the 1x1 convolutional layer and the YOLOV3 detector head network, which are the feature pyramid networks, is replaced by the coordinate convolutional layer, so as to achieve the purposes of introducing less computation overhead and improving the computation accuracy.

In some embodiments, the ResNet50-VD network and the feature pyramid network are connected by a spatial pyramid pooling layer.

It should be noted that the spatial pyramid pooling layer is proposed by SPPNet and comprises a plurality of different scale pooling windows for extracting different scale pooling features and combining the extracted different scale pooling features together as an output feature.

In this embodiment, a spatial pyramid pooling layer is added between the ResNet50-VD network and the feature pyramid network, and the ResNet50-VD network and the feature pyramid network are connected by the spatial pyramid pooling layer, so that before feature fusion of the feature pyramid network, a plurality of feature maps of different scales output by the ResNet50-VD network are respectively subjected to spatial pyramid pooling by the spatial pyramid pooling layer, the perception fields of the feature maps are improved, and then, the feature pyramid network performs feature fusion on the plurality of feature maps of different scales subjected to spatial pyramid pooling to obtain a fused feature map.

In some embodiments, the YOLOV3 detects that a classified branch in the head network is replaced with a merge ratio predicted branch.

It should be noted that, in the present embodiment, the defect is located only by using the YOLOV3 detector head network, and the defect is classified without using the YOLOV3 detector head network, so that the classification branch in the YOLOV3 detector head network, which implements the classification function, can be deleted. Wherein, the output of the positioning branch of the YOLOV3 detection head network comprises a defect frame for indicating the position of the defect area with the defect and the confidence of the defect frame.

In addition, because the YOLOV3 test head network does not consider the positioning accuracy of the defect frame, in order to improve the positioning accuracy of the defect frame, in this embodiment, the classification branch in the YOLOV3 test head network is replaced by an intersection ratio prediction branch, and the intersection ratio prediction branch is used for predicting the intersection ratio of the defect frame to obtain the prediction intersection ratio of the defect frame for subsequent defect frame screening.

Referring to fig. 9, in some embodiments, S1240 includes:

s12410, adjusting the confidence coefficient of the defect frame according to the prediction cross-over ratio of the defect frame;

and S12420, determining a defect region according to the defect frame of which the adjusted confidence coefficient is greater than or equal to the first preset confidence coefficient.

In this embodiment, since the merging ratio prediction branch is introduced into the YOLOV3 detection head network, the positioning result output by the YOLOV3 detection head network will include the defect frame, the confidence of the defect frame, and the prediction merging ratio of the defect frame.

Correspondingly, when a defect area with defects in the panel image is determined according to the positioning result of the detection head network, firstly, the confidence coefficient of the defect frame is adjusted according to the prediction intersection of the defect frame and the comparison.

For example, for a defect box, its prediction cross-over ratio and confidence may be multiplied, and the result of the multiplication is used as its adjusted confidence.

For another example, for a defect box, the predicted intersection ratio and the confidence may be added, and the added result may be used as the adjusted confidence.

As described above, after adjusting the confidence of the defect frames according to the predicted intersection and comparison of the defect frames, the defect regions are further determined according to the defect frames with the adjusted confidence greater than or equal to the preset confidence, that is, the defect frames with the adjusted confidence greater than or equal to the preset confidence are retained, and the regions indicated by the defect frames are determined as the defect regions.

In the embodiment, the information of the prediction intersection ratio is fused in the confidence coefficient of the defect frame, so that the positioning accuracy of the defect frame is improved.

In some embodiments, S1230 comprises:

performing feature decoding on the fusion feature map through a YOLOV3 detection head network to obtain logits values of the position coordinates of the defect frames;

inputting the logits value into an activation function to obtain an activation value of the logits value;

correcting the activation value according to a preset correction coefficient to obtain a correction value;

and obtaining the position coordinates of the defect frame according to the correction value and the preset scaling coefficient.

In this embodiment, when the fused feature map is subjected to feature decoding by the head network to obtain a positioning result for describing the position of the defect region, the fused feature map is subjected to feature decoding by the YOLOV3 head network first to obtain a logts value of the position coordinate of the defect frame, where the logts value represents an original predicted value of the YOLOV3 head network on the position coordinate of the defect frame.

As described above, after the logits value of the position coordinates of the defective frame is obtained, the logits value of the position coordinates of the defective frame is input to the activation function to be activated, and the activation value is obtained. The selection of the activation function is not particularly limited, and may be selected by a person skilled in the art according to actual needs, for example, the present embodiment selects a Sigmoid function as the activation function.

In addition, after the activation value of the logits value is obtained, the activation value is further corrected according to a preset correction coefficient to obtain a correction value, and finally, the position coordinate of the defect frame is obtained according to the correction value and a preset scaling coefficient.

It should be noted that the positioning principle of the YOLOv3 detection head network is to divide the image into a plurality of grids for positioning, and the above process of obtaining the position coordinate of the defect frame according to the logits value of the position coordinate of the defect frame can be directly implemented by using Grid Sensitive algorithm, which is expressed as the following formula (1):

wherein x represents an abscissa value in the position coordinates of the defective frame, y represents an ordinate value in the position coordinates of the defective frame, σ () represents an activation function, and p _x Values of logits, p, representing the abscissa in coordinates of the position of the defective frame _y Values of logits, g, representing the ordinate in the coordinates of the position of the defective frame _x Is expressed as (σ (p) _x )，σ(p _y ) Abscissa value, g, of the top left vertex of the grid in the panel image _y Is expressed as (σ (p) _x )，σ(p _y ) A represents a preset correction coefficient (an empirical value may be taken by those skilled in the art according to actual needs, for example, the value in this embodiment is 1.05), and S represents a preset scaling coefficient, which is determined according to the size of the panel image.

In the embodiment, a Grid Sensitive algorithm is introduced, the position coordinates of the defect frame are determined through the Grid Sensitive algorithm, and the positioning accuracy of the defect frame can be improved.

In some embodiments, before S12420, further including:

constructing a calculation matrix for calculating the intersection and parallel ratio of all the prediction frames, and obtaining the intersection and parallel ratio of all the prediction frames according to parallel calculation of the calculation matrix;

aiming at a prediction frame, reducing the confidence of other prediction frames with the intersection ratio reaching the preset intersection ratio;

and deleting the prediction frame with the confidence coefficient smaller than a second preset confidence coefficient, wherein the second preset confidence coefficient is smaller than the first preset confidence coefficient.

It should be noted that for a real defect, the network of detection heads may output a plurality of defect frames indicating the defect, which results in the existence of redundant defect frames, and the redundant defect frames need to be removed.

In this embodiment, redundant defect frames are also removed accordingly. Constructing a calculation matrix for calculating the intersection and parallel ratio of all the prediction frames, and obtaining the intersection and parallel ratio of all the prediction frames according to parallel calculation of the calculation matrix; aiming at a prediction frame, reducing the confidence coefficient of other prediction frames with the intersection ratio reaching the preset intersection ratio; and deleting the prediction frame with the confidence coefficient smaller than a second preset confidence coefficient, wherein the second preset confidence coefficient is smaller than the first preset confidence coefficient.

It should be noted that the above process of removing redundant defect frames can be directly implemented by using a matrix non-maximum suppression algorithm, which is an implementation idea of parallelizing soft non-maximum suppression. Redundant defect boxes can be efficiently removed by employing a matrix non-maximum suppression algorithm.

In some embodiments, a DropBlock layer is disposed in the feature pyramid network.

It should be noted that, when discarding the features, the DropBlock layer does not discard the features in the form of feature points, but discards the features of a certain region in the feature map in a centralized manner, so as to reduce overfitting and accordingly improve the generalization capability of the model. In this embodiment, to avoid excessive accuracy degradation, a DropBlock layer is only set in the feature pyramid network.

Illustratively, a DropBlock layer may be placed in the first convolution module of each feature pyramid layer of the feature pyramid network. For example, it may be placed after the 1x1 convolutional layer of the convolutional module.

In some embodiments, further comprising:

and optimizing the training process of the PP-YOLO model by adopting an exponential moving average algorithm.

In this embodiment, in the training process of the PP-YOLO model, the moving average of the training parameters is maintained by using the exponential moving average algorithm, so as to obtain a training result better than the final training value.

Wherein, the exponential moving average algorithm uses exponential decay to calculate the moving average of the training parameters, and for each parameter W in the PP-YOLO model, one hidden parameter W is reserved _EMA Expressed as:

W _EMA ＝λW _EMA +(1-λ)W (2)

in the formula (2), λ represents an attenuation coefficient, which can be taken by a person skilled in the art according to actual needs, for example, λ may be configured to be 0.9998.

By an exponential moving average algorithm, gradient jitter caused by difficult samples or error samples is smoothed by using a gradient historical weighted average value, so that the training process is smoother and more stable.

In S130, a defective region is cut out from the panel image.

In this embodiment, after the defect area having the defect in the panel image is determined, the defect area is further cut out from the panel image. It should be noted that there may be one or more determined defect regions.

For example, referring to fig. 10, for a panel image, defect location is performed through defect location model alignment, a defect area with a defect in the panel image is determined, and the defect area is cut out from the panel image for subsequent defect classification.

In S140, the defect classification model classifies defects in the defect area, and determines a defect type of the defect area.

The defect classification model is a model for defect classification, and the defect classification identifies the defect type of the defect area. In the embodiment, the defect types include an appearance defect such as a breakage defect or a scratch defect, and a functional defect such as a bright point defect or a line defect.

It should be noted that, in the present embodiment, the specific architecture of the defect classification model is not particularly limited, and can be selected by those skilled in the art according to actual needs, including but not limited to VGG Net, googleNet, resNet, densneet, and the like.

For example, in the embodiment, the DenseNet201 is used as the infrastructure for training to obtain the defect classification model.

In this embodiment, after the defect area is cut out from the panel image, the defect classification model further classifies the defect of the defect area, thereby determining the defect type of the defect area.

In some embodiments, the defect classification model includes a feature encoding network and a feature decoding network, the feature encoding network including at least a depth separable convolutional layer, S140 includes:

carrying out feature coding on the defect image through the depth separable convolution layer to obtain a feature map of the defect image;

performing feature decoding on the feature map through a feature decoding network to obtain a classification result for describing defect categories of the defect area;

and determining the defect type of the defect area according to the classification result.

In this embodiment, the defect classification model is functionally divided into two parts, which are a feature coding network for feature coding (which can be understood as extracting features from input) and a feature decoding network for feature decoding (which can be understood as further performing feature optimization and classification processing on coded features). The feature coding network at least comprises a depth separable convolutional layer, and the depth separable convolutional layer can decompose a three-dimensional matrix into two-dimensional matrix multiplication, so that the calculation amount is reduced on the premise of reducing the parameter number.

Correspondingly, when the defect classification model is used for classifying the defects of the defect area and determining the defect type of the defect area, firstly, the feature coding is carried out on the defect image through the depth separable convolution layer to realize the feature extraction of the defect image and obtain the feature map of the defect image, then, the feature decoding is carried out on the feature map through the feature decoding network to realize the optimization and classification of the feature map, and the classification result for describing the defect type of the defect area is correspondingly obtained. For example, the classification result includes a numerical value for characterizing the defect class and a corresponding confidence level.

As described above, after the classification result for describing the defect class of the defective area is obtained, the defect class of the defective area can be determined according to the classification result.

For example, the classification result output by the feature decoding network includes a value and a corresponding confidence level for characterizing the defect type a, a value and a corresponding confidence level for characterizing the defect type B, and a value and a corresponding confidence level for characterizing the defect type C. If the confidence corresponding to any of the values reaches the confidence threshold of the defect type characterized by the value (which can be obtained through post-processing tests in the training stage, for example, for a defect type, the confidence when the accuracy is greater than or equal to 90% is configured as the confidence threshold of the defect type), the defect type of the defect region is determined to be the defect type characterized by the value.

In some embodiments, the depth-separable convolutional layer includes channel-by-channel convolutional sublayers and point-by-point convolutional sublayers, and the feature coding is performed on the defect image through the depth-separable convolutional layer to obtain a feature map of the defect image, including:

respectively performing convolution operation on a plurality of channels of the defect image through the channel-by-channel convolution sublayer to obtain a plurality of convolution results;

and performing weighted combination on the plurality of convolution results through the point-by-point convolution layer to obtain a feature map of the defect image.

Wherein, one convolution kernel in the channel-by-channel convolution layer is only responsible for the convolution of one channel, and the number of the channels of the convolution result generated in the process is completely consistent with the number of the input channels. For example, for a 5 × 5 pixel and RGB three-channel defective image, the convolution operation is performed on the RGB three channels by 3 convolution kernels in the channel-by-channel convolution sublayer, and the convolution result of the R channel, the convolution result of the G channel, and the convolution result of the B channel are correspondingly obtained.

The operation of the point-by-point convolution layer is similar to the conventional convolution operation, the size of a convolution kernel of the point-by-point convolution layer is 1 multiplied by M, the value of M is the number of input channels, and the point-by-point convolution kernel is used for carrying out weighted combination on the input convolution result in the depth direction so as to correspondingly obtain a feature map of a defect image. For example, the convolution sub-layers channel by channel perform weighted combination on convolution results of the R channel, the G channel, and the B channel of the defect image in the depth direction, and accordingly obtain the feature map of the defect image.

In some embodiments, a depth separable convolution based defect classification model is trained as follows:

acquiring positive sample images of different defect types and acquiring negative sample images without defects;

and training a defect classification model by adopting a focusing loss function according to the positive sample image and the negative sample image.

In this embodiment, sample defect images of different defect types are obtained according to different defect types that may exist in the display panel and are recorded as positive sample images. For example, for a defect type, a display panel with a defect of the defect type is photographed by a camera, and a positive sample image is obtained accordingly.

In addition, a sample image without defects can be obtained and recorded as a negative sample image. For example, a display panel without defects is shot by a camera, and a negative sample image is obtained correspondingly.

It should be noted that, because the probability of the real occurrence of the defects of different categories is different, for example, some category defects occur more often, and some category defects occur more difficultly, the number of positive sample images obtained for different defect categories is unbalanced, and the accuracy of the model is ultimately affected.

Therefore, in this embodiment, when the defect classification model based on the depth separable convolution is trained, the focus loss function is used to train the defect classification model based on the depth separable convolution according to the positive sample image and the negative sample image until the preset stop condition is satisfied. The preset stop condition may be configured to set the number of iterations of the defect classification model parameter based on the depth separable convolution to a preset number in the training process, or may be convergence of a focus loss function.

Wherein the focus loss function can be expressed as formula (3):

y is a label of the sample image, y =1 represents the positive sample image, y =0 represents the negative sample image, y' is a predicted value (representing the defect type here) through a sigmoid activation function, and γ represents a balance factor, and a person skilled in the art can take any positive value according to actual needs, so that the weight of the positive sample images with a large number of defect types in training is reduced, and the weight of the positive sample images with a small number of defect types in training is correspondingly increased, thereby achieving the purpose of improving the model accuracy.

In some embodiments, to reduce the model volume and increase the operation speed, the method for detecting defects of a display panel provided by the present application further includes:

evaluating a degree of influence of each convolution core in the depth-separable convolution layers on feature coding of the depth-separable convolution layers;

and determining a target convolution kernel needing to be deleted in the depth separable convolution layer according to the influence degree corresponding to each convolution kernel, and deleting the target convolution kernel from the depth separable convolution layer.

It should be noted that the depth-separable convolutional layers include a plurality of convolutional kernels, and different convolutional cores have different degrees of influence on feature encoding of the depth-separable convolutional layers, wherein, for a convolutional kernel, the degree of influence of the convolutional kernel on feature encoding of the depth-separable convolutional layers can be understood in a colloquial way as: before and after deleting the convolution kernel from the depth separable convolution layer, the depth separable convolution layer carries out the difference degree of the feature map obtained by carrying out feature coding on the same defect image, and the larger the difference degree is, the smaller the influence degree is. Therefore, the convolution kernel with smaller influence degree in the depth separable convolution layer can be deleted, thereby achieving the purposes of reducing the model volume and improving the operation speed.

In this embodiment, the degree of influence of each convolution core in the depth separable convolution layers on feature encoding of the depth separable convolution layers is first evaluated. And then, determining the convolution kernels needing to be deleted in the depth separable convolution layer according to the influence degree corresponding to each convolution kernel, and marking as target convolution kernels. For example, a threshold value for characterizing the degree of influence with a small degree of influence (which can be performed by those skilled in the art according to actual needs) may be configured in advance, and the convolution kernel with the degree of influence smaller than the threshold value may be determined as the target convolution kernel.

As above, after the target convolution kernel is determined, the target convolution layer is deleted from the depth separable convolution layers accordingly, resulting in a new depth separable layer.

Illustratively, in particular implementations, pruning algorithms may be employed directly to remove convolution kernels in the depth separable convolution layer. The pruning algorithm is not specifically limited, and can be selected by those skilled in the art according to actual needs, including but not limited to α - β pruning algorithm, cart pruning algorithm, etc.

In some embodiments, in order to further reduce the model volume and increase the operation speed, after the defect classification model is trained by using the focus loss function, the method further includes:

and carrying out quantization processing on the defect classification model so as to convert the defect classification model from a floating point type to an integer type.

It can be understood that the trained models are stored in a floating-point manner, which usually requires tens of hundreds of megabits of storage space, and from the operation perspective, the operation of floating-point data occupies a large amount of operation resources. Therefore, the defect classification model trained in this embodiment is further subjected to quantization processing to convert the defect classification model from a floating point type to an integer type.

It should be noted that, in the present embodiment, there is no specific limitation on what quantization method is used, and the selection may be made by those skilled in the art according to actual needs, for example, the model quantization may be implemented by binnary connect, and the core of the present embodiment is to use binary weights instead of floating point weights.

In order to better implement the defect detection method of the display panel provided in the embodiments of the present application, the embodiments of the present application further provide a device based on the defect detection method of the display panel, which is denoted as a defect detection device of the display panel. The terms are the same as those in the defect detection method of the display panel, and please refer to the description in the above method embodiment for details.

Referring to fig. 11, fig. 11 is a schematic structural diagram of a defect detecting apparatus of a display panel according to an embodiment of the present disclosure, which may include an image obtaining module 310, a defect locating module 320, an image cropping module 330, and a defect classifying module 340, wherein,

an image obtaining module 310, configured to obtain a panel image of a display panel to be detected;

the defect positioning module 320 is used for positioning the defects of the panel image through the defect positioning model and determining the defect area with the defects in the panel image;

an image cropping module 330 for cropping the defect area from the panel image;

and the defect classification module 340 is configured to classify the defect of the defect area through a defect classification model, and determine a defect type of the defect area.

Optionally, in some embodiments, the defect classification model includes a feature coding network and a feature decoding network, the feature coding network includes at least a depth separable convolution layer, and the defect localization module 320 is configured to:

Optionally, in some embodiments, the depth separable convolutional layer includes channel-by-channel convolutional sublayers and point-by-point convolutional sublayers, and the defect localization module 320 is configured to:

and carrying out weighted combination on the plurality of convolution results through the point-by-point convolution layer to obtain a characteristic diagram.

Optionally, in some embodiments, the defect detecting apparatus of the display panel further includes a pruning module, configured to:

evaluating a degree of influence of each convolution core in the depth separable convolution layers on feature encoding of the depth separable convolution layers;

Optionally, in some embodiments, the defect detecting apparatus of the display panel further includes a training module, configured to:

Optionally, in some embodiments, the defect detecting apparatus of the display panel further includes a quantization module, configured to:

and performing quantization processing on the defect classification model to convert the defect classification model from a floating point type to an integer type.

Optionally, in some embodiments, the defect localization model is obtained by training using an L1 loss function and an intersection-to-parallel ratio loss function based on a PP-YOLO model, where the PP-YOLO model includes a backbone network, a neck network, and a detection head network, and the defect localization module 320 is configured to:

carrying out feature coding on the panel image through a backbone network to obtain a plurality of feature maps with different scales;

fusing a plurality of feature maps with different scales through a neck network to obtain a fused feature map;

performing feature decoding on the fusion feature map through a detection head network to obtain a positioning result for describing the position of the defect area;

and determining the defect area according to the positioning result.

Optionally, in some embodiments, the backbone network comprises a ResNet50-VD network, the neck network comprises a feature pyramid network, and the detector head network comprises a YOLOV3 detector head network.

Optionally, in some embodiments, some or all of the convolutional layers in the ResNet50-VD network are replaced with deformable convolutional layers.

Optionally, in some embodiments, the 1x1 convolutional layers in the feature pyramid network and the first layer convolutional layers in the YOLOV3 detector head network are replaced with coordinate convolutional layers.

Optionally, in some embodiments, the ResNet50-VD network and the feature pyramid network are connected by a spatial pyramid pooling layer.

Optionally, in some embodiments, the classification branch in the YOLOV3 detector head network is replaced with a merge ratio prediction branch.

Optionally, in some embodiments, the positioning result includes a defect frame, a predicted intersection ratio of the defect frame, and a confidence of the defect frame, and the defect positioning module 320 is configured to:

adjusting the confidence coefficient of the defect frame according to the forecast intersection ratio of the defect frame;

and determining a defect region according to the defect frame of which the adjusted confidence coefficient is greater than or equal to a first preset confidence coefficient.

Optionally, in some embodiments, the defect localization module 320 is configured to:

performing feature decoding on the fusion feature map through a YOLOV3 detection head network to obtain a decoded coordinate logits value;

inputting the coordinate logits value into an activation function to obtain a coordinate activation value of the coordinate logits value;

correcting the coordinate activation value according to a preset correction coefficient to obtain a coordinate correction value;

and obtaining the position coordinates of the defect frame according to the coordinate correction value and the preset scaling coefficient.

Optionally, in some embodiments, the defect localization module 320 is further configured to:

constructing a calculation matrix for calculating the intersection ratio of every two prediction frames, and performing parallel calculation according to the calculation matrix to obtain the intersection ratio of every two prediction frames;

aiming at a prediction frame, reducing the confidence coefficient of other prediction frames with the intersection ratio reaching the preset intersection ratio;

Optionally, in some embodiments, a DropBlock layer is disposed in the feature pyramid network.

Optionally, in some embodiments, the second training module is further configured to:

The above modules can be implemented in the foregoing embodiments, and are not described in detail herein.

The embodiment of the present application further provides an electronic device, which includes a memory and a processor, where the processor is configured to execute the steps in the method for detecting a defect of a display panel provided in this embodiment by calling a computer program stored in the memory.

Referring to fig. 12, fig. 12 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure.

The electronic device may include components such as a network interface 410, memory 420, processor 430, and a screen assembly. Those skilled in the art will appreciate that the electronic device configuration shown in fig. 12 does not constitute a limitation of the electronic device and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.

The network interface 410 may be used to make network connections between devices.

Memory 420 may be used to store computer programs and data. Memory 420 stores computer programs having executable code embodied therein. The computer program may be divided into various functional modules. The processor 430 executes various functional applications and data processing by executing computer programs stored in the memory 420.

The processor 430 is a control center of the electronic device, connects various parts of the entire electronic device using various interfaces and lines, performs various functions of the electronic device and processes data by operating or executing the computer program stored in the memory 420 and calling the data stored in the memory 420, thereby performing overall control of the electronic device.

In the embodiment of the present application, the processor 430 in the electronic device loads the executable code corresponding to one or more computer programs into the memory 420 according to the following instructions, and the processor 430 executes the steps in the defect detection method of the display panel provided by the present application, such as:

acquiring a panel image of a display panel to be detected;

cutting out a defect area from the panel image;

and classifying the defects of the defective area through a defect classification model, and determining the defect type of the defective area.

It should be noted that the electronic device provided in the embodiment of the present application and the method for detecting a defect of a display panel in the foregoing embodiment belong to the same concept, and specific implementation processes thereof are described in the foregoing related embodiments, and are not described herein again.

The present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program stored in the storage medium is executed on a processor of an electronic device provided in an embodiment of the present application, the processor of the electronic device is caused to execute the steps in the defect detection method for a display panel provided in the present application. The storage medium may be a magnetic disk, an optical disk, a Read Only Memory (ROM), a Random Access Memory (RAM), or the like.

The method, the apparatus, the storage medium, and the electronic device for detecting defects of a display panel provided by the present application are described in detail above, and a specific example is applied in the present application to explain the principle and the implementation of the present application, and the description of the above embodiment is only used to help understand the method and the core idea of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.

Claims

1. A method for detecting defects of a display panel is characterized by comprising the following steps:

acquiring a panel image of a display panel to be detected;

cropping the defect area from the panel image;

2. The method of claim 1, wherein the defect classification model comprises a feature coding network and a feature decoding network, the feature coding network comprising at least a depth separable convolutional layer, the classifying the defect region by the defect classification model, determining the defect class of the defect region, comprising:

performing feature coding on the defect image through the depth separable convolution layer to obtain a feature map of the defect image;

performing feature decoding on the feature map through the feature decoding network to obtain a classification result for describing the defect category of the defect area;

3. The method of claim 2, wherein the depth-separable convolutional layer comprises a channel-by-channel convolutional sublayer and a point-by-point convolutional sublayer, and wherein the feature encoding of the defect image by the depth-separable convolutional layer to obtain a feature map of the defect image comprises:

performing convolution operation on a plurality of channels of the defect image through the channel-by-channel convolution sublayer to obtain a plurality of convolution results;

and carrying out weighted combination on the plurality of convolution results through the point-by-point convolution layer to obtain the feature map.

4. The method of claim 2, wherein the method further comprises:

evaluating a degree of influence of each convolution core in the depth separable convolution layers on feature coding of the depth separable convolution layers;

5. The method of any of claims 1-4, wherein the defect classification model is generated by:

and training the defect classification model by adopting a focusing loss function according to the positive sample image and the negative sample image.

6. The method of claim 5, wherein after the training of the defect classification model using the focus loss function, further comprises:

7. The method of claim 1, wherein the defect localization model is obtained by training an L1 loss function and an intersection-to-parallel ratio loss function based on a PP-YOLO model, the PP-YOLO model includes a backbone network, a neck network and a detection head network, and the defect localization for the panel image through the defect localization model to determine a defect region with a defect in the panel image includes:

performing feature coding on the panel image through the backbone network to obtain a plurality of feature maps with different scales;

fusing the feature maps with different scales through the neck network to obtain a fused feature map;

performing feature decoding on the fused feature map through the detection head network to obtain a positioning result for describing the position of the defect area;

and determining the defect area according to the positioning result.

8. The method of claim 7, wherein the backbone network comprises a ResNet50-VD network, the neck network comprises a feature pyramid network, and the test head network comprises a YOLOV3 test head network.

9. The method of claim 8, wherein some or all of the convolutional layers in the ResNet50-VD network are replaced with deformable convolutional layers; and/or the presence of a gas in the gas,

replacing the 1x1 convolutional layer in the characteristic pyramid network and the first layer convolutional layer in the YOLOV3 detection head network with a coordinate convolutional layer; and/or the presence of a gas in the atmosphere,

the ResNet50-VD network is connected with the characteristic pyramid network through a space pyramid pooling layer; and/or the presence of a gas in the gas,

the classification branch in the YOLOV3 detection head network is replaced by a cross-over ratio prediction branch; and/or the presence of a gas in the gas,

and a DropBlock layer is arranged in the characteristic pyramid network.

10. The method of claim 9, wherein the localization result comprises a defect frame, a predicted intersection ratio of the defect frame, and a confidence level of the defect frame, and the determining the defect region according to the localization result comprises:

adjusting the confidence coefficient of the defect frame according to the predicted intersection and comparison of the defect frame;

and determining the defect region according to the defect frame with the adjusted confidence coefficient being greater than or equal to the first preset confidence coefficient.

11. The method according to claim 10, wherein said feature decoding said fused feature map by said network of detector heads to obtain a positioning result for describing the location of said defective region comprises:

performing feature decoding on the fusion feature map through the YOLOV3 detection head network to obtain a decoded coordinate logits value;

and obtaining the position coordinates of the defect frame according to the coordinate correction value and a preset scaling coefficient.

12. The method of claim 10, wherein before determining the defect region according to the defect frame with the adjusted confidence level greater than or equal to the first preset confidence level, the method further comprises:

constructing a calculation matrix for calculating the intersection and parallel ratio of all the prediction frames, and obtaining the intersection and parallel ratio of all the prediction frames according to the parallel calculation of the calculation matrix;

13. The method of any one of claims 7-12, further comprising:

14. A defect detecting apparatus for a display panel, comprising:

the defect positioning module is used for positioning the defects of the panel image through a defect positioning model and determining a defect area with the defects in the panel image;

15. An electronic device, comprising a memory and a processor, the memory storing a computer program operable on the processor, the processor implementing the steps of the method of any one of claims 1-13 when executing the computer program.

16. A storage medium having stored thereon a computer program for performing the steps of the method of any one of claims 1-13 when executed by a processor.