CN115115771A - Method, device, equipment and storage medium for three-dimensional reconstruction of printed circuit board - Google Patents
Method, device, equipment and storage medium for three-dimensional reconstruction of printed circuit board Download PDFInfo
- Publication number
- CN115115771A CN115115771A CN202110303214.6A CN202110303214A CN115115771A CN 115115771 A CN115115771 A CN 115115771A CN 202110303214 A CN202110303214 A CN 202110303214A CN 115115771 A CN115115771 A CN 115115771A
- Authority
- CN
- China
- Prior art keywords
- point cloud
- cloud data
- printed circuit
- circuit board
- dimensional
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003860 storage Methods 0.000 title claims abstract description 64
- 238000000034 method Methods 0.000 title claims abstract description 60
- 238000003384 imaging method Methods 0.000 claims abstract description 23
- 238000013507 mapping Methods 0.000 claims abstract description 17
- 230000006870 function Effects 0.000 claims description 99
- 239000013598 vector Substances 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 9
- 238000006243 chemical reaction Methods 0.000 claims description 7
- 238000000605 extraction Methods 0.000 claims description 6
- 230000015556 catabolic process Effects 0.000 claims description 4
- 238000006731 degradation reaction Methods 0.000 claims description 4
- 238000012545 processing Methods 0.000 abstract description 24
- 230000008569 process Effects 0.000 description 15
- 238000010586 diagram Methods 0.000 description 13
- 239000011159 matrix material Substances 0.000 description 7
- 238000004364 calculation method Methods 0.000 description 6
- 238000009826 distribution Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 238000007689 inspection Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000003466 welding Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/005—Tree description, e.g. octree, quadtree
Landscapes
- Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
The embodiment of the invention discloses a method and a device for three-dimensional reconstruction of a printed circuit board, terminal equipment and a storage medium. The method comprises the following steps: converting the initial point cloud data into a quadtree storage structure according to the relative position relationship between the initial point cloud data and a reference surface to obtain point cloud data, wherein the initial point cloud data is obtained by carrying out image acquisition on a printed circuit board to be detected by a three-dimensional imaging device, and the reference surface is a plane where the printed circuit board to be detected faces the board surface of the three-dimensional imaging device; determining a spatial implicit function of the point cloud data based on the point cloud data in the quadtree storage structure; extracting a zero-valued surface from the spatial implicit function so as to triangulate the three-dimensional surface of the printed circuit board; and mapping the three-dimensional coordinates of the three-dimensional surface of the printed circuit board to the two-dimensional image of the printed circuit board to obtain the three-dimensional surface image of the printed circuit board with the texture. The scheme realizes the balance of the processing speed and the processing result of the three-dimensional reconstruction and provides real-time and accurate three-dimensional reconstruction.
Description
Technical Field
The embodiment of the invention relates to the technical field of image processing, in particular to a method, a device, equipment and a storage medium for three-dimensional reconstruction of a printed circuit board.
Background
3D AOI (Automatic optical Inspection), i.e. 3D Automatic optical Inspection, is a technology for acquiring three-dimensional information by using a binocular, structured light and other three-dimensional imaging methods, so as to detect common defects encountered in PCB (Printed circuit board) welding. The 3D AOI user needs to visualize the three-dimensional surface of the PCB in the assembly line in real time, and the user (quality inspector) can conveniently observe whether the quality problem of the PCB occurs, so that the real-time reconstruction of the three-dimensional point cloud decoded by the 3D AOI system is particularly important.
In conventional three-dimensional reconstruction, three-dimensional point clouds are usually triangulated to obtain dense mesh information, and are rendered by texture mapping to obtain a compact three-dimensional surface. The existing three-dimensional reconstruction methods are divided into two types, namely an implicit surface method and an explicit surface method. The explicit surface method obtains a three-dimensional surface by directly solving a triangular patch or by deformation of an initial shape, and the method is difficult to process the condition of point cloud loss, so that a cavity exists in the reconstructed three-dimensional surface. The implicit surface method usually needs to define a space by using an implicit equation, wherein one isosurface is close to input data, the isosurface is visualized by using an isosurface extraction method, and surface holes can be completed by defining a global equation, however, the number of PCB point clouds decoded by a 3D AOI system is large, and the holes are numerous, and the method cannot achieve the purpose of real-time reconstruction.
The inventor finds that when the existing three-dimensional reconstruction method is used for three-dimensional reconstruction of the PCB point cloud, the requirement on the processing speed of the three-dimensional reconstruction of the point cloud data and the requirement on the processing result are in conflict, and real-time and accurate three-dimensional reconstruction cannot be realized.
Disclosure of Invention
The invention provides a method, a device, terminal equipment and a storage medium for three-dimensional reconstruction of a printed circuit board, which are used for solving the technical problems that the processing speed requirement and the processing result requirement for three-dimensional reconstruction of point cloud data in the prior art conflict and real-time and accurate three-dimensional reconstruction cannot be realized.
In a first aspect, an embodiment of the present invention provides a method for three-dimensional reconstruction of a printed circuit board, including:
converting the initial point cloud data into a quadtree storage structure according to the relative position relationship between the initial point cloud data and a reference surface to obtain point cloud data, wherein the initial point cloud data is obtained by carrying out image acquisition on a printed circuit board to be detected by a three-dimensional imaging device, and the reference surface is a plane where the printed circuit board to be detected faces the board surface of the three-dimensional imaging device;
determining a spatial implicit function of the point cloud data based on the point cloud data in the quadtree storage structure;
extracting a zero-valued surface from the spatial implicit function so as to triangulate the three-dimensional surface of the printed circuit board;
and mapping the three-dimensional coordinates of the three-dimensional surface of the printed circuit board to the two-dimensional image of the printed circuit board to obtain the three-dimensional surface image of the printed circuit board with the texture.
In a second aspect, an embodiment of the present invention further provides an apparatus for three-dimensional reconstruction of a printed circuit board, including:
the data conversion unit is used for converting the initial point cloud data into a quadtree storage structure according to the relative position relation between the initial point cloud data and a reference surface to obtain point cloud data, the initial point cloud data is obtained by image acquisition of the to-be-detected printed circuit board by the three-dimensional imaging equipment, and the reference surface is a plane where the to-be-detected printed circuit board faces the board surface of the three-dimensional imaging equipment;
the implicit function determining unit is used for determining a spatial implicit function of the point cloud data based on the point cloud data in the quadtree storage structure;
the zero-value surface extraction unit is used for extracting a zero-value surface from the spatial implicit function so as to triangulate the three-dimensional surface of the printed circuit board;
and the texture mapping unit is used for mapping the three-dimensional coordinates of the three-dimensional surface of the printed circuit board to the two-dimensional image of the printed circuit board to obtain the three-dimensional surface image of the printed circuit board with the texture.
In a third aspect, an embodiment of the present invention further provides a terminal device, including:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement a method of three-dimensional reconstruction of a printed circuit board as described in any of the first aspects.
In a fourth aspect, the present invention further provides a computer-readable storage medium, on which a computer program is stored, which when executed by a processor, implements the method for three-dimensional reconstruction of a printed circuit board according to any one of the first aspect.
According to the method, the device, the terminal equipment and the storage medium for three-dimensional reconstruction of the printed circuit board, the initial point cloud data are converted into a quadtree storage structure according to the relative position relationship between the initial point cloud data and a reference surface to obtain point cloud data, the initial point cloud data are obtained by carrying out image acquisition on the printed circuit board to be detected by a three-dimensional imaging device, and the reference surface is a plane where the printed circuit board to be detected faces the board surface of the three-dimensional imaging device; determining a spatial implicit function of the point cloud data based on the point cloud data in the quadtree storage structure; extracting a zero-valued surface from the spatial implicit function so as to triangulate the three-dimensional surface of the printed circuit board; and mapping the three-dimensional coordinates of the three-dimensional surface of the printed circuit board to the two-dimensional image of the printed circuit board to obtain the three-dimensional surface image of the printed circuit board with the texture. The point cloud data are stored through the quadtree storage structure, the space of the point cloud data in the quadtree storage structure is filled by utilizing the spatial implicit function, the relation between the spatial position of the point cloud data and the real surface of the printed circuit board is completely described based on the spatial implicit function, the three-dimensional surface with textures is obtained by combining the mapping of the two-dimensional graph, the processing speed and the processing result of three-dimensional reconstruction are balanced, and real-time and accurate three-dimensional reconstruction is provided.
Drawings
Fig. 1 is a flowchart of a method for three-dimensional reconstruction of a printed circuit board according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of storing point cloud data of a quadtree storage structure according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a neighbor point searching method in an embodiment of the present invention;
fig. 4 is a schematic structural diagram of an apparatus for three-dimensional reconstruction of a printed circuit board according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a terminal device according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are for purposes of illustration and not limitation. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
It should be noted that the present specification is not intended to exhaust all alternative embodiments for the sake of brevity, and it should be understood by those skilled in the art after reading the present specification that any combination of features may constitute an alternative embodiment as long as the features are not mutually inconsistent.
For example, in one embodiment of the embodiment, one technical feature is recited: the method comprises the following steps of determining a spatial implicit function of point cloud data based on the point cloud data in a quadtree storage structure, and recording another technical characteristic in another implementation mode of an embodiment: the area range is determined by the height difference and the number of the point cloud data in the area, and after reading the description of the present application, a person skilled in the art should think that an embodiment having both of these features is also an alternative embodiment, namely, after determining the area range by the height difference and the number, the spatial implicit function is determined by the point cloud data in each area.
Each example will be described in detail below.
Fig. 1 is a flowchart of a method for three-dimensional reconstruction of a printed circuit board according to an embodiment of the present invention. The method for three-dimensional reconstruction of a printed circuit board provided in the embodiment may be executed by an operating device corresponding to the method for three-dimensional reconstruction of a printed circuit board, where the operating device may be implemented in a software and/or hardware manner, and the operating device may be formed by two or more physical entities or may be formed by one physical entity.
Specifically, referring to fig. 1, the method for three-dimensional reconstruction of a printed circuit board specifically includes:
step S110: and converting the initial point cloud data into a quadtree storage structure according to the relative position relation between the initial point cloud data and the reference surface to obtain the point cloud data.
The initial point cloud data is acquired by the three-dimensional imaging equipment for the printed circuit board to be detected, and the reference surface is a plane where the printed circuit board to be detected faces the board surface of the three-dimensional imaging equipment.
In the field of detection, a point data set of the product appearance surface obtained by a measuring instrument is also called point cloud. Common measurement instruments are for example three-dimensional coordinate measuring machines, three-dimensional laser scanners, photographic scanners, etc. For the detected 3D point cloud data, the number of points obtained by the three-dimensional coordinate measuring machine is generally less, the distance between the points is larger, and the point cloud is sparse; the number of points obtained by using a three-dimensional laser scanner or a photographic scanner is generally large, the distance between the points is small, and the point cloud is dense.
For different scanning objects, the point cloud data has data characteristics adapted to the specific physical form characteristics of the scanning objects. For the printed circuit board, the corresponding point cloud data has the characteristics of large data volume and flat distribution. Conventional point cloud data storage typically employs octree or KD (K-Dimensional) tree for storage. For point cloud data with a large data volume and flat distribution, if an octree or KD tree is used for storage, the search efficiency in a subsequent specific data processing process is low due to the redundant structure of the octree or KD tree, so in the scheme, as shown in fig. 2, when the point cloud data in a three-dimensional space is stored, the vertical axis (Z axis) direction is not divided, a quadtree is constructed only on an XOY plane to realize the storage of the point cloud data, and a storage schematic diagram of a point cloud data set of one leaf node in a quadtree storage structure is exemplarily shown in fig. 2.
In order to adapt to the data characteristics of the point cloud data corresponding to the printed circuit board, the point cloud data processing in the scheme needs to perform coordinate conversion based on a reference surface in a real scene, specifically, a board surface is taken from the real printed circuit board as the reference surface, the point cloud data based on the three-dimensional imaging equipment is rotated according to a normal vector of the point cloud data, the collected point cloud data is located right above the actual position of the collected point cloud data on the printed circuit board, and the point cloud data can be rapidly stored in a quadtree storage structure through division of a projection area of the reference surface. In the scheme, when the projection area is divided, the bottom surface determined by the corrected point cloud data needs to be ensured to be parallel or coincident with the reference surface, and in the actual product form of the printed circuit board, the bottom surface of the solid structure corresponding to the point cloud data is the board surface for mounting the electronic element, so that the reference surface can be any one of two board surfaces which are parallel to each other, and does not need to be the board surface on which the electronic element is actually mounted.
In a specific implementation process, step S110 may be implemented by steps S111 to S112:
step S111: and correcting the initial point cloud data to obtain point cloud data, wherein the bottom surface of a three-dimensional graph formed by the point cloud data is parallel to the reference surface.
For actually acquired initial point cloud data, because of the orientation problem of the three-dimensional imaging device, there may be an inclination angle between the orientation of the board surface of the actual printed circuit board and the coordinate axis of the three-dimensional imaging device when generating the point cloud data, that is, the axial direction of the three-dimensional imaging device is not perpendicular to the orientation of the board surface of the actual printed circuit board, at this time, the point cloud data cannot be directly stored as a quadtree structure according to the projection area division of the reference surface, the coordinates need to be rotated based on the normal vector of the initial point cloud data, so that the bottom surface of the three-dimensional graph formed by the point cloud data is parallel to the reference surface, and new point cloud data is obtained to implement subsequent quadtree storage.
Step S112: and dividing the reference surface into a plurality of projection areas based on the preset area size, wherein the height difference of the point cloud data projected in a single projection area is within a first preset value or the number of the point cloud data is smaller than a second preset value.
The reference plane as a limited plane may be divided into a limited number of projection regions with the same size, and the projection regions are not only the division of the reference plane, but also the division of point cloud data, that is, the point cloud data projected in the same projection region belong to the same point cloud set, please refer to fig. 2, where a projection region is exemplarily represented in the reference plane (XOY plane), and the point cloud data projected in the projection region may be regarded as being constrained within the rectangular parallelepiped shown in fig. 2.
The specific dividing process of dividing the reference surface into a plurality of projection areas based on the preset area size is to divide the reference surface into a plurality of projection areas according to the preset area size, then further split according to the number of point cloud data corresponding to each projection area and the height relationship among the number, specifically, obtain the next-level projection area from the previous-level projection area in a 2 x 2 equal division mode until point cloud data of a quadtree storage structure is obtained, and finally enable the point cloud data in each projection area to meet the set point cloud data number and the relationship among the point cloud data.
In the actual processing, the division of the projection can be realized by steps S1121 to S1122 for step S112:
step S1121: dividing the reference surface into a plurality of projection areas based on the size of a preset initial area, and confirming the height difference of the point cloud data in the projection range corresponding to each projection area.
Step S1122: and if the height difference of the point cloud data in the projection area is larger than a first preset value and the number of the point cloud data in the projection area is more than a second preset value, equally dividing the projection area into 2 multiplied by 2 projection areas, and confirming the height difference and the number of the point cloud data in the equally divided projection areas until the height difference of the point cloud data in a single projection area is within the first preset value or the number of the point cloud data in the single projection area is smaller than the second preset value.
For the printed circuit board, the point cloud data not only has the integral characteristics of large data volume and flat distribution, but also has the characteristic of abrupt change of the edge of each component on the printed circuit board, when the point cloud data are stored based on the quad-tree storage structure in the scheme, in order to prevent the edges of components from being smooth, when the leaf node corresponding to the projection area is finally determined, whether the height variance of the point cloud data in the projection area in the vertical axis (Z axis) direction is larger than a preset first preset value or not is judged in advance, if the height variance is larger than a preset first preset value, indicating that the point cloud data in the projection area has a height step, namely, the edge of the component exists in the projection area, so the projection area is further divided into 2 multiplied by 2 evenly, thus, 4 child nodes can be obtained, which is also the forming process of the quad-tree storage structure in the scheme. Certainly, in order to reduce the amount of calculation and the depth of the quadtree storage structure and improve the data processing efficiency, when the number of point cloud data in the projection area is within the second preset value, even if the height variance is greater than the preset first threshold value, the averaging is not performed any more. The quadtree storage structure formed through the equipartition process stores the point cloud data, the three-dimensional space position of the point cloud data and the adjacent points of the point cloud data can be rapidly positioned, and the edges of components and parts can be guaranteed to be fine and smooth.
It can be seen that not all projection areas are the same in size, for the part where the point cloud data is dense and the height difference between the point cloud data is large, the projection area may need to be divided equally by 2 × 2 many times, the corresponding projection area may be small, and the quadtree storage structure based on the initial area has more levels and deeper depth; for the part with sparse point cloud data and small height difference between the point cloud data, the projection area may not need to be equally divided by 2 × 2 or the number of times is small, the corresponding projection area may be large, the level of the quadtree storage structure based on the initial area is small, and the depth is shallow.
Step S120: and determining a spatial implicit function of the point cloud data based on the point cloud data in the quadtree storage structure.
The point cloud data stored in the quadtree storage structure may have holes of the point cloud data, so that accurate and vivid reconstructed textures are presented for users (quality inspection personnel) finally, the holes need to be supplemented, and the holes in the point cloud data are supplemented in the scheme by determining a spatial implicit function of the point cloud data. The spatial implicit function is used for surface reconstruction of a three-dimensional image, and mainly converts unstructured three-dimensional directed point cloud data into a local smooth spatial implicit function so as to reflect the detailed shape of the surface of the three-dimensional directed point cloud. The spatial implicit function has high flexibility and can describe scenes with various complex shapes; due to the adoption of global constraint, the spatial implicit function is robust to noise and can smoothly process the holes in the directed point cloud.
The process of determining the spatial implicit function in step S120 can be specifically implemented by steps S121 to S123:
step S121: and defining a general expression of the spatial implicit function according to the point cloud data and the corresponding normal vector.
Based on point cloud data P ═ { P ═ P 1 ,p 2 ,…,p n The sum normal vector N ═ N 1 ,n 2 ,…,n n A general expression of the spatial implicit function may be defined. The general expression determines the basic form of the spatial implicit function, and determines a plurality of coefficients in the general expression, so that the actual expression of the spatial implicit function can be obtained.
Step S122: and constructing a linear equation system based on the values of the general expression, the normal vector and the spatial implicit function.
Step S123: the system of linear equations is solved to determine the spatial implicit function.
The coefficients in the general expression are realized by constructing an equation set and solving, namely, the equation set only containing the coefficient unknowns is constructed, and the general expression is substituted after the equation set is solved, so that the spatial implicit function of the point cloud data corresponding to a specific printed circuit board can be obtained.
In a specific implementation, the general expression of the spatial implicit function is:
in the general expression, in the following,
denotes the radial basis function, phi p (r)＝φ(r/ρ)，
Wherein p is j Indicating pointsCloud data, n represents the number of point cloud data, a j And b j Is the unknown to be solved for, p represents the support radius, r is the euler distance.
On the basis of the above general expression, the constraint of the spatial implicit function may identify two equations:
where i is 1,2 …, n, H is the hessian matrix, c represents the value of the spatial implicit function, and for a known point cloud, c is 0 (the surface is the zero-valued surface of the spatial implicit function), a j And b j Respectively representing the coefficients to be solved. The above two equations can be further written as a system of linear equations with the addition of a regularization term:
(A+ηI)λ＝y
wherein λ and y are vectors of length 4n, respectively, and the ith element is [ a [ ], respectively i ,b i ] T And [ c i ,n i ] T . A is a 4n × 4n coefficient matrix, A i,j A 4 × 4 sub-matrix, defined as:
by solving the above system of linear equations, a can be obtained j And b j Thereby determining the expression mode of the spatial implicit function in the specific point cloud data.
On the basis of the general expression, step S123 is specifically realized by steps S1231-S1232:
step S1231: and carrying out degradation on the linear equation set based on a closed form formula to obtain an approximate solution.
Step S1232: and determining the spatial implicit function according to the approximate solution.
In a specific processing process, considering that the increase of the number of point clouds causes problems of unstable numerical values, high calculation cost and the like in the solving process of the linear equation system, a closed form formula based on a quasi-interpolation theorem is adopted so as to reconstruct the surface in a more stable and effective mode. In particular, consider a constraint g (x) with a function value i )＝f i Interpolation function g (x) sigma i λ i Ψ i (x) In that respect When lambda is i ≡f i It is good to approximate the interpolation function g (x) to sigma i f i Ψ i (x) .1. the However, since the interpolation constraint contains both the value of the function and the gradient of the function (i.e., the two equations described above), a closed-form equation is used to approximate the solution to the system of equations. By matrix calculation, λ when the coefficient matrix is approximated by the identity matrix I i ≡f i Is an approximate solution to the pseudo-interpolation. Coefficient matrix A described above i,i The degradation is as follows:
the linear equation to be solved in the linear equation system corresponding to the addition of the regularization term is reduced to D λ ═ y, D ═ D (D ═ y) i,j ) n×n And obtain an approximate solution
Finally, according to the principle that the reconstructed surface is a zero-valued surface of the spatial implicit function, c ═ 0 yields:
in a specific implementation, in addition to the completion manner described above, there may be a second implementation manner of the completion manner, and overall, the general expression of the spatial implicit function is still:
in the general expression, however,
denotes the radial basis function, phi p (r)＝φ(r/ρ)，
Wherein p is j Representing point cloud data, n representing the number of point cloud data, a j And b j Is the unknown to be solved for, ρ represents the support radius, and r is the euler distance.
Due to the fact that the point cloud data of the printed circuit board decoded by the 3D AOI system has a cavity in the vertical plane of the component, the position of the cavity cannot calculate an implicit function, and surface deletion is caused. In order to further increase the computational efficiency of the completion process, it is additionally proposed to complete the surface during reconstruction without additional computational expense. According to the distribution of the point cloud data of the printed circuit board, a radial basis function aiming at the point cloud data of the printed circuit board is designed, so that a hidden function value can be calculated at the three-dimensional position of the cavity. As shown in fig. 3, a black point is a three-dimensional space position with a positive implicit function value, a shadow point is a three-dimensional space position with a negative implicit function value, a blank point is a missing part, and a black line is a reconstructed surface, that is, a zero-valued surface of a spatial implicit function. Using the modified radial basis function of the second implementation, reducing vertical (Z) axis constraints allows calculation of implicit function values in the vertical plane of the components on the printed circuit board (i.e., by using the modified radial basis function of the second implementation
Calculation). In general, by defining the distance between the horizontal axis (X axis) and the vertical axis (Y axis) as the radius of the radial basis function, the missing part of the spatial hidden function value (point cloud data in the dashed line frame in fig. 3) can be effectively compensated.
In the second implementation mode, aiming at the condition of point cloud loss of the side surface of a component on the printed circuit board, the spatial implicit function is complemented on the premise of not introducing calculation overhead by improving the expression of the radial basis function, so that the surface cavity is eliminated, and the processing speed is integrally improved.
Step S130: and extracting a zero surface from the spatial implicit function so as to triangulate the three-dimensional surface of the printed circuit board.
After the space implicit function is triangulated, a three-dimensional mesh with a topological structure can be obtained, and therefore a three-dimensional model is rendered in a high-quality mode.
Step S140: and mapping the three-dimensional coordinates of the three-dimensional surface of the printed circuit board to the two-dimensional image of the printed circuit board to obtain the three-dimensional surface image of the printed circuit board with the texture.
During specific rendering, the two-dimensional image synchronously acquired and shot during point cloud data acquisition is mapped to the reconstructed three-dimensional coordinate to obtain a three-dimensional surface image of the printed circuit board with the texture, and a user (quality inspector) observes the three-dimensional surface image to finish observation and detection of the printed circuit board to be detected.
Converting the initial point cloud data into a quadtree storage structure according to the relative position relationship between the initial point cloud data and a reference surface to obtain point cloud data, wherein the initial point cloud data is obtained by carrying out image acquisition on a printed circuit board to be detected by a three-dimensional imaging device, and the reference surface is a plane where the printed circuit board to be detected faces the board surface of the three-dimensional imaging device; determining a spatial implicit function of the point cloud data based on the point cloud data in the quadtree storage structure; extracting a zero-valued surface from the spatial implicit function so as to triangulate the three-dimensional surface of the printed circuit board; and mapping the three-dimensional coordinates of the three-dimensional surface of the printed circuit board to the two-dimensional image of the printed circuit board to obtain the three-dimensional surface image of the printed circuit board with the texture. The point cloud data are stored through the quad-tree storage structure, the space of the point cloud data in the quad-tree storage structure is complemented by the spatial implicit function, the relation between the spatial position of the point cloud data and the real surface of the printed circuit board is completely described based on the spatial implicit function, the three-dimensional surface with the texture is obtained by combining the mapping of the two-dimensional graph, the processing speed and the processing result of three-dimensional reconstruction are balanced, and real-time and accurate three-dimensional reconstruction is provided.
Fig. 4 is a schematic structural diagram of an apparatus for three-dimensional reconstruction of a printed circuit board according to an embodiment of the present invention. Referring to fig. 4, the apparatus for three-dimensional reconstruction of a printed circuit board includes: a data conversion unit 210, a implicit function determination unit 220, a zero-valued surface extraction unit 230, and a texture mapping unit 240.
The data conversion unit 210 is configured to convert the initial point cloud data into a quadtree storage structure according to a relative position relationship between the initial point cloud data and a reference surface to obtain point cloud data, where the initial point cloud data is obtained by performing image acquisition on a printed circuit board to be detected by a three-dimensional imaging device, and the reference surface is a plane where the printed circuit board to be detected faces a board surface of the three-dimensional imaging device; a implicit function determination unit 220, configured to determine a spatial implicit function of the point cloud data based on the point cloud data in the quadtree storage structure; a zero-valued surface extraction unit 230, configured to extract a zero-valued surface for the spatial implicit function, so as to triangulate a three-dimensional surface of the printed circuit board; and a texture mapping unit 240 for mapping the three-dimensional coordinates of the three-dimensional surface of the printed circuit board to the two-dimensional image of the printed circuit board to obtain a three-dimensional surface image of the printed circuit board with texture.
On the basis of the above embodiment, the data conversion unit includes:
the point cloud correction module is used for correcting the initial point cloud data to obtain point cloud data, and the bottom surface of a three-dimensional graph formed by the point cloud data is parallel to the reference surface;
the area dividing module is used for dividing the reference surface into a plurality of projection areas based on the preset area size, and the height difference of point cloud data projected in a single projection area is within a first preset value or the number of the point cloud data is smaller than a second preset value;
wherein, the projection area of the upper level obtains the projection area of the lower level in a 2 × 2 equal division manner to obtain the point cloud data of the quadtree storage structure.
On the basis of the above embodiment, the area dividing module includes:
the division submodule is used for dividing the reference surface into a plurality of projection areas based on the size of a preset initial area and confirming the height difference of point cloud data in a projection range corresponding to each projection area;
and the judgment submodule is used for equally dividing the projection area into 2 multiplied by 2 projection areas if the height difference of the point cloud data in the projection area is larger than a first preset value and the number of the point cloud data in the projection area is more than a second preset value, and confirming the height difference and the number of the point cloud data in the equally divided projection area until the height difference of the point cloud data in a single projection area is within the first preset value or the number of the point cloud data in the single projection area is smaller than the second preset value.
On the basis of the above embodiment, the implicit function determination unit includes:
the expression definition module is used for defining a general expression of the space implicit function according to the point cloud data and the corresponding normal vector;
the system comprises an equation set construction module, a data processing module and a data processing module, wherein the equation set construction module is used for constructing a linear equation set based on values of a general expression, a normal vector and a space implicit function;
and the equation system solving module is used for solving the linear equation system to determine the spatial implicit function.
On the basis of the above embodiment, the general expression of the spatial implicit function is:
in the general expression, it is stated that,
denotes the radial basis function, phi p (r)＝φ(r/ρ)，
Wherein p is j Representing point cloud data, n representing the number of point cloud data, a j And b j Is the unknown to be solved for, ρ represents the support radius, and r is the euler distance.
On the basis of the above embodiment, the equation system solving module includes:
the approximate solution submodule is used for carrying out degradation on the linear equation set based on a closed form formula to obtain an approximate solution;
and the latent function determining submodule is used for determining the spatial latent function according to the approximate solution.
On the basis of the above embodiment, the general expression of the spatial implicit function is:
in the general expression, it is stated that,
denotes the radial basis function, phi p (r)＝φ(r/ρ)，
Wherein p is j Representing point cloud data, n representing the number of point cloud data, a j And b j Is the unknown to be solved for, p represents the support radius, r is the euler distance.
The device for three-dimensional reconstruction of the printed circuit board provided by the embodiment of the invention is contained in the terminal equipment, can be used for executing the method for three-dimensional reconstruction of any printed circuit board provided by the embodiment, and has corresponding functions and beneficial effects.
It should be noted that, in the embodiment of the apparatus for three-dimensional reconstruction of a printed circuit board, the included units and modules are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.
Fig. 5 is a schematic structural diagram of a terminal device according to an embodiment of the present invention. As shown in fig. 5, the terminal device includes a processor 310, a memory 320, an input means 330, an output means 340, and a communication means 350; the number of the processors 310 in the terminal device may be one or more, and one processor 310 is taken as an example in fig. 5; the processor 310, the memory 320, the input device 330, the output device 340 and the communication device 350 in the terminal equipment may be connected by a bus or other means, and the connection by the bus is exemplified in fig. 5.
The memory 320 is a computer-readable storage medium, and can be used for storing software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the method for three-dimensional reconstruction of a printed circuit board in the embodiment of the present invention (for example, the data conversion unit 210, the implicit function determination unit 220, the zero-valued surface extraction unit 230, and the texture mapping unit 240 in the apparatus for three-dimensional reconstruction of a printed circuit board). The processor 310 executes various functional applications and data processing of the terminal device by executing the software programs, instructions and modules stored in the memory 320, namely, the method for three-dimensional reconstruction of the printed circuit board is realized.
The memory 320 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal device, and the like. Further, the memory 320 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, memory 320 may further include memory located remotely from processor 310, which may be connected to the terminal device via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 330 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the terminal apparatus. The output device 340 may include a display device such as a display screen.
The terminal equipment comprises a device for three-dimensional reconstruction of the printed circuit board, can be used for executing a method for three-dimensional reconstruction of any printed circuit board, and has corresponding functions and beneficial effects.
Embodiments of the present invention also provide a storage medium containing computer-executable instructions, which when executed by a computer processor, are configured to perform operations related to the method for three-dimensional reconstruction of a printed circuit board provided in any of the embodiments of the present application, and have corresponding functions and advantages.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product.
Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein. The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks. These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory. The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in some detail by the above embodiments, the invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the invention, and the scope of the invention is determined by the scope of the appended claims.
Claims (10)
1. A method of three-dimensional reconstruction of a printed circuit board, comprising:
converting initial point cloud data into a quadtree storage structure according to the relative position relationship between the initial point cloud data and a reference surface to obtain point cloud data, wherein the initial point cloud data is obtained by carrying out image acquisition on a printed circuit board to be detected by a three-dimensional imaging device, and the reference surface is a plane where the printed circuit board to be detected faces the board surface of the three-dimensional imaging device;
determining a spatial implicit function of the point cloud data based on the point cloud data in the quadtree storage structure;
extracting a zero-value surface from the spatial implicit function so as to triangulate the three-dimensional surface of the printed circuit board;
and mapping the three-dimensional coordinates of the three-dimensional surface of the printed circuit board to the two-dimensional image of the printed circuit board to obtain the three-dimensional surface image of the printed circuit board with textures.
2. The method of claim 1, wherein converting the initial point cloud data into a quadtree storage structure according to the relative position relationship between the initial point cloud data and a reference surface to obtain point cloud data comprises:
correcting the initial point cloud data to obtain point cloud data, wherein the bottom surface of a three-dimensional graph formed by the point cloud data is parallel to the reference surface;
dividing the reference surface into a plurality of projection areas based on the size of a preset area, wherein the height difference of point cloud data projected in a single projection area is within a first preset value or the number of the point cloud data is smaller than a second preset value;
wherein, the projection area of the upper level obtains the projection area of the lower level in a 2 × 2 equal division manner to obtain the point cloud data of the quadtree storage structure.
3. The method of claim 2, wherein the dividing the reference plane into a plurality of projection areas based on a preset area size, and the height difference of the point cloud data projected in a single projection area is within a first preset value or the number of the point cloud data is smaller than a second preset value comprises:
dividing the reference surface into a plurality of projection areas based on the size of a preset initial area, and confirming the height difference of point cloud data in a projection range corresponding to each projection area;
and if the height difference of the point cloud data in the projection area is larger than a first preset value and the number of the point cloud data in the projection area is more than a second preset value, equally dividing the projection area into 2 multiplied by 2 projection areas, and confirming the height difference and the number of the point cloud data in the equally divided projection areas until the height difference of the point cloud data in a single projection area is within the first preset value or the number of the point cloud data in the single projection area is smaller than the second preset value.
4. The method of claim 1, wherein determining the spatial implicit function of the point cloud data based on the point cloud data in the quadtree storage structure comprises:
defining a general expression of a spatial implicit function according to the point cloud data and the corresponding normal vector;
constructing a linear equation set based on the values of the general expression, the normal vector and the spatial implicit function;
solving the system of linear equations to determine the spatial implicit function.
5. The method of claim 4, wherein the general expression of the spatial implicit function is:
in the general expression, in the above-mentioned general expression,
denotes the radial basis function, phi p (r)＝φ(r/ρ)，
Wherein p is j Representing point cloud data, n representing the number of point cloud data, a j And b j Is the unknown to be solved for, ρ represents the support radius, and r is the euler distance.
6. The method of claim 5, wherein said solving the system of linear equations to determine the spatial implicit function comprises:
carrying out degradation on the linear equation set to obtain an approximate solution based on a closed form formula;
and determining the spatial implicit function according to the approximate solution.
7. The method of claim 4, wherein the general expression of the spatial implicit function is:
in the general expression, the expression,
denotes the radial basis function, phi p (r)＝φ(r/ρ)，
Wherein p is j Representing point cloud data, n representing the number of point cloud data, a j And b j Is the unknown to be solved for, ρ represents the support radius, and r is the euler distance.
8. An apparatus for three-dimensional reconstruction of a printed circuit board, comprising:
the data conversion unit is used for converting the initial point cloud data into a quadtree storage structure according to the relative position relationship between the initial point cloud data and a reference surface to obtain point cloud data, the initial point cloud data is obtained by carrying out image acquisition on a printed circuit board to be detected by a three-dimensional imaging device, and the reference surface is a plane where the printed circuit board to be detected faces the board surface of the three-dimensional imaging device;
a hidden function determining unit, configured to determine a spatial hidden function of the point cloud data based on the point cloud data in the quadtree storage structure;
a zero-value surface extraction unit, configured to extract a zero-value surface for the spatial implicit function to triangulate a three-dimensional surface of the printed circuit board;
and the texture mapping unit is used for mapping the three-dimensional coordinates of the three-dimensional surface of the printed circuit board to the two-dimensional image of the printed circuit board to obtain the three-dimensional surface image of the printed circuit board with the texture.
9. A terminal device, comprising:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement a method for three-dimensional reconstruction of a printed circuit board as recited in any of claims 1-7.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out a method for three-dimensional reconstruction of a printed circuit board according to one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110303214.6A CN115115771A (en) | 2021-03-22 | 2021-03-22 | Method, device, equipment and storage medium for three-dimensional reconstruction of printed circuit board |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110303214.6A CN115115771A (en) | 2021-03-22 | 2021-03-22 | Method, device, equipment and storage medium for three-dimensional reconstruction of printed circuit board |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115115771A true CN115115771A (en) | 2022-09-27 |
Family
ID=83324478
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110303214.6A Pending CN115115771A (en) | 2021-03-22 | 2021-03-22 | Method, device, equipment and storage medium for three-dimensional reconstruction of printed circuit board |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115115771A (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100085352A1 (en) * | 2008-10-04 | 2010-04-08 | Microsoft Corporation | Parallel surface reconstruction |
| CN106504332A (en) * | 2016-10-19 | 2017-03-15 | 未来科技（襄阳）有限公司 | The curve reestablishing method and device of three-dimensional point cloud |
| CN108230432A (en) * | 2017-12-12 | 2018-06-29 | 中国南方电网有限责任公司超高压输电公司广州局 | A kind of insulator laser point cloud three-dimensional rebuilding method based on CS-RBF |
| US10509947B1 (en) * | 2017-04-11 | 2019-12-17 | Zoox, Inc. | Converting multi-dimensional data for image analysis |
| CN111222516A (en) * | 2020-01-06 | 2020-06-02 | 西南电子技术研究所（中国电子科技集团公司第十研究所） | Method for extracting key outline characteristics of point cloud of printed circuit board |
| CN111681318A (en) * | 2020-06-10 | 2020-09-18 | 上海城市地理信息系统发展有限公司 | Point cloud data modeling method and device and electronic equipment |
| CN111797734A (en) * | 2020-06-22 | 2020-10-20 | 广州视源电子科技股份有限公司 | Vehicle point cloud data processing method, device, equipment and storage medium |
| CN112308963A (en) * | 2020-11-13 | 2021-02-02 | 四川川大智胜软件股份有限公司 | Non-inductive three-dimensional face reconstruction method and acquisition reconstruction system |
-
2021
- 2021-03-22 CN CN202110303214.6A patent/CN115115771A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100085352A1 (en) * | 2008-10-04 | 2010-04-08 | Microsoft Corporation | Parallel surface reconstruction |
| CN106504332A (en) * | 2016-10-19 | 2017-03-15 | 未来科技（襄阳）有限公司 | The curve reestablishing method and device of three-dimensional point cloud |
| US10509947B1 (en) * | 2017-04-11 | 2019-12-17 | Zoox, Inc. | Converting multi-dimensional data for image analysis |
| CN108230432A (en) * | 2017-12-12 | 2018-06-29 | 中国南方电网有限责任公司超高压输电公司广州局 | A kind of insulator laser point cloud three-dimensional rebuilding method based on CS-RBF |
| CN111222516A (en) * | 2020-01-06 | 2020-06-02 | 西南电子技术研究所（中国电子科技集团公司第十研究所） | Method for extracting key outline characteristics of point cloud of printed circuit board |
| CN111681318A (en) * | 2020-06-10 | 2020-09-18 | 上海城市地理信息系统发展有限公司 | Point cloud data modeling method and device and electronic equipment |
| CN111797734A (en) * | 2020-06-22 | 2020-10-20 | 广州视源电子科技股份有限公司 | Vehicle point cloud data processing method, device, equipment and storage medium |
| CN112308963A (en) * | 2020-11-13 | 2021-02-02 | 四川川大智胜软件股份有限公司 | Non-inductive three-dimensional face reconstruction method and acquisition reconstruction system |
Non-Patent Citations (3)
| Title |
|---|
| CARR JC等: "reconstruction and representation of 3D objects with radial basis functions", SIGGRAPH 2001CONFERENCE PROCEEDINGS, 31 August 2001 (2001-08-31), pages 67 - 76, XP001049875 * |
| 张懿: "基于RBF的三维重建关键技术的研究", 万方学位论文数据库, 20 March 2013 (2013-03-20), pages 1 - 72 * |
| 潘风: "点云模型隐式曲面重建技术的研究", 中国优秀硕士学位论文全文数据库信息科技辑, 15 January 2021 (2021-01-15), pages 138 - 2165 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10706622B2 (en) | Point cloud meshing method, apparatus, device and computer storage media | |
| CN110363858B (en) | Three-dimensional face reconstruction method and system | |
| CN111210429B (en) | Point cloud data partitioning method and device and obstacle detection method and device | |
| WO2021120846A1 (en) | Three-dimensional reconstruction method and device, and computer readable medium | |
| TWI536318B (en) | Depth measurement quality enhancement | |
| Takimoto et al. | 3D reconstruction and multiple point cloud registration using a low precision RGB-D sensor | |
| US20180025496A1 (en) | Systems and methods for improved surface normal estimation | |
| CN111553946B (en) | Method and device for removing ground point cloud and method and device for detecting obstacle | |
| CN114998328A (en) | Workpiece spraying defect detection method and system based on machine vision and readable storage medium | |
| WO2020141161A1 (en) | Method for 3d reconstruction of an object | |
| Ylimäki et al. | Accurate 3-d reconstruction with rgb-d cameras using depth map fusion and pose refinement | |
| CN115115771A (en) | Method, device, equipment and storage medium for three-dimensional reconstruction of printed circuit board | |
| CN112258635B (en) | Three-dimensional reconstruction method and device based on improved binocular matching SAD algorithm | |
| di Filippo et al. | A methodological proposal for the comparison of 3D photogrammetric models | |
| Choi et al. | Implementation of Real‐Time Post‐Processing for High‐Quality Stereo Vision | |
| CN110570511A (en) | point cloud data processing method, device and system and storage medium | |
| CN114494404A (en) | Object volume measurement method, system, device and medium | |
| CN115393381A (en) | Straightness detection method and device, computer equipment and storage medium | |
| CN113421292A (en) | Three-dimensional modeling detail enhancement method and device | |
| CN112802175B (en) | Large-scale scene shielding and eliminating method, device, equipment and storage medium | |
| CN113763558A (en) | Information processing method and device, equipment and storage medium | |
| WO2023105646A1 (en) | Image correction device, image correction method, and image correction program | |
| Recker et al. | Depth data assisted structure-from-motion parameter optimization and feature track correction | |
| CN118643447B (en) | Steep-environment-oriented ground filtering method and device, electronic equipment and medium | |
| CN116051792A (en) | Product graph generation method, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |