CN112422848A - Video splicing method based on depth map and color map - Google Patents
Video splicing method based on depth map and color map Download PDFInfo
- Publication number
- CN112422848A CN112422848A CN202011288939.4A CN202011288939A CN112422848A CN 112422848 A CN112422848 A CN 112422848A CN 202011288939 A CN202011288939 A CN 202011288939A CN 112422848 A CN112422848 A CN 112422848A
- Authority
- CN
- China
- Prior art keywords
- depth
- color
- map
- cameras
- point cloud
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 34
- 238000005457 optimization Methods 0.000 claims description 6
- 239000011159 matrix material Substances 0.000 description 3
- 239000000284 extract Substances 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/265—Mixing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/10—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/95—Computational photography systems, e.g. light-field imaging systems
- H04N23/951—Computational photography systems, e.g. light-field imaging systems by using two or more images to influence resolution, frame rate or aspect ratio
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a video splicing method based on a depth map and a color map, which comprises the following steps: preparing a plurality of groups of color cameras and depth cameras; extracting frames from the shot color video stream and depth video stream to obtain a color image and a depth image corresponding to a certain moment; calibrating by adopting a checkerboard method to acquire the poses of a plurality of groups of color cameras and depth cameras; obtaining the three-dimensional coordinates of the current shooting scene relative to a depth camera coordinate system from the depth map, acquiring point cloud data from the depth map, and deleting the point cloud data which are not in the depth range and correspond to the pixels of the color map; performing image splicing according to the screened point cloud data; and repeating the operation to obtain the spliced video stream. The video splicing method based on the depth map and the color map has the advantages of high splicing precision and efficiency and the like.
Description
Technical Field
The invention relates to the technical field of video splicing, in particular to a video splicing method based on a depth map and a color map.
Background
The existing video splicing method firstly extracts pictures in a video, then extracts feature points of each picture, performs feature matching on corresponding pictures in different videos according to the feature points, performs picture external reference settlement according to matching results, or calculates a homography matrix to splice adjacent pictures. The method has long processing time, and picture mismatching often exists on a complex scene, so that the calculation of external parameters or homography matrixes is not correct, and the splicing effect is not correct. In addition, the picture contains most scenes such as the sky and the sea surface, which are sometimes not of interest to the user because the scenes are basically consistent and often result in matching failure and inaccurate matching.
Disclosure of Invention
The invention provides a video splicing method based on a depth map and a color map, and aims to solve the problem of inaccurate matching of the existing video splicing technology.
According to the embodiment of the application, a video splicing method based on a depth map and a color map is provided, and comprises the following steps:
preparing a plurality of groups of color cameras and depth cameras;
extracting frames from the shot color video stream and depth video stream to obtain the color image and depth image corresponding to a certain time
A depth map;
calibrating by adopting a checkerboard method to acquire the poses of a plurality of groups of color cameras and depth cameras;
obtaining the three-dimensional coordinates of the current shooting scene relative to a depth camera coordinate system from the depth map, acquiring point cloud data from the depth map, and deleting the point cloud data which are not in the depth range and correspond to the pixels of the color map;
performing image splicing according to the screened point cloud data;
and repeating the operation to obtain the spliced video stream.
Preferably, a color image M and a depth image D of a certain moment in a plurality of groups of color cameras and depth cameras are obtained;
sets of color and depth map matching pairs (M1, D1), (M2, D2), …, (Mn, Dn);
preferably, the method comprises the following steps:
setting a plane with the average depth vertical to the Z axis as a projection plane;
projecting the point cloud data obtained by the depth map onto the projection surface to obtain a plurality of two-dimensional points;
filling the two-dimensional points on the plane according to the pixel value obtained by projecting each three-dimensional point to the color map;
carrying out hole filling optimization on the color image of the projection surface;
and obtaining a complete spliced image frame.
Preferably, the calibration is carried out by adopting a checkerboard method, and the poses of a plurality of groups of color cameras and depth cameras are obtained, and the method comprises the following steps:
the multiple groups of color cameras and the depth cameras observe the same checkerboard or multiple checkerboards;
taking one of the color cameras as a reference to obtain the positions of all the color cameras;
calculating the relative positions of a plurality of groups of color cameras;
the poses of the depth camera and the color camera on each device are fixed, and the poses of all the depth cameras are obtained.
Preferably, obtaining three-dimensional coordinates of the current shooting scene relative to a depth camera coordinate system from the depth map comprises the following steps:
converting the local three-dimensional coordinates according to the calculated relative pose to convert the local three-dimensional coordinates into three-dimensional point cloud;
fusing all three-dimensional point cloud data;
processing the three-dimensional point cloud data, including denoising and reprojection error optimization;
deleting the point cloud data which are not in the depth range;
and obtaining the information of the observation scene.
The technical scheme provided by the embodiment of the application can have the following beneficial effects: compared with the traditional scheme, the video splicing method based on the depth map and the color map only splices scenes in the depth range according to the specified depth range. Therefore, the user can obtain the image in the desired observation range, the interference of other scenes is eliminated, and the splicing precision and efficiency can be greatly improved. The method has great advantages in the aspect of fast real-time splicing, and can focus on certain scenes in a targeted manner. The depth map has the advantages of accurate matching, high speed and convenient scene designation.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a schematic flow chart of a video stitching method based on a depth map and a color map according to the present invention;
fig. 2 is a schematic flow chart of step S2 in the video stitching method based on depth map and color map according to the present invention;
FIG. 3 is a schematic flow chart of step S4 in the video stitching method based on depth map and color map according to the present invention;
FIG. 4 is a schematic flow chart of step S3 in the video stitching method based on the depth map and the color map according to the present invention;
fig. 5 is a schematic flowchart of step S4 in the video stitching method based on depth map and color map according to the present invention.
Description of reference numerals:
10. a video splicing method based on a depth map and a color map.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in this specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
Referring to fig. 1, the present invention discloses a video stitching method 10 based on a depth map and a color map, which includes the following steps:
step S1: preparing a plurality of groups of color cameras and depth cameras;
step S2: extracting frames from the shot color video stream and depth video stream to obtain a color image and a depth image corresponding to a certain moment;
step S3: calibrating by adopting a checkerboard method to acquire the poses of a plurality of groups of color cameras and depth cameras;
step S4: obtaining the three-dimensional coordinates of the current shooting scene relative to a depth camera coordinate system from the depth map, acquiring point cloud data from the depth map, and deleting the point cloud data which are not in the depth range and correspond to the pixels of the color map;
step S5: performing image splicing according to the screened point cloud data;
step S6: and repeating the operation to obtain the spliced video stream.
Compared with the traditional scheme, the video splicing method based on the depth map and the color map only splices scenes in the depth range according to the specified depth range. Therefore, the user can obtain the image in the desired observation range, the interference of other scenes is eliminated, and the splicing precision and efficiency can be greatly improved. The method has great advantages in the aspect of fast real-time splicing, and can focus on certain scenes in a targeted manner. The depth map has the advantages of accurate matching, high speed and convenient scene designation.
At present, a plurality of devices capable of acquiring depth images are available, such as Microsoft Kinect and Ophiometric 3d cameras. The devices can conveniently and simultaneously acquire the color image and the corresponding depth image. Since the color camera and the depth camera are fixed on the device, the relative pose of each color map and depth map is fixed.
Referring to fig. 2, the step S2 includes the following steps:
step S21: acquiring a color image M and a depth image D of a certain moment in a plurality of groups of color cameras and depth cameras;
step S22: sets of color and depth map matching pairs (M1, D1), (M2, D2), …, (Mn, Dn);
the 3D camera acquires the color image M and the depth image D at the same time every time of acquisition. The splicing requires multiple pictures at different positions, so that a picture of a larger range of scenes can be spliced. In practice, the color picture M1 … Mn is mainly used for stitching, and the purpose of the depth map is to assist stitching, so that stitching is more accurate, and meanwhile, the depth information can be used to remove an unnecessary range.
Referring to fig. 3, the step S4 includes the following steps:
step S41: setting a plane with the average depth vertical to the Z axis as a projection plane;
step S42: projecting the point cloud data obtained by the depth map onto the projection surface to obtain a plurality of two-dimensional points;
step S43: filling the two-dimensional points on the plane according to the pixel value obtained by projecting each three-dimensional point to the color map;
step S44: carrying out hole filling optimization on the color image of the projection surface;
step S45: and obtaining a complete spliced image frame.
Each point cloud data has a three-dimensional coordinate, and the Z-axis coordinate can be regarded as a depth value. Therefore, a depth range is set in advance, each point is traversed, and the point clouds of which the Z-axis values are not in the range are deleted.
Referring to fig. 4, the step S3 includes the following steps:
step S31: the multiple groups of color cameras and the depth cameras observe the same checkerboard or multiple checkerboards;
step S32: taking one of the color cameras as a reference to obtain the positions of all the color cameras;
step S33: calculating the relative positions of a plurality of groups of color cameras;
step S34: the poses of the depth camera and the color camera on each device are fixed, and the poses of all the depth cameras are obtained.
For example: taking the first 3d sensor position as a reference, the position relationship among other pre-calibrated 3d sensors is (R2, T2), … (Rn, Tn), R is a rotation matrix, dimension 3 × 3, T is a translation matrix, and dimension 3 × 1, so that the coordinate converted from one three-dimensional point p on the ith sensor to the reference coordinate system is Ri × p + Ti.
Referring to fig. 5, the step S4 further includes the following steps:
step S46: converting the local three-dimensional coordinates according to the calculated relative pose to convert the local three-dimensional coordinates into three-dimensional point cloud;
step S47: fusing all three-dimensional point cloud data;
step S48: processing the three-dimensional point cloud data, including denoising and reprojection error optimization;
step S49: deleting the point cloud data which are not in the depth range;
step S50: and obtaining the information of the observation scene.
The possible gaps in the picture are generally not considered because the content shot by the part is not in the depth observation range. If for aesthetic reasons, it can also be interpolated with surrounding color pixel values.
While the invention has been described with reference to specific embodiments, the invention is not limited thereto, and various equivalent modifications and substitutions can be easily made by those skilled in the art within the technical scope of the invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (5)
1. A video splicing method based on a depth map and a color map is characterized by comprising the following steps:
preparing a plurality of groups of color cameras and depth cameras;
extracting frames from the shot color video stream and depth video stream to obtain a color image and a depth image corresponding to a certain moment;
calibrating by adopting a checkerboard method to acquire the poses of a plurality of groups of color cameras and depth cameras;
obtaining the three-dimensional coordinates of the current shooting scene relative to a depth camera coordinate system from the depth map, acquiring point cloud data from the depth map, and deleting the point cloud data which are not in the depth range and correspond to the pixels of the color map;
performing image splicing according to the screened point cloud data;
and repeating the operation to obtain the spliced video stream.
2. The method for video stitching based on depth map and color map as claimed in claim 1, comprising the steps of:
acquiring a color image M and a depth image D of a certain moment in a plurality of groups of color cameras and depth cameras;
multiple sets of color and depth map matching pairs (M1, D1), (M2, D2), …, (Mn, Dn).
3. The method for video stitching based on depth map and color map as claimed in claim 2, comprising the steps of:
setting a plane with the average depth vertical to the Z axis as a projection plane;
projecting the point cloud data obtained by the depth map onto the projection surface to obtain a plurality of two-dimensional points;
filling the two-dimensional points on the plane according to the pixel value obtained by projecting each three-dimensional point to the color map;
carrying out hole filling optimization on the color image of the projection surface;
and obtaining a complete spliced image frame.
4. The video stitching method based on the depth map and the color map as claimed in claim 1, wherein a checkerboard method is adopted for calibration to obtain the poses of a plurality of groups of color cameras and depth cameras, and the method comprises the following steps:
the multiple groups of color cameras and the depth cameras observe the same checkerboard or multiple checkerboards;
taking one of the color cameras as a reference to obtain the positions of all the color cameras;
calculating the relative positions of a plurality of groups of color cameras;
the poses of the depth camera and the color camera on each device are fixed, and the poses of all the depth cameras are obtained.
5. The method for video stitching based on the depth map and the color map as claimed in claim 1, wherein the step of obtaining the three-dimensional coordinates of the current shooting scene relative to the depth camera coordinate system from the depth map comprises the following steps:
converting the local three-dimensional coordinates according to the calculated relative pose to convert the local three-dimensional coordinates into three-dimensional point cloud;
fusing all three-dimensional point cloud data;
processing the three-dimensional point cloud data, including denoising and reprojection error optimization;
deleting the point cloud data which are not in the depth range;
and obtaining the information of the observation scene.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011288939.4A CN112422848B (en) | 2020-11-17 | 2020-11-17 | Video stitching method based on depth map and color map |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011288939.4A CN112422848B (en) | 2020-11-17 | 2020-11-17 | Video stitching method based on depth map and color map |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112422848A true CN112422848A (en) | 2021-02-26 |
| CN112422848B CN112422848B (en) | 2024-03-29 |
Family
ID=74831990
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011288939.4A Active CN112422848B (en) | 2020-11-17 | 2020-11-17 | Video stitching method based on depth map and color map |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112422848B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114972019A (en) * | 2021-04-14 | 2022-08-30 | 华东师范大学 | Depth image stitching method, device and computer equipment based on TOF camera |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20110133677A (en) * | 2010-06-07 | 2011-12-14 | 삼성전자주식회사 | 3D image processing device and method |
| CN104008569A (en) * | 2014-02-24 | 2014-08-27 | 惠州学院 | 3D scene generation method based on depth video |
| US20140320491A1 (en) * | 2013-04-26 | 2014-10-30 | Tsinghua University | Method And System For Three-Dimensionally Reconstructing Non-Rigid Body Based On Multi-Depth-Map |
| CN105678683A (en) * | 2016-01-29 | 2016-06-15 | 杭州电子科技大学 | Two-dimensional storage method of three-dimensional model |
| US20160189358A1 (en) * | 2014-12-29 | 2016-06-30 | Dassault Systemes | Method for calibrating a depth camera |
| KR20170005312A (en) * | 2015-07-03 | 2017-01-12 | 전자부품연구원 | System and method for concurrent calibration of camera and depth sensor |
| CN107154014A (en) * | 2017-04-27 | 2017-09-12 | 上海大学 | A kind of real-time color and depth Panorama Mosaic method |
| WO2018113082A1 (en) * | 2016-12-21 | 2018-06-28 | 深圳市掌网科技股份有限公司 | 3d panoramic photographing system and method |
| WO2018125369A1 (en) * | 2016-12-30 | 2018-07-05 | Google Llc | Multi-view scene flow stitching |
| CN108389157A (en) * | 2018-01-11 | 2018-08-10 | 江苏四点灵机器人有限公司 | A kind of quick joining method of three-dimensional panoramic image |
| CN109087388A (en) * | 2018-07-12 | 2018-12-25 | 南京邮电大学 | Object dimensional modeling method based on depth transducer |
| CN109961506A (en) * | 2019-03-13 | 2019-07-02 | 东南大学 | A kind of fusion improves the local scene three-dimensional reconstruction method of Census figure |
| CN110243390A (en) * | 2019-07-10 | 2019-09-17 | 北京华捷艾米科技有限公司 | The determination method, apparatus and odometer of pose |
| US20190323843A1 (en) * | 2018-07-04 | 2019-10-24 | Baidu Online Network Technology (Beijing) Co., Ltd. | Method for generating a high precision map, apparatus and storage medium |
| WO2019229293A1 (en) * | 2018-05-31 | 2019-12-05 | Nokia Technologies Oy | An apparatus, a method and a computer program for volumetric video |
-
2020
- 2020-11-17 CN CN202011288939.4A patent/CN112422848B/en active Active
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20110133677A (en) * | 2010-06-07 | 2011-12-14 | 삼성전자주식회사 | 3D image processing device and method |
| US20140320491A1 (en) * | 2013-04-26 | 2014-10-30 | Tsinghua University | Method And System For Three-Dimensionally Reconstructing Non-Rigid Body Based On Multi-Depth-Map |
| CN104008569A (en) * | 2014-02-24 | 2014-08-27 | 惠州学院 | 3D scene generation method based on depth video |
| US20160189358A1 (en) * | 2014-12-29 | 2016-06-30 | Dassault Systemes | Method for calibrating a depth camera |
| KR20170005312A (en) * | 2015-07-03 | 2017-01-12 | 전자부품연구원 | System and method for concurrent calibration of camera and depth sensor |
| CN105678683A (en) * | 2016-01-29 | 2016-06-15 | 杭州电子科技大学 | Two-dimensional storage method of three-dimensional model |
| WO2018113082A1 (en) * | 2016-12-21 | 2018-06-28 | 深圳市掌网科技股份有限公司 | 3d panoramic photographing system and method |
| WO2018125369A1 (en) * | 2016-12-30 | 2018-07-05 | Google Llc | Multi-view scene flow stitching |
| CN107154014A (en) * | 2017-04-27 | 2017-09-12 | 上海大学 | A kind of real-time color and depth Panorama Mosaic method |
| CN108389157A (en) * | 2018-01-11 | 2018-08-10 | 江苏四点灵机器人有限公司 | A kind of quick joining method of three-dimensional panoramic image |
| WO2019229293A1 (en) * | 2018-05-31 | 2019-12-05 | Nokia Technologies Oy | An apparatus, a method and a computer program for volumetric video |
| US20190323843A1 (en) * | 2018-07-04 | 2019-10-24 | Baidu Online Network Technology (Beijing) Co., Ltd. | Method for generating a high precision map, apparatus and storage medium |
| CN109087388A (en) * | 2018-07-12 | 2018-12-25 | 南京邮电大学 | Object dimensional modeling method based on depth transducer |
| CN109961506A (en) * | 2019-03-13 | 2019-07-02 | 东南大学 | A kind of fusion improves the local scene three-dimensional reconstruction method of Census figure |
| CN110243390A (en) * | 2019-07-10 | 2019-09-17 | 北京华捷艾米科技有限公司 | The determination method, apparatus and odometer of pose |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114972019A (en) * | 2021-04-14 | 2022-08-30 | 华东师范大学 | Depth image stitching method, device and computer equipment based on TOF camera |
| CN114972019B (en) * | 2021-04-14 | 2024-05-24 | 华东师范大学 | Depth image stitching method and device based on TOF camera and computer equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112422848B (en) | 2024-03-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109308719B (en) | Binocular parallax estimation method based on three-dimensional convolution | |
| US8798387B2 (en) | Image processing device, image processing method, and program for image processing | |
| CN101630406B (en) | Camera calibration method and camera calibration device | |
| CN103337094B (en) | A kind of method of applying binocular camera and realizing motion three-dimensional reconstruction | |
| WO2012114639A1 (en) | Object display device, object display method, and object display program | |
| KR20150023370A (en) | Method and apparatus for fusion of images | |
| CN110458932B (en) | Image processing method, device, system, storage medium and image scanning apparatus | |
| JP2010009417A (en) | Image processing apparatus, image processing method, program and recording medium | |
| KR20110059506A (en) | Systems and methods for obtaining camera parameters from a plurality of images and computer program products thereof | |
| CN111009030A (en) | A multi-view high-resolution texture image and binocular 3D point cloud mapping method | |
| JP2016171463A (en) | Image processing system, image processing method, and program | |
| JP2023546739A (en) | Methods, apparatus, and systems for generating three-dimensional models of scenes | |
| CN109785390B (en) | Method and device for image correction | |
| CN109785225B (en) | Method and device for correcting image | |
| CN111105347A (en) | Method, device and storage medium for generating panoramic image with depth information | |
| KR102723109B1 (en) | Disparity estimation from wide-angle images | |
| DK3189493T3 (en) | PERSPECTIVE CORRECTION OF DIGITAL PHOTOS USING DEPTH MAP | |
| CN110276714B (en) | Method and device for synthesizing rapid scanning panoramic image | |
| CN112422848B (en) | Video stitching method based on depth map and color map | |
| CN115409707A (en) | Image fusion method and system based on panoramic video stitching | |
| JP2005063041A (en) | Three-dimensional modeling apparatus, method, and program | |
| CN113763544A (en) | Image determination method, image determination device, electronic equipment and computer-readable storage medium | |
| JP2005141655A (en) | Three-dimensional modeling apparatus and three-dimensional modeling method | |
| JP2019032660A (en) | Imaging system and imaging method | |
| KR20110133677A (en) | 3D image processing device and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |