Nothing Special   »   [go: up one dir, main page]

CN108777777A - A kind of monitor video crop straw burning method for inspecting based on deep neural network - Google Patents

A kind of monitor video crop straw burning method for inspecting based on deep neural network Download PDF

Info

Publication number
CN108777777A
CN108777777A CN201810419435.8A CN201810419435A CN108777777A CN 108777777 A CN108777777 A CN 108777777A CN 201810419435 A CN201810419435 A CN 201810419435A CN 108777777 A CN108777777 A CN 108777777A
Authority
CN
China
Prior art keywords
neural network
pyrotechnics
crop straw
straw burning
recognition processing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201810419435.8A
Other languages
Chinese (zh)
Inventor
马静
陈湘军
邓晓军
俞士贤
周睿康
周煜颉
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Jiangsu University of Technology
Original Assignee
Jiangsu University of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Jiangsu University of Technology filed Critical Jiangsu University of Technology
Priority to CN201810419435.8A priority Critical patent/CN108777777A/en
Publication of CN108777777A publication Critical patent/CN108777777A/en
Pending legal-status Critical Current

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00Television systems
    • H04N7/18Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00Scenes; Scene-specific elements
    • G06V20/40Scenes; Scene-specific elements in video content
    • G06V20/46Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
    • GPHYSICS
    • G08SIGNALLING
    • G08BSIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
    • G08B21/00Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
    • G08B21/02Alarms for ensuring the safety of persons
    • G08B21/12Alarms for ensuring the safety of persons responsive to undesired emission of substances, e.g. pollution alarms

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Artificial Intelligence (AREA)
  • Data Mining & Analysis (AREA)
  • Business, Economics & Management (AREA)
  • Signal Processing (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Toxicology (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Emergency Management (AREA)
  • Evolutionary Computation (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Environmental & Geological Engineering (AREA)
  • Alarm Systems (AREA)
  • Image Analysis (AREA)

Abstract

The present invention discloses a kind of monitor video crop straw burning method for inspecting based on deep neural network, belongs to information technology field.It constructs the pyrotechnics recognition processing module based on neural network classifier based on deep learning neural network framework, the pyrotechnics recognition processing module takes every 10 frame of video to be truncated into picture and the method that increases data set and amplification data collection feature tag, optimize residual error network, to improve the accuracy of identification image and anomalous event in video;The pyrotechnics recognition processing module according to neural network classifier to the judgement of certain areas cigarette and fire as a result, alarm from trend patrolman person and ordinary user, the monitoring to realization to crop straw burning event.

Description

A kind of monitor video crop straw burning method for inspecting based on deep neural network
Technical field
The invention belongs to information technology field, especially a kind of monitor video crop straw burning based on deep neural network patrols Detecting method is mainly used for environmental monitoring.
Background technology
With the economic development with science and technology, problem of environmental pollution getting worse, especially air pollution, haze phenomenon is spread China, but under the mainstream of " environmental protection ", crop straw burning phenomenon still remains incessant after repeated prohibition.In conventional methods where, generally by patrolling The video that personnel shoot according to monitoring camera is patrolled, related area processing crop straw burning scene is gone to.However, usually event without exception When, a large amount of human and material resources will be wasted, therefore raising routing inspection efficiency, intelligent extraction key message are very necessary.
It is built based on deep learning neural network framework, the feature tag of pyrotechnics is captured using neural network classifier, just The cigarette and fire in picture and video are may recognize that, to judge crop straw burning phenomenon.But the implementation of the method, in actual scene In, can by night light, ponding is reflective, rainy day water mist etc. factor is influenced, and generates erroneous judgement or misjudgement, cause to identify thing The inaccuracy of part sends the information of mistake.In addition, training data scarcity is also to optimize one of the difficult point of residual error network.In the following table 1 List the label characteristics of cigarette and fire and the difficult point of identifying processing.
1 pyrotechnics information of table identifies difficult point
Invention content
For the inefficiencies of solution traditional monitoring crop straw burning and since pyrotechnics is judged in the interference of other things in scene by accident The problem of event, the present invention provide a kind of monitor video crop straw burning method for inspecting based on deep neural network, construct cigarette Fiery recognition processing module using neural network classifier, automatic identification monitor video crop straw burning event, and actively sends short message Alarm, realizes the high efficiency and interactivity of information, manpower and materials is greatly saved.
To achieve the above object, the present invention uses following technical proposals:
A kind of monitor video crop straw burning method for inspecting based on deep neural network is based on deep learning neural network frame Frame constructs the pyrotechnics recognition processing module based on neural network classifier, which takes every 10 frame of video The method for being truncated into picture and increasing data set and amplification data collection feature tag optimizes residual error network, to improve identification image With the accuracy of anomalous event in video;The pyrotechnics recognition processing module is according to neural network classifier to certain areas cigarette and fire Judgement as a result, alarming from trend patrolman person and ordinary user, to realize the monitoring to crop straw burning event.
Further, pyrotechnics recognition processing module includes movable signal tower video acquisition module, deep learning image pyrotechnics Identification module and automatic alarm module;
Deep learning image pyrotechnics identification module extracts the spy of sample according to linear classifier function and loss function Label and identification classification samples are levied, categorized data set is thus trained;During training data, using amplification data Ji Te The method for levying label is accurately positioned extraction sample characteristics label, solves in a practical situation to the erroneous judgement of scene and misjudgement, and give Related personnel sends the problem of error message, while carrying out repetition training and identification to special sample, optimizes residual error network layer by layer, It excludes to identify inaccurate problem caused by the external factor such as night light, reflective, the rainy day water mist of ponding influence;Simultaneously certainly It is dynamic to sort out the operation for collecting sample, being overturn, stretched and translated to sample using Python tools, to reach extension sample Purpose solves the problems, such as data training set deficiency.
Further, automatic alarm module sends an SMS to patrol personnel automatically, and is taken the photograph by movable signal tower locating and monitoring It as the position of head, is indicated with red point abnormal regional, and is pushed away from trend user by associated wechat small routine on GPS map It delivers letters breath.
Advantageous effect:
1. the present invention utilizes movable signal tower video acquisition module, realize that stalk is burnt based on deep learning neural network framework Automatic detection and alarm are burnt, compared with traditional crop straw burning monitoring method, on the one hand can reduce the workload of patrol personnel, it is another Aspect can send related personnel to in-situ processing event in time, there is the advantages of high efficiency and promptness, can effectively reduce stalk The phenomenon that burning;
2. for when tional identification classification cigarette is with fiery feature tag, often night light, the ponding in example scenario are anti- The complex scenes such as light, rainy day water mist can influence the accuracy of identification, and the present invention constructs the pyrotechnics based on neural network classifier Recognition processing module can carry out repetition training and identification to special sample, optimize residual error network layer by layer, and extend sample, arrange Except the disturbing factor under special screne, keep recognition result more accurate.
3. the present invention is proposed based under Python environment, by amplification data collection feature tag, optimize residual error network Method solves the problem of the erroneous judgement to scene and misjudgement in a practical situation and sends error message to related personnel;Simultaneously will Special sample repetition training and identification, and the operations such as overturn, converted to it, to extend sample, solve training data not The defect of foot.
4. the present invention proposes automatic alarm module.Processing of the monitor video through neural network classifier, pyrotechnics identification As a result, on the one hand related patrol personnel can be informed in such a way that system sends short message automatically, notify them to go to in-situ processing On the other hand crop straw burning phenomenon was photographed the monitoring camera of anomalous event by the positioning of movable signal tower, was marked out on GPS map Corresponding red point is pushed from trend user by associated wechat small routine, manpower and materials is not only greatly saved, also enhance The consciousness of masses' environmental protections mutually supervises, safeguards home jointly.
Description of the drawings
Fig. 1 is the structure diagram of the pyrotechnics identification module of one embodiment of the invention;
Fig. 2 is the logical schematic based on the training of neural network classifier data of one embodiment of the invention;
Fig. 3 is the flow diagram of the sample spread training of one embodiment of the invention;
Fig. 4 is the structure diagram of the automatic alarm module of one embodiment of the invention.
Specific implementation mode
Present invention will be further explained below with reference to the attached drawings and examples.
The present embodiment proposes a kind of monitor video crop straw burning method for inspecting based on deep neural network, for working as forward pass System crop straw burning method for inspecting expends a large amount of manpowers and supervises the very low situation of implementation result, based on deep learning nerve Network frame is constructed including movable signal tower video acquisition module 1, deep learning image pyrotechnics identification module 2 and automatic report Intelligent service system including 3 these three modules of alert module, i.e. pyrotechnics recognition processing module, structure are as shown in Figure 1.Optimizing While residual error network, for the inaccurate problem of pyrotechnics feature tag identification, the method for proposing amplification data collection feature tag, The various example scenarios that will allow for are added data set and are trained, Optimized model, improve recognition accuracy;Automatic alarm is specific Object is patrol officer and ordinary user, realizes that mark is abnormal regional by movable signal tower, to send short message and wechat little Cheng The mode automatic push data of sequence.
As shown in Figure 1, pyrotechnics recognition processing module includes movable signal tower video acquisition module 1, deep learning image cigarette Fiery identification module 2 and automatic alarm module 3.After shooting video by monitoring camera, by data transport to receiving platform, god is given Through network classifier, classification is identified according to the feature tag of cigarette and fire, its recognition result is finally automatically processed, on the one hand gives Patrol personnel send alarming short message automatically, and notice is gone to in-situ processing crop straw burning event, on the other hand determined by movable signal tower Position is abnormal regional, and corresponding red point is marked out on GPS map, user is automatically pushed to by associated wechat small routine.
Wherein, it is key component, neural network classifier to be based on deep learning neural network framework classification based training data set The logical schematic of training data as shown in Fig. 2, by the data single cent part being collected into be respectively labeled as cigarette, fire, normal scene 1, Normal scene 2, specific implementation process are to be stored in 4 files respectively, and the picture that a file is set fire a, file folds up cigarette Picture, there are two files to fold up the picture of normal field scape, and the picture number of four files keeps balance, forms data Collection is put into training in residual error network.
Using loss function, loss is preferably minimized, and extracts the feature tag of sample, while weighing weight matrix W and reflecting Penetrate result and the goodness of fit of concrete class:
Wherein, xiIndicate the data of the i-th pictures, W is weight matrix, and Δ is a fixed parameter, scores to Amount is by sj=f (xi, W) and it indicates, then the score of jth class can be denoted as Again loss function is obtained with regularization:
Classification processing is carried out then according to linear classifier function pair sample:
F (x, W, b)=Wx+b (4)
Wherein, x indicates that inputted 3072 dimensional vectors, W are weight matrix, and b is indicated independent as bias Weight parameter.X can not change, but can change W, and be set to the value that can be correctly exported per pictures in training set, use The method of this linear classification obtains the optimum value of W and b, keeps in balance, keep classification results more accurate.
After training, then with four files, it is respectively put into corresponding picture, quantity is the 1/ of training picture number 4, to detect the accuracy of this Model Identification pyrotechnics and normal scene.
While classification based training pyrotechnics picture, and continue to optimize the process of residual error network.The stream of sample spread training Journey schematic diagram is as shown in Figure 3, wherein is faced with two problems, one is accuracy that external factor influences identification pyrotechnics, another A is the deficiency of data set.
For in actual scene, because of external condition, such as:The factors such as reflective, the rainy day water mist of night light, ponding influence and The inaccurate problem of caused identification, the method for taking amplification data collection feature tag.It will become clear in picture and color be partially orange red One piece of region according to coordinate mark off come, as fire feature tag be individually trained, increase identification fire accuracy.Needle To identify cigarette normal scene the problem of, the figures such as reflective, the rainy day water mist of a large amount of ponding can be added in the file of normal scene Piece is trained identification, optimizes residual error network.
Insufficient defect for data sets is handled using the increased method of data set.Specific implementation process is training In the process automatic clustering collect sample, using Python tools to sample overturning, stretch, translation etc. operations, reach extension sample Purpose, solve the problems, such as because data set deficiency due to cause to identify it is inaccurate.
After the result for having received monitoring camera data, output pyrotechnics detection, artificial treatment information is not needed, is directly passed through Automatic alarm module 3 achievees the purpose that announce information to the external world.On the one hand patrol personnel are sent an SMS to automatically, and notice is related Personnel go to in-situ processing crop straw burning event;On the other hand by the position of movable signal tower locating and monitoring camera, in GPS Its abnormal area is indicated with red point, on figure by associated wechat small routine from trend user's pushed information.Automatic alarm The structure diagram of module is as shown in Figure 4.
Limiting the scope of the invention, those skilled in the art should understand that, in technical scheme of the present invention On the basis of, the various modifications or variations that can be made by those skilled in the art with little creative work still the present invention's Within protection domain.

Claims (3)

1. a kind of monitor video crop straw burning method for inspecting based on deep neural network, it is characterised in that:Based on deep learning Neural network framework constructs the pyrotechnics recognition processing module based on neural network classifier, which takes The method that video interception increases data set and amplification data collection feature tag;The pyrotechnics recognition processing module is according to neural network point Class device is to certain areas cigarette with fiery judgement as a result, alarming from trend patrolman person and ordinary user.
2. the monitor video crop straw burning method for inspecting according to claim 1 based on deep neural network, feature exist In:The pyrotechnics recognition processing module includes movable signal tower video acquisition module (1), deep learning image pyrotechnics identification module (2) and automatic alarm module (3);
The deep learning image pyrotechnics identification module (2) extracts sample according to linear classifier function and loss function Feature tag and identification classification samples, thus train categorized data set;During training data, using amplification data Collect the method for feature tag, and repetition training and identification are carried out to special sample;Automatic clustering is collected sample, is used simultaneously The operation that Python tools overturn sample, stretched and translated.
3. the monitor video crop straw burning method for inspecting according to claim 2 based on deep neural network, feature exist In:The automatic alarm module (3) sends an SMS to patrol personnel automatically, and by the position of movable signal tower locating and monitoring camera Set, indicated with red point on GPS map it is abnormal regional, and by associated wechat small routine from trend user's pushed information.
CN201810419435.8A 2018-05-04 2018-05-04 A kind of monitor video crop straw burning method for inspecting based on deep neural network Pending CN108777777A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810419435.8A CN108777777A (en) 2018-05-04 2018-05-04 A kind of monitor video crop straw burning method for inspecting based on deep neural network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810419435.8A CN108777777A (en) 2018-05-04 2018-05-04 A kind of monitor video crop straw burning method for inspecting based on deep neural network

Publications (1)

Publication Number Publication Date
CN108777777A true CN108777777A (en) 2018-11-09

Family

ID=64026999

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810419435.8A Pending CN108777777A (en) 2018-05-04 2018-05-04 A kind of monitor video crop straw burning method for inspecting based on deep neural network

Country Status (1)

Country Link
CN (1) CN108777777A (en)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109410497A (en) * 2018-11-20 2019-03-01 江苏理工学院 A kind of monitoring of bridge opening space safety and alarm system based on deep learning
CN110062208A (en) * 2019-04-23 2019-07-26 上海赫煊自动化系统工程有限公司 A kind of security protection intelligence real-time analyzer and method
CN110351522A (en) * 2019-07-12 2019-10-18 四川长虹电子系统有限公司 Community grid management system and method based on positioning terminal
CN110855940A (en) * 2019-11-01 2020-02-28 武汉纺织大学 Method for processing video stream in straw burning detection
CN111432182A (en) * 2020-04-29 2020-07-17 上善智城(苏州)信息科技有限公司 Safety supervision method and system for oil discharge place of gas station
CN111462451A (en) * 2019-11-01 2020-07-28 武汉纺织大学 Straw burning detection alarm system based on video information
US11334759B2 (en) * 2019-01-31 2022-05-17 Canon Kabushiki Kaisha Information processing apparatus, information processing method, and medium
CN115409073A (en) * 2022-10-31 2022-11-29 之江实验室 I/Q signal identification-oriented semi-supervised width learning method and device

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101010892A (en) * 2004-12-07 2007-08-01 株式会社日立国际电气 Radio communication system
CN102163358A (en) * 2011-04-11 2011-08-24 杭州电子科技大学 Smoke/flame detection method based on video image analysis
CN102594595A (en) * 2012-02-08 2012-07-18 中国科学院上海微系统与信息技术研究所 Hybrid system for pipe network monitoring based on wireless sensor networks
CN104408469A (en) * 2014-11-28 2015-03-11 武汉大学 Firework identification method and firework identification system based on deep learning of image
CN105869341A (en) * 2016-04-20 2016-08-17 中国石油化工股份有限公司 Visual fire alarm integrated monitoring method
CN106056690A (en) * 2016-05-24 2016-10-26 国网天津武清供电有限公司 Line-protecting intelligent inspection method and system based on big data, cloud platform and internet of things
CN106339657A (en) * 2015-07-09 2017-01-18 张�杰 Straw incineration monitoring method and device based on monitoring video
CN106603973A (en) * 2016-11-15 2017-04-26 彭小林 Internet based intelligent monitoring and alarming system
CN107507134A (en) * 2017-09-21 2017-12-22 大连理工大学 Super-resolution method based on convolutional neural networks

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101010892A (en) * 2004-12-07 2007-08-01 株式会社日立国际电气 Radio communication system
CN102163358A (en) * 2011-04-11 2011-08-24 杭州电子科技大学 Smoke/flame detection method based on video image analysis
CN102594595A (en) * 2012-02-08 2012-07-18 中国科学院上海微系统与信息技术研究所 Hybrid system for pipe network monitoring based on wireless sensor networks
CN104408469A (en) * 2014-11-28 2015-03-11 武汉大学 Firework identification method and firework identification system based on deep learning of image
CN106339657A (en) * 2015-07-09 2017-01-18 张�杰 Straw incineration monitoring method and device based on monitoring video
CN105869341A (en) * 2016-04-20 2016-08-17 中国石油化工股份有限公司 Visual fire alarm integrated monitoring method
CN106056690A (en) * 2016-05-24 2016-10-26 国网天津武清供电有限公司 Line-protecting intelligent inspection method and system based on big data, cloud platform and internet of things
CN106603973A (en) * 2016-11-15 2017-04-26 彭小林 Internet based intelligent monitoring and alarming system
CN107507134A (en) * 2017-09-21 2017-12-22 大连理工大学 Super-resolution method based on convolutional neural networks

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
汪彩霞: "基于深度学习的动态纹理分类方法研究及其在森林防火中的应用", 《中国优秀硕士学位论文全文数据库信息科技辑》 *
董海鹰: "《智能控制理论及应用》", 30 September 2016 *

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109410497A (en) * 2018-11-20 2019-03-01 江苏理工学院 A kind of monitoring of bridge opening space safety and alarm system based on deep learning
US11334759B2 (en) * 2019-01-31 2022-05-17 Canon Kabushiki Kaisha Information processing apparatus, information processing method, and medium
CN110062208A (en) * 2019-04-23 2019-07-26 上海赫煊自动化系统工程有限公司 A kind of security protection intelligence real-time analyzer and method
CN110351522A (en) * 2019-07-12 2019-10-18 四川长虹电子系统有限公司 Community grid management system and method based on positioning terminal
CN110855940A (en) * 2019-11-01 2020-02-28 武汉纺织大学 Method for processing video stream in straw burning detection
CN111462451A (en) * 2019-11-01 2020-07-28 武汉纺织大学 Straw burning detection alarm system based on video information
CN111462451B (en) * 2019-11-01 2022-04-26 武汉纺织大学 Straw burning detection alarm system based on video information
CN111432182A (en) * 2020-04-29 2020-07-17 上善智城(苏州)信息科技有限公司 Safety supervision method and system for oil discharge place of gas station
CN115409073A (en) * 2022-10-31 2022-11-29 之江实验室 I/Q signal identification-oriented semi-supervised width learning method and device
CN115409073B (en) * 2022-10-31 2023-03-24 之江实验室 I/Q signal identification-oriented semi-supervised width learning method and device

Similar Documents

Publication Publication Date Title
CN108777777A (en) A kind of monitor video crop straw burning method for inspecting based on deep neural network
WO2021142902A1 (en) Danet-based unmanned aerial vehicle coastline floating garbage inspection system
Long et al. Safety helmet wearing detection based on deep learning
CN111462451B (en) Straw burning detection alarm system based on video information
CN111881730A (en) Wearing detection method for on-site safety helmet of thermal power plant
CN102854192B (en) Method for detecting apple surface defect based on apple surface defect detecting system
CN105869340B (en) A kind of fiery point monitoring system of exception based on unmanned plane and monitoring method
CN111401419A (en) Improved RetinaNet-based employee dressing specification detection method
CN106339657B (en) Crop straw burning monitoring method based on monitor video, device
CN108038424B (en) Visual automatic detection method suitable for high-altitude operation
CN103324937A (en) Method and device for labeling targets
CN113516076A (en) Improved lightweight YOLO v4 safety protection detection method based on attention mechanism
CN108761237A (en) Unmanned plane electric inspection process image vital electrical component diagnoses automatically and labeling system
CN104320610A (en) Intelligent recognition system for small mammals (rats)
CN113850562B (en) Intelligent side station supervision method and system
CN103279760B (en) A kind of plant quarantine larva real-time grading method
CN114898238B (en) Wild animal remote sensing identification method and device
CN113487570A (en) High-temperature continuous casting billet surface defect detection method based on improved yolov5x network model
CN113935364A (en) Unmanned aerial vehicle flame detection method and system based on YOLOv4 network
CN207366475U (en) A kind of relief pattern Online Video detection device
CN111931573A (en) Helmet detection and early warning method based on YOLO evolution deep learning model
CN205983585U (en) Join in marriage net transformer system of patrolling and examining, RFID module and handheld piece module
CN108195847A (en) A kind of relief pattern Online Video detection device and its detection method
CN112861676B (en) Smoke and fire identification marking method, system, terminal and storage medium
CN107657782A (en) A kind of condition of a fire Automated visual inspection system for high

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20181109