CN108742159A - Intelligent control device of water dispenser based on RGB-D cameras and its control method - Google Patents
Intelligent control device of water dispenser based on RGB-D cameras and its control method Download PDFInfo
- Publication number
- CN108742159A CN108742159A CN201810306420.0A CN201810306420A CN108742159A CN 108742159 A CN108742159 A CN 108742159A CN 201810306420 A CN201810306420 A CN 201810306420A CN 108742159 A CN108742159 A CN 108742159A
- Authority
- CN
- China
- Prior art keywords
- rgb
- cup
- processing unit
- information storage
- cameras
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 title claims abstract description 176
- 238000000034 method Methods 0.000 title claims abstract description 36
- 238000000605 extraction Methods 0.000 claims abstract description 13
- 230000035622 drinking Effects 0.000 claims description 20
- 238000012544 monitoring process Methods 0.000 claims description 12
- 238000006243 chemical reaction Methods 0.000 claims description 6
- 238000009499 grossing Methods 0.000 claims description 6
- 238000004458 analytical method Methods 0.000 claims description 4
- 238000005286 illumination Methods 0.000 claims description 4
- 238000005259 measurement Methods 0.000 claims description 4
- 230000003287 optical effect Effects 0.000 claims description 4
- 230000011218 segmentation Effects 0.000 claims description 4
- 239000007921 spray Substances 0.000 claims description 3
- 230000009466 transformation Effects 0.000 claims description 3
- 239000003651 drinking water Substances 0.000 claims 1
- 235000020188 drinking water Nutrition 0.000 claims 1
- 238000007363 ring formation reaction Methods 0.000 claims 1
- 238000002347 injection Methods 0.000 description 3
- 239000007924 injection Substances 0.000 description 3
- PCTMTFRHKVHKIS-BMFZQQSSSA-N (1s,3r,4e,6e,8e,10e,12e,14e,16e,18s,19r,20r,21s,25r,27r,30r,31r,33s,35r,37s,38r)-3-[(2r,3s,4s,5s,6r)-4-amino-3,5-dihydroxy-6-methyloxan-2-yl]oxy-19,25,27,30,31,33,35,37-octahydroxy-18,20,21-trimethyl-23-oxo-22,39-dioxabicyclo[33.3.1]nonatriaconta-4,6,8,10 Chemical compound C1C=C2C[C@@H](OS(O)(=O)=O)CC[C@]2(C)[C@@H]2[C@@H]1[C@@H]1CC[C@H]([C@H](C)CCCC(C)C)[C@@]1(C)CC2.O[C@H]1[C@@H](N)[C@H](O)[C@@H](C)O[C@H]1O[C@H]1/C=C/C=C/C=C/C=C/C=C/C=C/C=C/[C@H](C)[C@@H](O)[C@@H](C)[C@H](C)OC(=O)C[C@H](O)C[C@H](O)CC[C@@H](O)[C@H](O)C[C@H](O)C[C@](O)(C[C@H](O)[C@H]2C(O)=O)O[C@H]2C1 PCTMTFRHKVHKIS-BMFZQQSSSA-N 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- NJPPVKZQTLUDBO-UHFFFAOYSA-N novaluron Chemical compound C1=C(Cl)C(OC(F)(F)C(OC(F)(F)F)F)=CC=C1NC(=O)NC(=O)C1=C(F)C=CC=C1F NJPPVKZQTLUDBO-UHFFFAOYSA-N 0.000 description 1
- 239000000243 solution Substances 0.000 description 1
- 238000004148 unit process Methods 0.000 description 1
Classifications
-
- A—HUMAN NECESSITIES
- A47—FURNITURE; DOMESTIC ARTICLES OR APPLIANCES; COFFEE MILLS; SPICE MILLS; SUCTION CLEANERS IN GENERAL
- A47J—KITCHEN EQUIPMENT; COFFEE MILLS; SPICE MILLS; APPARATUS FOR MAKING BEVERAGES
- A47J31/00—Apparatus for making beverages
- A47J31/44—Parts or details or accessories of beverage-making apparatus
- A47J31/46—Dispensing spouts, pumps, drain valves or like liquid transporting devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
- G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/56—Extraction of image or video features relating to colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Food Science & Technology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The present invention discloses Intelligent control device of water dispenser and its control method based on RGB-D cameras, including：RGB-D cameras, control and information storage/processing unit and power supply；The output end connection control of power supply and the power input of information storage/processing unit, for directly or indirectly in order to control and information storage/processing unit and RGB-D cameras provide working power；Control and information storage/processing unit and RGB-D phase mechatronics.The water intaking container that the present invention is obtained by RGB-D cameras（Cup）Depth image, extraction water dispenser water intaking container（Cup）Opening's edge position and its height etc. characteristic informations, realize intelligent control water dispenser startup and close effluent functions.
Description
Technical field
The invention belongs to water dispenser technical fields, and in particular to a kind of water outlet intelligent controlling device of water dispenser and its control
Method.
Background technology
Continuous with water dispenser is popularized, and consumers in general are also higher and higher to the functional requirement of water dispenser.Current drink
Water dispenser must complete quantitative water intaking by button operation, can not achieve intelligent control water intaking process, brought centainly to user
Inconvenience.
Invention content
The technical problem to be solved by the present invention is to provide a kind of water fountain effluent intelligent controlling device and its control methods, with RGB-D
Camera technique means judge below water outlet of drinking machine whether with the presence of cup and cup opening's edge whether to be directed at water outlet, lead to
Above-mentioned judgement is crossed to control the water outlet flow and intelligent control water yield that open or close water dispenser.
To realize that above-mentioned control device and its control method, the technical solution used in the present invention are as follows：
A kind of Intelligent control device of water dispenser based on RGB-D cameras, the water fountain effluent based on RGB-D cameras are intelligently controlled
Device processed includes：RGB-D cameras, control and information storage/processing unit and power supply；The output end connection control of power supply and information
The power input of storage/processing unit, for directly or indirectly in order to control and information storage/processing unit and RGB-D cameras
Working power is provided；Control and information storage/processing unit and RGB-D phase mechatronics.
Further, the above-mentioned Intelligent control device of water dispenser based on RGB-D cameras, RGB-D cameras are fixedly mounted on drink
On water dispenser body, close on water outlet of drinking machine；The target area of RGB-D camera setup photographs includes multiple positions in target area
In fixed index point on water dispenser, as establishing three-dimensional system of coordinate calibration point.
Further, the above-mentioned Intelligent control device of water dispenser based on RGB-D cameras, RGB-D cameras include colored phase
Machine and depth camera：Color camera shoots RGB color image, and depth camera shoots depth image；The work of depth camera therein
It is one kind in following three kinds of principles as principle：I）Structural light measurement principle, II）Binocular parallax measuring principle, III）TOF flies
Time measuring principle.
Further, the above-mentioned Intelligent control device of water dispenser based on RGB-D cameras, it is characterised in that；It further include auxiliary
Lighting unit, power supply provide working power for it；Floor light unit is fixedly mounted on the water dispenser body on target area top
On.
Water dispenser intelligent control method based on RGB-D cameras, includes the following steps：
S1. the registration rear backdrop model of target area is established：Target area does not have foreign body, control and information storage/processing
Unit is by the background color image and depth image of RGB-D camera synchronization gains target area and is pocessed, and image is turned
It is changed to background point cloud chart picture, coloured image and depth image are registrated, color image pixel point is established and depth image is deep
The correspondence between information is spent, to obtain the location of pixels and depth image transformation model of coloured image；Control and information
Storage/processing unit includes according to the location of pixels that obtains of registration and depth conversion model, method for registering：Using feature based point
Matched registration, wherein detecting RGB- with Scale invariant features transform algorithm (SIFT) or other relevant characteristic point algorithms
The characteristic point of the coloured image and depth image of D camera synchronization gains, and the characteristic point of above-mentioned image is matched；With random
Consistency algorithm (RANSAC) or other algorithms remove the matching double points of above-mentioned image mistake, obtain correct matched characteristic point
It is right；According to the position of correct matched characteristic point pair and depth information, the correspondence between them is obtained, this two frame is calculated
Location of pixels between image and depth conversion model, depth image after the registration to obtain higher resolution；
According to the parameter of the position three-dimensional coordinate information and RGB-D cameras of known index point and water outlet, target area is established
The registration rear backdrop model in domain, control and information storage/processing unit store the registration rear backdrop model；
S2. foreground depth image after the registration of extraction target area：Foreign body is placed in target area, and water outlet of drinking machine does not go out
In the case of water, control and information storage/processing unit synchronize the coloured image for obtaining target area in real time by RGB-D cameras
And depth image, noise threshold processing, smoothing processing etc. are done to coloured image and depth image, carry out matching described in S1 steps
Standard realizes foreground and background segmentation, and efficient algorithm is selected to extract prospect profile, the i.e. profile of target area foreign body, selection
Efficient algorithm extracts the characteristic point of foreign body, obtains foreign body such as regional area pixels tall pole：Regional area pole
Extremely low point of high point, regional area etc., the three-dimensional coordinate information of the characteristic points such as pixel peak, pixel minimum point；
S3. judge whether foreign body is cup in foreground depth image after being registrated：Basis for estimation is foreground depth figure after registration
It whether there is a pixel subset with the following characteristics in the universal class of foreign body all pixels point as in：1）The picture
Vegetarian refreshments subset includes maximum pixel point or the high point of regional area pixel in the universal class, and 2）Pixel is in xy in the subset
Subpoint in plane is distributed in the close ring that a narrowband surrounds, and the width of narrowband is dW, and 3）The pixel universal class
The pixel in the close ring central area is fallen in three-dimensional coordinate not comprising those subpoints on x/y plane in the subset
It is that height value on z coordinate axis is far smaller than height value of the pixel on three-dimensional system of coordinate z coordinate axis in the subset；
Pixel in the subset includes that those in foreground depth image process, are carried out after extraction is registrated with efficient algorithm
Eliminate noise, the singular point that the means such as smoothing processing are filled by Mathematical Fitting, the pixel quantity of the singular point institute in the subset
The ratio accounted for is no more than certain threshold value；
If depth image meets features described above after the registration of the foreign body, after the registration of extraction in foreground depth image
Foreign body is then confirmed as having complete cup opening's edge, the cup with certain depth；
Control and information storage/processing unit are divided by using effective algorithm from depth image after the registration of cup, are carried
Cup opening's edge and its interior zone depth image are taken, as monitoring area；
S4. judge whether cup opening's edge is directed at water outlet of drinking machine：According to the registration rear backdrop mould for the target area having built up
The location parameter of type and water outlet in three-dimensional system of coordinate, control and information storage/processing unit judge water outlet in x/y plane
Whether upper projection falls inside the close ring that the projection in the corresponding pixel of cup opening's edge on x/y plane surrounds, and leaves closure
The nearest distance in ring edge is more than a certain predetermined value L；
Further, control and information storage/processing unit obtain water outlet according to known coordinate information by analytic operation
With the degree of cup opening's edge central area alignment, degree of registration, is set in different specific heights of water level and closes water dispenser accordingly
The water of specific quantity is injected in water outlet to cup；
S5. water dispenser is to water filling in cup：By above step, the control and information storage/processing unit determine water outlet
It projects and is fallen really inside the close ring that the projection in the corresponding pixel of cup opening's edge on x/y plane surrounds on x/y plane, and
The nearest distance in edge for leaving close ring is more than a certain predetermined value L, i.e. water outlet of drinking machine is targeted by the opening's edge of cup really,
Control and information storage/processing unit send out instruction：Open water outlet of drinking machine flow, to cup in water filling；
S6. the height of water level injected in cup detects in real time：In water dispenser into the injecting process in cup, control and information storage
Deposit/processing unit control RGB-D cameras obtain the cromogram of the monitoring area with the speed sync of certain frame number per second in real time
Picture and depth image, and colored to the monitoring area in real time and depth image is pocessed, is registrated, and analyzes, it is flat to calculate xy
Height value of the pixel on three-dimensional system of coordinate z coordinate axis on face after the corresponding registration of close ring internal point in depth image,
Pixel after the corresponding registration of the close ring internal point in depth image is the pixel on the real-time cup water surface, from
And obtain the real-time height value of water level in cup；
Control and information storage/processing unit processes, registration, analysis, the coloured image and depth for calculating water level in cup
Image process includes：The noise that the flow that water outlet of drinking machine flows down is brought is filtered out by effective algorithm, and rejects flow
Fall the disturbing factors such as the noise that the spray splash and vapor are brought；
S7. completion cup water intaking process：When control and information storage/processing unit pass through the synchronization that is obtained to RGB-D cameras
The processing of the real-time monitoring area coloured image and depth image, registration, analyze, the water level that calculates in cup reach it is default
When fixed specific height of water level, control and information storage/processing unit send out instruction：Water fountain effluent is closed, is stopped to cup
Middle water filling stops shoot coloured image and depth image；
S8. the registration rear backdrop model of target area is updated：After water intaking person removes cup from target area, in certain time, do not have
There are object or cup to be again introduced into target area, control and information storage/processing unit start RGB-D camera synchronization gain multiframes
The coloured image and depth image of target area establish target area newer registration rear backdrop mould again according to S1 steps
Type is used in combination the newly-established update registration rear backdrop model to substitute the original back of the body stored in control and information storage/processing unit
Scape model, the S1 steps as water intaking next time：Establish the registration rear backdrop model of target area.
Further, the above-mentioned water fountain effluent intelligent control method based on RGB-D cameras, the control and information storage
Deposit/processing unit according to the lighting condition of the preset target area, judge to open floor light unit, to improve target
The optical illumination condition in region improves the picture quality of shoot coloured image and depth image, such as：Clarity and details differentiate energy
Power etc.；After opening floor light unit, similarly implement above-mentioned S1-S8 steps, wherein after the completion of S7 steps, control and information
Storage/processing unit sends out instruction, closes floor light unit.
Advantageous effect：The present invention realizes that water dispenser water intaking process is intelligent, according to the different water withdrawal demands of user,
Automatic outletting prevents spilling water；It simultaneously effective avoids and is accidentally discharged caused by misoperation.
The present invention uses RGB-D cameras, can ensure to obtain high-resolution depth image, contribute to the knowledge of depth image
Not.
Description of the drawings
Fig. 1 is that the present invention is based on the modules without floor light unit in the Intelligent control device of water dispenser of RGB-D cameras to show
It is intended to.
Fig. 2 is that the present invention is based on the water dispenser structural schematic diagrams of the Intelligent control device of water dispenser of RGB-D cameras.
Fig. 3 is that the present invention is based on have the module of floor light unit to show in the Intelligent control device of water dispenser of RGB-D cameras
It is intended to.
Fig. 4, which is that the present invention is based on the step of water dispenser intelligent control method of RGB-D cameras, to scheme.
Fig. 5 is that the present invention is based on the flows without floor light unit in the water dispenser intelligent control method of RGB-D cameras
Figure.
Specific implementation mode
A kind of Intelligent control device of water dispenser based on RGB-D cameras, the water fountain effluent based on RGB-D cameras are intelligently controlled
Device 1 processed includes：RGB-D cameras 2, control and information storage/processing unit 3 and power supply 4；The output end connection control of power supply 4
And the power input of information storage/processing unit 3, for directly or indirectly in order to control and information storage/3 He of processing unit
RGB-D cameras 2 provide working power；Control and information storage/processing unit 3 are electrically connected with RGB-D cameras 2, as shown in Figure 1.
The above-mentioned Intelligent control device of water dispenser based on depth camera, the RGB-D cameras include color camera and
Depth camera：Color camera shoots RGB color image, and depth camera shoots depth image；The work of depth camera therein is former
Reason is one kind in following three kinds of principles：I）Structural light measurement principle, II）Binocular parallax measuring principle, III）The TOF flight time
Measuring principle.
The above-mentioned Intelligent control device of water dispenser based on RGB-D cameras, water outlet of drinking machine lower section, when user fetches water
The region that cup is likely to occur is the target area of RGB-D cameras 2 shoot coloured image and depth image；Include in target area
The multiple and fixed index point of water dispenser relative position, as three-dimensional system of coordinate calibration point, for the coloured image by obtaining
It is target area background modeling with depth image；RGB-D cameras 2 are fixedly mounted on the top water dispenser body of target area,
Water outlet of drinking machine is closed on, it is positioned such that RGB-D cameras 2 can either obtain the coloured image and depth map of entire target area
Picture, and do not interfere water dispenser to water filling in cup, RGB-D cameras 2 are fixed relative to water dispenser position.
Fig. 2 show the water dispenser schematic diagram that the Intelligent control device of water dispenser based on RGB-D cameras is related to；Its
In：ABCD is the plane that the pedestal cup in water dispenser water intaking region is placed, and P is water outlet, and o-xyz is the three-dimensional where water dispenser
Space coordinates, ABCD is on x/y plane.
The above-mentioned Intelligent control device of water dispenser based on RGB-D cameras further includes floor light unit 5, such as Fig. 3 institutes
Show, power supply 4 provides working power for it；Floor light unit 5 is fixedly mounted on the top water dispenser body of target area, i.e.,
It can be irradiated to target area, and do not interfere water dispenser to the place of water filling in cup.
The above-mentioned water dispenser intelligent control method based on RGB-D cameras, includes the following steps, as shown in Figure 4：
S1. the registration rear backdrop model of target area is established：Target area does not have foreign body, control and information storage/processing
Unit is by the background color image and depth image of RGB-D camera synchronization gains target area and is pocessed, and image is turned
It is changed to background point cloud chart picture, coloured image and depth image are registrated, establishes color image information and depth image depth
Correspondence between information, to obtain the location of pixels and depth image transformation model of coloured image；Control and information storage
Deposit/processing unit includes according to the location of pixels that obtains of registration and depth conversion model, method for registering：Using feature based point
The registration matched, wherein detecting RGB-D with Scale invariant features transform algorithm (SIFT) or other relevant characteristic point algorithms
The coloured image of camera synchronization gain and the characteristic point of depth image, and the characteristic point of above-mentioned image is matched；With random
Consistency algorithm (RANSAC) or other algorithms remove the matching double points of above-mentioned image mistake, obtain correct matched characteristic point
It is right；According to the position of correct matched characteristic point pair and depth information, the correspondence between them is obtained, calculates this colour
Location of pixels between image and depth image and depth conversion model, depth map after the registration to obtain higher resolution
Picture；
According to the parameter of the position three-dimensional coordinate information and RGB-D cameras of known index point and water outlet, target area is established
The registration rear backdrop model in domain, control and information storage/processing unit store the registration rear backdrop model；
S2. foreground depth image after the registration of extraction target area：Foreign body is placed in target area, and water outlet of drinking machine does not go out
In the case of water, control and information storage/processing unit synchronize the coloured image for obtaining target area in real time by RGB-D cameras
And depth image, noise threshold processing is done to coloured image and depth image（Such as：Median filter method）, smoothing processing etc.,
The registration described in S1 steps is carried out, realizes that foreground and background segmentation, selection use efficient algorithm（Such as：Watershed, seed are searched
Rope, background subtraction, Binarization methods etc.）, the profile of foreground is extracted, i.e. the profile of foreign body described in target area, selection has
The algorithm of effect（Such as：ORB, SIFT, FAST, MSER, STAR etc.）, the characteristic point of the foreign body is extracted,
Obtain the foreign body such as regional area pixels tall pole（The high point of regional area, the extremely low point of regional area etc.）, as
The three-dimensional coordinate information of the characteristic points such as plain peak, pixel minimum point；
S3. judge whether foreign body is cup in foreground depth image after being registrated：Basis for estimation is foreground depth figure after registration
It whether there is a pixel subset with the following characteristics in the universal class of foreign body all pixels point as in：1）The picture
Vegetarian refreshments subset includes maximum pixel point or the high point of regional area pixel in the universal class, and 2）Pixel is in xy in the subset
Subpoint in plane is distributed in the close ring that a narrowband surrounds, and the width of narrowband is dW, such as：DW=2mm, 3）It is described
Pixel universal class does not include those subpoints on x/y plane in the subset and falls the pixel in the close ring central area
Height value of the point on three-dimensional system of coordinate z coordinate axis is far smaller than in the subset pixel on three-dimensional system of coordinate z coordinate axis
Height value, such as：Height and the son of the pixel of the close ring central area on three-dimensional system of coordinate z coordinate axis
The difference of height of the pixel on three-dimensional system of coordinate z coordinate axis is concentrated to be more than 20mm；
Pixel in the subset includes that those in foreground depth image process, are carried out after extraction is registrated with efficient algorithm
Eliminate noise, the singular point that the means such as smoothing processing are filled by Mathematical Fitting, the pixel quantity of the singular point institute in the subset
The ratio accounted for is no more than certain threshold value, such as：Less than 5%；
If depth image meets features described above after the registration of the foreign body, after the registration of extraction in foreground depth image
Foreign body is then confirmed as having complete cup opening's edge, the cup with certain depth；
Further, control and information storage/processing unit 3 from depth image after the registration of cup by using effective algorithm
Segmentation, extraction cup rim of the mouth edge and its interior zone depth image, as monitoring area；
S4. judge whether cup opening's edge is directed at water outlet of drinking machine：According to the registration rear backdrop mould for the target area having built up
The location parameter of type and water outlet in three-dimensional system of coordinate, control and information storage/processing unit 3 judge water outlet in x/y plane
Whether upper projection falls inside the close ring that the projection in the corresponding pixel of cup opening's edge on x/y plane surrounds, and leaves closure
The nearest distance in ring edge is more than a certain predetermined value L, such as：L=5mm；
Further, control and information storage/processing unit 3 obtain water outlet according to known coordinate information by analytic operation
With the degree of cup opening's edge central area alignment, degree of registration, is set in different specific heights of water level and closes water dispenser accordingly
The water of specific quantity is injected in water outlet to cup；
Such as：If water outlet faces the central area of cup opening's edge, to the preset water level height distance cup of cup injection
Rim of the mouth is along height difference 10mm；If projection of the water outlet on x/y plane is positioned at the edge and central area of the close ring
Centre position, then to cup injection preset water level height distance cup opening's edge height differ 30mm；And so on, if gone out
Central area of the mouth of a river far from cup opening's edge, then it is lower to the height of water level of cup injection, until water outlet is in x/y plane
On projection, leave the small Mr. Yu's setting value L of minimum distance for the close ring that cup opening's edge projects formation on x/y plane（It is assumed to be
5mm）, then it is assumed that the flow of water outlet outflow cannot safely entirely fall in cup, in the case, the control and information
Storage/processing unit will not open water outlet to cup water filling；
S5. water dispenser is to water filling in cup：By above step, the control and information storage/processing unit 3 determine water outlet
It projects and is fallen really inside the close ring that the projection in the corresponding pixel of cup opening's edge on x/y plane surrounds on x/y plane, and
The nearest distance in edge for leaving close ring is more than a certain predetermined value L（Such as：L=5mm）, i.e. water outlet of drinking machine is aligned really
The opening's edge of cup, control and information storage/processing unit 3 send out instruction：Open water outlet of drinking machine flow, to cup in note
Water；
S6. the height of water level injected in cup detects in real time：In water dispenser into the injecting process in cup, the control and information
Storage/processing unit 3 controls the RGB-D cameras and obtains the monitoring area in real time with the speed sync of certain frame number per second
Coloured image and depth image, and the coloured image to the monitoring area and depth image are pocessed, are registrated in real time, point
It analyses, calculate pixel on x/y plane after the corresponding registration of close ring internal point in depth image in three-dimensional system of coordinate z coordinate
Height value on axis, the pixel after the corresponding registration of the close ring internal point in depth image is the real-time cup water surface
On pixel, to obtain the real-time height value of water level in cup,
The control and information storage/processing unit 3 processing, registration, analysis, calculate in cup the coloured image of water level and
Depth image process includes：The noise that the flow that water outlet of drinking machine flows down is brought is filtered out by effective algorithm, and is rejected
Flow falls the disturbing factors such as the noise that the spray splash and vapor are brought；
S7. completion cup water intaking process：When control and information storage/processing unit 3 pass through the synchronization that is obtained to RGB-D cameras
The processing of the real-time monitoring area coloured image and depth image, registration, analyze, the water level that calculates in cup reach it is default
When fixed specific height of water level, control and information storage/processing unit 3 send out instruction：Water fountain effluent is closed, is stopped to cup
Middle water filling stops shoot coloured image and depth image；
S8. the registration rear backdrop model of target area is updated：After water intaking person removes cup from target area, in certain time, do not have
There are object or cup to be again introduced into target area, it is more that control and information storage/processing unit 3 start RGB-D camera synchronization gains
The coloured image and depth image of frame target area establish target area newer registration rear backdrop again according to S1 steps
Model, be used in combination newly-established update registration rear backdrop model substitute stored in control and information storage/processing unit 3 it is original
Background model, the S1 steps as water intaking next time：Establish the registration rear backdrop model of target area.
The implementing procedure figure of water dispenser water intaking process described above is as shown in Figure 5.
The above-mentioned water dispenser intelligent control method based on RGB-D cameras, the control and information storage/processing unit 3
According to the different optical illumination conditions of the preset target area, such as：Ambient light is too miscellaneous, extraneous photo-labile etc., judges
The floor light unit 5 is opened, to improve the optical illumination condition of target area, improves the picture quality of acquisition, such as：Clearly
Clear degree and details resolution capability etc.；After opening the floor light unit 5, similarly implement the S1-S8 steps, wherein in S7
After the completion of step, the control and information storage/processing unit 3 send out instruction, close the floor light unit 5.
Claims (6)
1. the Intelligent control device of water dispenser based on RGB-D cameras, it is characterised in that：The water dispenser based on RGB-D cameras
It is discharged intelligent controlling device（1）Including：RGB-D cameras（2）, control and information storage/processing unit（3）And power supply（4）；Power supply
（4）Output end connection control and information storage/processing unit（3）Power input, for directly or indirectly in order to control and
Information storage/processing unit（3）With RGB-D cameras（2）Working power is provided；Control and information storage/processing unit（3）With
RGB-D cameras（2）Electrical connection.
2. the Intelligent control device of water dispenser according to claim 1 based on RGB-D cameras, it is characterised in that：RGB-D phases
Machine（2）It is fixedly mounted on water dispenser body, closes on water outlet of drinking machine；RGB-D cameras（2）The target area of shooting is set,
Include in target area it is multiple be located at fixed index point on water dispenser, as establishing three-dimensional system of coordinate calibration point.
3. the Intelligent control device of water dispenser according to claim 1 or 2 based on RGB-D cameras, which is characterized in that RGB-
D cameras include color camera and depth camera：Color camera shoots RGB color image, and depth camera shoots depth image；Its
In the operation principle of depth camera be one kind in following three kinds of principles：I）Structural light measurement principle, II）Binocular parallax measures
Principle, III）TOF flight time measurement principles.
4. the Intelligent control device of water dispenser according to claim 1 based on RGB-D cameras, it is characterised in that；Further include
Floor light unit（5）, power supply（4）Working power is provided for it；Floor light unit（5）It is fixedly mounted on target area top
Water dispenser body on.
5. the water dispenser intelligent control method based on RGB-D cameras, is characterized in that：Include the following steps：
S1. the registration rear backdrop model of target area is established：Target area does not have foreign body, control and information storage/processing
Unit（3）Pass through RGB-D cameras（2）The background color image and depth image of synchronization gain target area are simultaneously pocessed, will
Image is converted to background point cloud chart picture, is registrated to coloured image and depth image, establishes color image pixel point and depth
Correspondence between image depth information, to obtain the location of pixels and depth image transformation model of coloured image；Control
And information storage/processing unit（3）Include according to the location of pixels that obtains of registration and depth conversion model, method for registering：Using
The matched registration of feature based point, wherein with Scale invariant features transform algorithm (SIFT) or other relevant characteristic point algorithms
Come detect RGB-D camera synchronization gains coloured image and depth image characteristic point, and to the characteristic point of above-mentioned image carry out
Matching；The matching double points that above-mentioned image mistake is removed with random consistency algorithm (RANSAC) or other algorithms obtain correct
Matched characteristic point pair；According to the position of correct matched characteristic point pair and depth information, the correspondence between them is obtained,
Calculate the location of pixels and depth conversion model between this two field pictures, depth map after the registration to obtain higher resolution
Picture；
According to the position three-dimensional coordinate information and RGB-D cameras of known index point and water outlet（2）Parameter, establish target
The registration rear backdrop model in region, control and information storage/processing unit（3）Store the registration rear backdrop model；
S2. foreground depth image after the registration of extraction target area：Foreign body is placed in target area, and water outlet of drinking machine does not go out
In the case of water, control and information storage/processing unit（3）Pass through RGB-D cameras（2）Synchronize the coloured silk for obtaining target area in real time
Color image and depth image do noise threshold processing, smoothing processing etc. to coloured image and depth image, carry out described in S1 steps
Registration, realize foreground and background segmentation, efficient algorithm selected to extract prospect profile, i.e. the profile of target area foreign body,
The characteristic point for selecting efficient algorithm extraction foreign body, obtains foreign body such as regional area pixels tall pole：Partial zones
The high point in domain, the extremely low point of regional area etc., the three-dimensional coordinate information of the characteristic points such as pixel peak, pixel minimum point；
S3. judge whether foreign body is cup in foreground depth image after being registrated：Basis for estimation is foreground depth figure after registration
It whether there is a pixel subset with the following characteristics in the universal class of foreign body all pixels point as in：1）The picture
Vegetarian refreshments subset includes maximum pixel point or the high point of regional area pixel in the universal class, and 2）Pixel is in xy in the subset
Subpoint in plane is distributed in the close ring that a narrowband surrounds, and the width of narrowband is dW, and 3）The pixel universal class
The pixel in the close ring central area is fallen in three-dimensional coordinate not comprising those subpoints on x/y plane in the subset
It is that height value on z coordinate axis is far smaller than height value of the pixel on three-dimensional system of coordinate z coordinate axis in the subset；
Pixel in the subset includes that those in foreground depth image process, are carried out after extraction is registrated with efficient algorithm
Eliminate noise, the singular point that the means such as smoothing processing are filled by Mathematical Fitting, the pixel quantity of the singular point institute in the subset
The ratio accounted for is no more than certain threshold value；
If depth image meets features described above after the registration of the foreign body, after the registration of extraction in foreground depth image
Foreign body is then confirmed as having complete cup opening's edge, the cup with certain depth；
Control and information storage/processing unit（3）Divided by using effective algorithm from depth image after the registration of cup,
Extraction cup rim of the mouth edge and its interior zone depth image, as monitoring area；
S4. judge whether cup opening's edge is directed at water outlet of drinking machine：According to the registration rear backdrop mould for the target area having built up
The location parameter of type and water outlet in three-dimensional system of coordinate, control and information storage/processing unit（3）Judge that water outlet is flat in xy
It projects and whether is fallen inside the close ring that the projection in the corresponding pixel of cup opening's edge on x/y plane surrounds on face, and leave and close
The nearest distance in cyclization edge is more than a certain predetermined value L；
Further, control and information storage/processing unit（3）It is discharged by analytic operation according to known coordinate information
The degree of mouth and the alignment of cup opening's edge central area, degree of registration, is set in different specific heights of water level and closes drinking-water accordingly
Machine is discharged, and the water of specific quantity is injected to cup；
S5. water dispenser is to water filling in cup：By above step, the control and information storage/processing unit（3）Determine water outlet
Mouth projects on x/y plane to be fallen really inside the close ring that the projection in the corresponding pixel of cup opening's edge on x/y plane surrounds,
And the nearest distance in the edge that leaves close ring is more than a certain predetermined value L, i.e. water outlet of drinking machine is targeted by the mouth of cup really
Edge, control and information storage/processing unit（3）Send out instruction：Open water outlet of drinking machine flow, to cup in water filling；
S6. the height of water level injected in cup detects in real time：In water dispenser into the injecting process in cup, control and information storage
Deposit/processing unit（3）Control RGB-D cameras（2）The monitoring area is obtained in real time with the speed sync of certain frame number per second
Coloured image and depth image, and colored to the monitoring area in real time and depth image is pocessed, is registrated, and analyzes, calculates
Go out height of the pixel on three-dimensional system of coordinate z coordinate axis on x/y plane after the corresponding registration of close ring internal point in depth image
Angle value, the pixel after the corresponding registration of the close ring internal point in depth image are the pixel on the real-time cup water surface
Point, to obtain the real-time height value of water level in cup；
Control and information storage/processing unit（3）Processing, registration, analysis, the coloured image and depth for calculating water level in cup
Spending image process includes：The noise that the flow that water outlet of drinking machine flows down is brought is filtered out by effective algorithm, and rejects water
Wander down the disturbing factors such as the noise that the spray splash and vapor are brought；
S7. completion cup water intaking process：When control and information storage/processing unit（3）By to RGB-D cameras（2）It obtains
The processing for synchronizing the real-time monitoring area coloured image and depth image, registration, analyze, the water level that calculates in cup arrives
When up to presetting specific height of water level, control and information storage/processing unit（3）Send out instruction：Water fountain effluent is closed, is stopped
The only water filling into cup stops shoot coloured image and depth image；
S8. the registration rear backdrop model of target area is updated：After water intaking person removes cup from target area, in certain time, do not have
There are object or cup to be again introduced into target area, control and information storage/processing unit（3）Start RGB-D cameras（2）It synchronizes and obtains
The coloured image and depth image for obtaining multiframe target area, according to S1 steps, after establishing newer registration again to target area
Background model is used in combination the newly-established update registration rear backdrop model to substitute control and information storage/processing unit（3）Middle storage
Original background model, as next time water intaking S1 steps：Establish the registration rear backdrop model of target area.
6. the water dispenser intelligent control method according to claim 5 based on RGB-D cameras, it is characterised in that：Control and
Information storage/processing unit（3）According to the lighting condition of the preset target area, judge to open floor light unit（5）,
To improve the optical illumination condition of target area, the clarity and details resolution capability of shoot coloured image and depth image are improved
Etc. picture qualities；Open floor light unit（5）Afterwards, implementation steps include S1-S8 in claim 5, wherein complete in S7 steps
Cheng Hou, control and information storage/processing unit（3）Instruction is sent out, floor light unit is closed（5）.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810306420.0A CN108742159B (en) | 2018-04-08 | 2018-04-08 | Intelligent control device and method for water dispenser based on RGB-D camera |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810306420.0A CN108742159B (en) | 2018-04-08 | 2018-04-08 | Intelligent control device and method for water dispenser based on RGB-D camera |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108742159A true CN108742159A (en) | 2018-11-06 |
| CN108742159B CN108742159B (en) | 2021-01-15 |
Family
ID=63981105
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810306420.0A Active CN108742159B (en) | 2018-04-08 | 2018-04-08 | Intelligent control device and method for water dispenser based on RGB-D camera |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108742159B (en) |
Cited By (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109717749A (en) * | 2018-11-30 | 2019-05-07 | 青岛海尔智能技术研发有限公司 | A kind of Intelligent water injection device and method |
| CN110211093A (en) * | 2019-04-30 | 2019-09-06 | 上海工程技术大学 | A kind of water outlet control method for automatic drinking water apparatus |
| CN111568234A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water surface detection method, water dispenser and computer readable storage medium |
| CN111568211A (en) * | 2020-02-28 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water dispenser control method, water dispenser and computer readable storage medium |
| CN111568212A (en) * | 2020-02-28 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water dispenser control method, water dispenser and computer readable storage medium |
| CN111568228A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water purifying and drinking machine control method, water purifying and drinking machine and readable storage medium |
| CN111568237A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water dispenser control method, water dispenser and computer readable storage medium |
| CN111568235A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water dispenser control method, water dispenser and computer readable storage medium |
| CN111568220A (en) * | 2020-02-28 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water dispenser control method, water dispenser and computer readable storage medium |
| CN111568225A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water dispenser control method, water dispenser and computer readable storage medium |
| CN111568232A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water outlet control method of water dispenser, water dispenser and computer readable storage medium |
| CN111568230A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water outlet control method of water dispenser, water dispenser and computer readable storage medium |
| CN111568236A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water dispenser control method, water dispenser and computer readable storage medium |
| CN111568196A (en) * | 2020-02-28 | 2020-08-25 | 佛山市云米电器科技有限公司 | Intelligent water outlet method, system and storage medium |
| CN111568239A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water dispenser control method, water dispenser and computer readable storage medium |
| CN111568244A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water dispenser control method, water dispenser and computer readable storage medium |
| CN111568238A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water outlet control method of water dispenser, water dispenser and computer readable storage medium |
| CN111642975A (en) * | 2020-02-29 | 2020-09-11 | 佛山市云米电器科技有限公司 | Water outlet control method, water dispenser and computer readable storage medium |
| CN111820753A (en) * | 2019-04-18 | 2020-10-27 | 佛山市美的清湖净水设备有限公司 | Water dispenser and control method and device thereof |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105654481A (en) * | 2015-12-29 | 2016-06-08 | 深圳市未来媒体技术研究院 | Depth camera-based extravascular leakage condition detection system |
| CN107103311A (en) * | 2017-05-31 | 2017-08-29 | 西安工业大学 | A kind of recognition methods of continuous sign language and its device |
| CN107240129A (en) * | 2017-05-10 | 2017-10-10 | 同济大学 | Object and indoor small scene based on RGB D camera datas recover and modeling method |
| CN107273866A (en) * | 2017-06-26 | 2017-10-20 | 国家电网公司 | A kind of human body abnormal behaviour recognition methods based on monitoring system |
| CN107369158A (en) * | 2017-06-13 | 2017-11-21 | 南京邮电大学 | The estimation of indoor scene layout and target area extracting method based on RGB D images |
| CN107456086A (en) * | 2017-08-25 | 2017-12-12 | 深圳市浮思特科技有限公司 | The water outlet Intelligent Measurement module and water outlet control method of view-based access control model detection |
| CN107578053A (en) * | 2017-09-25 | 2018-01-12 | 重庆虚拟实境科技有限公司 | Contour extraction method and device, computer installation and readable storage medium storing program for executing |
| CN107730526A (en) * | 2017-09-25 | 2018-02-23 | 中国科学院声学研究所 | A kind of statistical method of the number of fish school |
| CN107872644A (en) * | 2016-09-23 | 2018-04-03 | 亿阳信通股份有限公司 | Video frequency monitoring method and device |
-
2018
- 2018-04-08 CN CN201810306420.0A patent/CN108742159B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105654481A (en) * | 2015-12-29 | 2016-06-08 | 深圳市未来媒体技术研究院 | Depth camera-based extravascular leakage condition detection system |
| CN107872644A (en) * | 2016-09-23 | 2018-04-03 | 亿阳信通股份有限公司 | Video frequency monitoring method and device |
| CN107240129A (en) * | 2017-05-10 | 2017-10-10 | 同济大学 | Object and indoor small scene based on RGB D camera datas recover and modeling method |
| CN107103311A (en) * | 2017-05-31 | 2017-08-29 | 西安工业大学 | A kind of recognition methods of continuous sign language and its device |
| CN107369158A (en) * | 2017-06-13 | 2017-11-21 | 南京邮电大学 | The estimation of indoor scene layout and target area extracting method based on RGB D images |
| CN107273866A (en) * | 2017-06-26 | 2017-10-20 | 国家电网公司 | A kind of human body abnormal behaviour recognition methods based on monitoring system |
| CN107456086A (en) * | 2017-08-25 | 2017-12-12 | 深圳市浮思特科技有限公司 | The water outlet Intelligent Measurement module and water outlet control method of view-based access control model detection |
| CN107578053A (en) * | 2017-09-25 | 2018-01-12 | 重庆虚拟实境科技有限公司 | Contour extraction method and device, computer installation and readable storage medium storing program for executing |
| CN107730526A (en) * | 2017-09-25 | 2018-02-23 | 中国科学院声学研究所 | A kind of statistical method of the number of fish school |
Cited By (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109717749A (en) * | 2018-11-30 | 2019-05-07 | 青岛海尔智能技术研发有限公司 | A kind of Intelligent water injection device and method |
| CN111820753A (en) * | 2019-04-18 | 2020-10-27 | 佛山市美的清湖净水设备有限公司 | Water dispenser and control method and device thereof |
| CN110211093A (en) * | 2019-04-30 | 2019-09-06 | 上海工程技术大学 | A kind of water outlet control method for automatic drinking water apparatus |
| CN111568220A (en) * | 2020-02-28 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water dispenser control method, water dispenser and computer readable storage medium |
| CN111568212A (en) * | 2020-02-28 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water dispenser control method, water dispenser and computer readable storage medium |
| CN111568196A (en) * | 2020-02-28 | 2020-08-25 | 佛山市云米电器科技有限公司 | Intelligent water outlet method, system and storage medium |
| CN111568211A (en) * | 2020-02-28 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water dispenser control method, water dispenser and computer readable storage medium |
| CN111568235A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water dispenser control method, water dispenser and computer readable storage medium |
| CN111568237A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water dispenser control method, water dispenser and computer readable storage medium |
| CN111568225A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water dispenser control method, water dispenser and computer readable storage medium |
| CN111568232A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water outlet control method of water dispenser, water dispenser and computer readable storage medium |
| CN111568230A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water outlet control method of water dispenser, water dispenser and computer readable storage medium |
| CN111568236A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water dispenser control method, water dispenser and computer readable storage medium |
| CN111568228A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water purifying and drinking machine control method, water purifying and drinking machine and readable storage medium |
| CN111568239A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water dispenser control method, water dispenser and computer readable storage medium |
| CN111568244A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water dispenser control method, water dispenser and computer readable storage medium |
| CN111568238A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water outlet control method of water dispenser, water dispenser and computer readable storage medium |
| CN111642975A (en) * | 2020-02-29 | 2020-09-11 | 佛山市云米电器科技有限公司 | Water outlet control method, water dispenser and computer readable storage medium |
| CN111568234A (en) * | 2020-02-29 | 2020-08-25 | 佛山市云米电器科技有限公司 | Water surface detection method, water dispenser and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108742159B (en) | 2021-01-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108742159A (en) | Intelligent control device of water dispenser based on RGB-D cameras and its control method | |
| CN108921883A (en) | Water dispenser control device and its control method based on the identification of two positions depth image | |
| CN108427940A (en) | Water fountain effluent intelligent controlling device based on depth camera and its control method | |
| CN107563347B (en) | Passenger flow counting method and device based on TOF camera | |
| CN105346706B (en) | Flight instruments, flight control system and method | |
| CN110211093A (en) | A kind of water outlet control method for automatic drinking water apparatus | |
| CN103473542B (en) | Multi-clue fused target tracking method | |
| CN104299011A (en) | Skin type and skin problem identification and detection method based on facial image identification | |
| CN109167928A (en) | Fast automatic exposure method and system based on defects of display panel detection | |
| CN110276265A (en) | Pedestrian monitoring method and device based on intelligent three-dimensional solid monitoring device | |
| CN105160328B (en) | A kind of human body contour outline recognition methods based on binary image | |
| CN110927174B (en) | White spirit segmentation detection device and method based on embedded module | |
| CN103971135A (en) | Human body target detection method based on head and shoulder depth information features | |
| CN106127732B (en) | Human body sex-screening method and apparatus in microwave imagery | |
| CN109270853A (en) | A kind of working method of intelligent ecological Water Management System | |
| CN109612993A (en) | A kind of part quality detection system and method | |
| CN110210292A (en) | A kind of target identification method based on deep learning | |
| CN106657937A (en) | Virtual clothes color changing device, and implementation method thereof | |
| CN107168117B (en) | A method of wine brewing distillation technique rice steamer material is controlled using image | |
| CN106388441A (en) | Method and device for displaying and intelligent mirror | |
| CN105631868A (en) | Depth information extraction method based on image classification | |
| CN108814444A (en) | A kind of sweeping robot leg follows cleaning method and device | |
| CN107680076A (en) | Stereo-picture comfort level contrast range measuring method based on salient region | |
| CN107944424A (en) | Front end human image collecting and Multi-angle human are distributed as comparison method | |
| CN106131448A (en) | The 3 d stereoscopic vision system of brightness of image can be automatically adjusted |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20191211 Address after: 214031 No. 32 Wuai North Road, Liangxi District, Wuxi City, Jiangsu Province Applicant after: Xu Yu Address before: 311800 Jiefang Road 698, Dian Kou Town, Zhuji City, Shaoxing, Zhejiang Applicant before: Zhejiang an Jing Intelligent Technology Co., Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |