WO2021057394A1

WO2021057394A1 - Robot meal delivery method and system based on machine vision

Info

Publication number: WO2021057394A1
Application number: PCT/CN2020/112532
Authority: WO
Inventors: 翟懿奎; 邓文博; 柯琪锐; 甘俊英
Original assignee: 五邑大学
Priority date: 2019-09-29
Filing date: 2020-08-31
Publication date: 2021-04-01
Also published as: CN110605728B; CN110605728A

Abstract

Disclosed are a robot meal delivery method and system based on machine vision. The method comprises: a client receives order information of a user, and sends the order information to a server; the server responds to the information and schedules a visual robot to go to a merchant restaurant to wait for taking a meal; the visual robot performs three-dimensional reconstruction on a store environment of the merchant restaurant, and outputs a three-dimensional diagram to the server; when an incubator mounted on the visual robot receives foods, the visual robot automatically walks to the location of the client by means of a GPS positioning module and a binocular camera; the visual robot sends a meal taking verification code to the client by means of the server, and waits for meal taking; and when the visual robot receives the correct meal taking verification code, the incubator is unlocked, and the meal delivery is completed. According to the method and the system, automatic meal delivery can be achieved, a restaurant situation can also be photographed and displayed during meal delivery, transparent meal delivery is achieved, food safety is guaranteed, and in addition, the incubator can ensure the temperature of the foods so as to improve the user experience sense.

Description

Robot food delivery method and food delivery system based on machine vision

Technical field

The invention relates to the technical field of intelligent meal delivery, in particular to a robot meal delivery method and a meal delivery system based on machine vision.

Background technique

With the rapid development of society, people's work intensity is increasing, and the pace of life is getting faster and faster, leading to the proliferation of some food delivery platforms and merchants. However, the existing food delivery methods have the following situations. First: With the gradual expansion of the food delivery market and the increase in the number of food delivery personnel, the code of conduct of the food delivery staff has become difficult to regulate and manage, making the food delivery There are frequent incidents such as people entering the house, disputes with businesses, and traffic accidents; second: At the same time, due to insufficient supervision of the food delivery platform, some businesses that do not meet the hygiene standards have entered the food delivery platform to deliver unhygienic food to users, resulting in a series of health Problems and safety hazards. Third: Since the current delivery mode of food delivery personnel is one person, one area and multiple orders at the same time, this will cause hot food to become cold and ice food to melt, making the user experience poor.

Summary of the invention

The present invention aims to solve at least one of the technical problems existing in the prior art. For this reason, the present invention proposes a robot food delivery method based on machine vision, which can realize automatic food delivery and is intelligent and safe; and during the food delivery period, the restaurant will be photographed and displayed to achieve transparent food delivery and ensure food to the greatest extent. Hygiene; in addition, a constant temperature box is also set up, which can ensure the temperature of the food to the greatest extent to enhance the user experience.

The present invention also provides a food delivery system with the above-mentioned machine vision-based robot food delivery method.

According to the embodiment of the first aspect of the present invention, the robot food delivery method based on machine vision includes the following steps:

The client receives the user's order information, and sends the order information to the server;

The server responds and dispatches the vision robot to the merchant restaurant to wait for the meal;

The vision robot performs three-dimensional reconstruction of the store environment of the business restaurant, and outputs a three-dimensional map to the server;

After the incubator installed on the vision robot receives food, the vision robot automatically walks to the location of the client by using a GPS positioning module and a binocular camera;

The vision robot sends the meal fetching verification code to the client through the server end, and waits for the meal to be fetched;

When the vision robot receives the correct verification code for taking the meal, the incubator is unlocked and the meal delivery is completed.

According to the embodiment of the first aspect of the present invention, the robot food delivery method based on machine vision has at least the following beneficial effects: the present invention can realize automatic food delivery, intelligent and safe; and during the food delivery period, the restaurant will be photographed and displayed to achieve Transparent food delivery guarantees food hygiene to the utmost extent; in addition, a constant temperature box is also set up, which can guarantee the temperature of food to the greatest extent and enhance the user experience.

According to some embodiments of the present invention, the server-side responding and scheduling the vision robot to go to the merchant restaurant to wait for the meal includes:

The server obtains the position information of each of the visual robots, calculates the distance between each of the visual robots and the business restaurant, and selects the visual robot with the smallest distance in an idle state for scheduling.

According to some embodiments of the present invention, the method further includes the following steps:

Judging whether the client terminal receives a user's image viewing instruction;

When received, the server sends the three-dimensional image to the client;

When it is not received, the server side saves the three-dimensional image.

Establishing a user credit rating database on the server side, the user credit rating database including the credit ratings of different users;

When the waiting time of the visual robot to fetch a meal exceeds a preset time value, the server responds by lowering the user's credit rating.

When the user's credit level is lower than the preset level value, the server refuses to receive the order information of the corresponding client.

According to some embodiments of the present invention, the time value is five minutes.

The robot food delivery system based on machine vision according to the embodiment of the second aspect of the present invention includes:

Vision robot, equipped with communication module, GPS positioning module, binocular camera, walking wheels, incubator and password lock;

Server side, connected to the communication module;

The client is connected to the server.

According to some embodiments of the present invention, the vision robot is further equipped with a display screen.

According to some embodiments of the present invention, the walking wheels include driving wheels and steering wheels.

The additional aspects and advantages of the present invention will be partly given in the following description, and partly will become obvious from the following description, or be understood through the practice of the present invention.

Description of the drawings

The above and/or additional aspects and advantages of the present invention will become obvious and easy to understand from the description of the embodiments in conjunction with the following drawings, in which:

FIG. 1 is a flowchart of a robot food delivery method based on machine vision according to an embodiment of the present invention;

2 is a schematic diagram of a three-dimensional reconstruction convolutional neural network applied to a machine vision-based robot food delivery method according to an embodiment of the present invention;

3 is a schematic structural diagram of a robot food delivery system based on machine vision according to an embodiment of the present invention;

FIG. 4 is a diagram of the installation structure of the visual robot in the robot food delivery system based on machine vision according to the embodiment of the present invention.

Reference signs:

Client 100;

Server side 200;

The vision robot 300, the communication module 310, the GPS positioning module 320, the binocular camera 330, the walking wheels 340, the driving wheels 341, the steering wheels 342, the incubator 350, the password lock 360, and the display screen 370.

detailed description

The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings, in which the same or similar reference numerals denote the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary, and are only used to explain the present invention, but should not be understood as limiting the present invention.

In the description of the present invention, it should be understood that the orientation description involved, such as up, down, front, back, left, right, etc. indicates the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, but In order to facilitate the description of the present invention and simplify the description, it does not indicate or imply that the device or element referred to must have a specific orientation, be configured and operate in a specific orientation, and therefore cannot be understood as a limitation to the present invention.

In the description of the present invention, several means one or more, multiple means two or more, greater than, less than, exceeding, etc. are understood to not include the number, and above, below, and within are understood to include the number. If it is described that the first and second are only used for the purpose of distinguishing technical features, and cannot be understood as indicating or implying the relative importance or implicitly specifying the number of the indicated technical features or implicitly specifying the order of the indicated technical features relationship.

In the description of the present invention, unless otherwise clearly defined, terms such as setting, installation, and connection should be understood in a broad sense, and those skilled in the art can reasonably determine the specific meaning of the above terms in the present invention in combination with the specific content of the technical solution.

1, the machine vision-based robot food delivery method according to the embodiment of the first aspect of the present invention includes the following steps:

S1: The client receives the user's order information, and sends the order information to the server;

S2: The server responds and dispatches the vision robot to the merchant restaurant to wait for the meal;

S3: The vision robot performs three-dimensional reconstruction of the store environment of the merchant restaurant, and outputs a three-dimensional image to the server;

S4: After the incubator installed on the vision robot receives food, the vision robot uses the GPS positioning module and binocular camera to automatically walk to the location of the client; fully utilizes the binocular camera and GPS positioning module to achieve three-dimensional Reconfiguration, precise positioning, precise ranging, intelligent obstacle avoidance and compliance with traffic rules;

S5: The vision robot sends the meal retrieval verification code to the client through the server side, and waits for the meal retrieval;

S6: When the vision robot receives the correct verification code for taking the meal, the incubator is unlocked and the meal delivery is completed.

In the embodiment of the present invention, the binocular camera of the vision robot performs real-time three-dimensional reconstruction of the scene to realize human body, object recognition, traffic rule silence and real-time ranging, and realize the vision robot is automatically sent from the restaurant to the user's location, when the vision robot reaches the user's location When using the GPS positioning module to send the user a verification code for the meal. The invention can effectively avoid food delivery disputes caused by human factors and safety hazards caused by food delivery personnel, can liberate productivity to the greatest extent, improve delivery efficiency and user experience to obtain a better return rate. The present invention uses visual robots to replace traditional food delivery staff. From the time the food is put in the incubator to the user, the convolutional neural network is used to realize real-time scene three-dimensional reconstruction, realize intelligent obstacle avoidance, real-time distance measurement, and traffic compliance. rule. When the food is delivered to the user's location, the user will be sent a verification code for the meal and wait for the user to go downstairs to pick up the meal. Secondly, after the server receives the user’s order information, the idle robot that is closest to the merchant rushes to the merchant’s restaurant, uses binocular cameras to reconstruct the merchant’s storefront and dining environment in three dimensions and displays it to the user to achieve transparent food delivery. Ensure food hygiene to the greatest extent. In addition, the present invention is equipped with a constant temperature box inside the vision robot to ensure the temperature of the food to the greatest extent so as to enhance the user experience.

S3a: Determine whether the client terminal receives the user's image viewing instruction;

S3b: When received, the server sends the three-dimensional image to the client;

S3c: When it is not received, the server side saves the three-dimensional image.

S7: Establish a user credit rating database on the server side, where the user credit rating database includes the credit ratings of different users;

S8: When the waiting time of the vision robot to fetch a meal exceeds a preset time value, the server responds to lower the user's credit level.

S9: When the user's credit level is lower than a preset level value, the server refuses to receive the order information of the corresponding client.

The embodiment of the present invention establishes a user credit rating system. When the visual robot waits for a meal downstairs for more than five minutes, the user's credit is downgraded. When the user's credit rating is low to a certain value, the user is prohibited from being unable to do so for a week. Use this food delivery system. This can effectively improve the work efficiency of the visual robot and increase the social credit rating.

Referring to Figure 2, for real-time 3D reconstruction of the scene, the present invention intends to use a deep convolutional neural network to achieve this function. The 3D reconstruction network module includes a coding structure composed of a convolutional neural network, a cyclic learning unit, and a convolutional neural network. Decoding structure composed of product neural network. The coding layer converts the color image into a low-dimensional feature matrix, and passes this matrix to the 3D convolutional long and short memory network layer for cyclic learning. The coding structure is realized by a convolutional neural network. Unlike ordinary convolutional neural networks, the convolutional neural network in the patent of the present invention adds a residual connection between every two convolutional layers. Experiments have proved that this can effectively speed up the optimization process of deep neural networks.

The 3D convolutional long and short-term memory network unit module will selectively update the information in its memory unit according to the read pictures, and it can also selectively retain still useful objects when reading multiple pictures Characteristic information. Every time a picture is read, the 3D long-term and short-term memory network will update the state in its own hidden layer, which contains the characteristic information of the target object and the three-dimensional structure information. In this 3D convolutional long and short-term memory network structure, the core idea is still similar to the recurrent neural network (RNN). The memory unit of the long-term and short-term memory network is updated by reading different pictures at each moment to save the new While retaining the previous information.

After that, the decoding module reads the state of the hidden layer of the 3D long and short-term memory network and converts them into the final pixel block occupancy probability map. Using this probability map, you can start the restoration of the 3D model. The decoding module is also based on the convolutional neural network. The difference is that a three-dimensional convolution layer with a 3×3×3 size convolution kernel is used here, because after the pixel data in the picture is passed through the 3D convolutional long-term and short-term memory network layer, all the pixel information is in two-dimensional On the basis of the information, three-dimensional structural information is given, so ordinary convolutional layers cannot process these data.

3 to 4, according to the second embodiment of the present invention, the machine vision-based robot food delivery system includes:

The vision robot 300 is installed with a communication module 310, a GPS positioning module 320, a binocular camera 330, a walking wheel 340, a thermostat 350 and a password lock 360;

The server 200 is connected to the communication module 310;

The client 100 is connected to the server 200.

The invention can effectively avoid disputes between food delivery personnel and users, and can also effectively avoid some unequal treatment of food delivery personnel. It can alleviate the hygiene and health problems caused by the food delivery system in the market, increase the user experience, and liberate productivity.

According to some embodiments of the present invention, the vision robot 300 is further equipped with a display screen 370.

According to some embodiments of the present invention, the walking wheel 340 includes a driving wheel 341 and a steering wheel 342.

The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above-mentioned embodiments, and within the scope of knowledge possessed by a person of ordinary skill in the technical field, it is possible to make various changes without departing from the purpose of the present invention. Kind of change.

Claims

A robot food delivery method based on machine vision is characterized in that it comprises the following steps:

The client receives the user's order information, and sends the order information to the server;

The server responds and dispatches the vision robot to the merchant restaurant to wait for the meal;

The vision robot performs three-dimensional reconstruction of the store environment of the business restaurant, and outputs a three-dimensional map to the server;

After the incubator installed on the vision robot receives food, the vision robot automatically walks to the location of the client by using a GPS positioning module and a binocular camera;

The vision robot sends the meal fetching verification code to the client through the server end, and waits for the meal to be fetched;

When the vision robot receives the correct verification code for taking the meal, the incubator is unlocked and the meal delivery is completed.
The machine vision-based robot food delivery method according to claim 1, wherein the server responds and dispatches the vision robot to a merchant restaurant to wait for food to be picked up by the server side comprising:

The server obtains the position information of each of the visual robots, calculates the distance between each of the visual robots and the business restaurant, and selects the visual robot with the smallest distance in an idle state for scheduling.
The robot food delivery method based on machine vision according to claim 1, characterized in that it further comprises the following steps:

Judging whether the client terminal receives a user's image viewing instruction;

When received, the server sends the three-dimensional image to the client;

When it is not received, the server side saves the three-dimensional image.
The robot food delivery method based on machine vision according to claim 1, characterized in that it further comprises the following steps:

Establishing a user credit rating database on the server side, the user credit rating database including the credit ratings of different users;

When the waiting time of the visual robot to fetch a meal exceeds a preset time value, the server responds by lowering the user's credit rating.
The robot food delivery method based on machine vision according to claim 4, further comprising the following steps:

When the user's credit level is lower than the preset level value, the server refuses to receive the order information of the corresponding client.
The robot food delivery method based on machine vision according to claim 4, wherein the time value is five minutes.
A robot food delivery system based on machine vision, which is characterized in that it includes:

Vision robot, equipped with communication module, GPS positioning module, binocular camera, walking wheels, incubator and password lock;

Server side, connected to the communication module;

The client is connected to the server.
The robot food delivery system based on machine vision according to claim 7, wherein the vision robot is also equipped with a display screen.
The robot food delivery system based on machine vision according to claim 7, wherein the walking wheels include driving wheels and steering wheels.