CN109032348B - Intelligent manufacturing method and equipment based on augmented reality - Google Patents

Abstract

The purpose of this application is to provide an augmented reality-based intelligent manufacturing method, wherein the method includes: acquiring first position information of a target object in an operation area in a first coordinate system corresponding to the operation area; The coordinate mapping relationship between the second coordinate system corresponding to a user equipment and the first coordinate system, determine the second position information of the target object in the second coordinate system; obtain the operation instruction information corresponding to the target object ; superimpose and display the operation instruction information corresponding to the target object on the target object according to the second position information. The present application guides workers to complete the assembly of the workpiece by presenting some auxiliary information related to the assembly of the workpiece in the current operating area, such as the installation position and sequence of the workpiece, to the user holding the augmented reality device, effectively reducing the cost of assembly and maintenance and the installation process. It reduces the probability of errors, improves production efficiency, and realizes rapid response and precise intervention in the production process.

Description

Augmented reality-based intelligent manufacturing method and equipment

technical field

The present application relates to the field of communications, and in particular, to a technology of intelligent manufacturing based on augmented reality.

Background technique

In the industrial production process, although many production lines are now automated, there are still some links that require manual assembly. The assembly process of industrial equipment is generally complex and involves many parts, which requires very high skills for workers. Not only do workers have to spend a lot of time and energy for professional training, but the key to successful assembly lies in the memory of workers. At present, there are many cases of auxiliary assembly operations relying on paper instructions, which leads to a large workload for workers, high training costs, low assembly efficiency, and an unsatisfactory error rate.

SUMMARY OF THE INVENTION

An object of the present application is to provide an augmented reality-based smart manufacturing method and device.

According to an aspect of the present application, there is provided an augmented reality-based smart manufacturing method at the first user equipment side, the method comprising:

obtaining the first position information of the target object in the operation area in the first coordinate system corresponding to the operation area;

determining the second position information of the target object in the second coordinate system according to the coordinate mapping relationship between the second coordinate system corresponding to the first user equipment and the first coordinate system;

obtaining the operation instruction information corresponding to the target object;

The operation instruction information corresponding to the target object is superimposed and displayed on the target object according to the second position information.

According to another aspect of the present application, an augmented reality-based smart manufacturing method at a network device side is provided, the method comprising:

The next step operation instruction information corresponding to the operation instruction information is sent to the corresponding first user equipment.

According to yet another aspect of the present application, an augmented reality-based smart manufacturing method at a second user equipment side is provided, the method comprising:

Obtain the next operation request submitted by the user through the second user equipment and corresponding to the work instruction information in the corresponding first user equipment, wherein the first user equipment and the second user equipment serve the same intelligent manufacturing Task;

sending the next step operation request to the corresponding network device;

receiving the next-step operation instruction information corresponding to the operation instruction information sent by the network device based on the next-step operation request;

Sending the next step operation instruction information to the first user equipment corresponding to the operation instruction information.

According to one aspect of the present application, an augmented reality-based smart manufacturing method is provided, wherein the method includes:

The first user equipment obtains the first position information of the target object in the operation area in the first coordinate system corresponding to the operation area, according to the second coordinate system corresponding to the first user equipment and the coordinates of the first coordinate system mapping relationship, determine the second position information of the target object in the second coordinate system, obtain the work instruction information corresponding to the target object, and assign the work instruction corresponding to the target object according to the second position information The information is superimposed and displayed on the target object;

The network device sends the next step operation instruction information corresponding to the operation instruction information to the first user equipment;

The first user equipment receives the next-step operation instruction information, and superimposes and displays the next-step operation instruction information on the target object.

According to another aspect of the present application, an augmented reality-based smart manufacturing method is provided, wherein the method includes:

The first user equipment obtains the first position information of the target object in the operation area in the first coordinate system corresponding to the operation area, according to the second coordinate system corresponding to the first user equipment and the coordinates of the first coordinate system mapping relationship, determine the second position information of the target object in the second coordinate system, obtain the work instruction information corresponding to the target object, and assign the work instruction corresponding to the target object according to the second position information The information is superimposed and displayed on the target object;

The second user equipment acquires the next operation request submitted by the user through the second user equipment and corresponding to the work instruction information in the first user equipment, wherein the first user equipment and the second user equipment Serving the same intelligent manufacturing task;

sending, by the second user equipment, the next step operation request to the corresponding network device;

The network device receives the next operation request, determines the next operation instruction information corresponding to the operation instruction information, and sends the next operation instruction information to the second user equipment.

The second user equipment receives the next step operation instruction information, and sends the next step operation instruction information to the first user equipment.

The first user equipment receives the next-step operation instruction information, and superimposes and displays the next-step operation instruction information on the target object.

According to an aspect of the present application, there is provided a first user equipment for intelligent manufacturing based on augmented reality, wherein the equipment includes:

a first position acquisition module, configured to acquire the first position information of the target object in the operation area in the first coordinate system corresponding to the operation area;

A second position acquisition module, configured to determine the second position of the target object in the second coordinate system according to the coordinate mapping relationship between the second coordinate system corresponding to the first user equipment and the first coordinate system information;

a guidance information acquisition module, used for acquiring the operation guidance information corresponding to the target object;

The instruction information superimposing module is configured to superimpose and display the operation instruction information corresponding to the target object on the target object according to the second position information.

According to another aspect of the present application, an augmented reality-based intelligent manufacturing network device is provided, wherein the device includes:

The next step instruction sending module is configured to send the next step operation instruction information corresponding to the operation instruction information to the corresponding first user equipment.

According to yet another aspect of the present application, a variety of augmented reality-based smart manufacturing second user equipment is provided, wherein the device includes:

The next step request acquisition module is configured to acquire the next step operation request submitted by the user through the second user equipment and corresponding to the work instruction information in the corresponding first user equipment, wherein the first user equipment and the first user equipment Two user equipments serve the same intelligent manufacturing task;

The next step request sending module is used to send the next step operation request to the corresponding network device;

a next-step guidance receiving module, configured to receive the next-step operation guidance information corresponding to the operation guidance information sent by the network device based on the next-step operation request;

The next step instruction sending module is configured to send the next step operation instruction information to the first user equipment corresponding to the operation instruction information.

According to an aspect of the present application, an augmented reality-based intelligent manufacturing system is provided, wherein the system includes the above-mentioned first user equipment and the above-mentioned network equipment.

According to another aspect of the present application, an augmented reality-based intelligent manufacturing system is provided, wherein the system includes the above-mentioned first user equipment, the above-mentioned network equipment, and the above-mentioned second user equipment.

According to one aspect of the present application, an augmented reality-based smart manufacturing device is provided, wherein the device includes:

processor; and

memory arranged to store computer-executable instructions which, when executed, cause the processor to perform:

obtaining the first position information of the target object in the operation area in the first coordinate system corresponding to the operation area;

determining the second position information of the target object in the second coordinate system according to the coordinate mapping relationship between the second coordinate system corresponding to the first user equipment and the first coordinate system;

obtaining the operation instruction information corresponding to the target object;

The operation instruction information corresponding to the target object is superimposed and displayed on the target object according to the second position information.

According to another aspect of the present application, an augmented reality-based smart manufacturing device is provided, wherein the device includes:

processor; and

memory arranged to store computer-executable instructions which, when executed, cause the processor to perform:

The next step operation instruction information corresponding to the operation instruction information is sent to the corresponding first user equipment.

According to yet another aspect of the present application, an augmented reality-based smart manufacturing device is provided, wherein the device includes:

processor; and

memory arranged to store computer-executable instructions which, when executed, cause the processor to perform:

Obtain the next operation request submitted by the user through the second user equipment and corresponding to the work instruction information in the corresponding first user equipment, wherein the first user equipment and the second user equipment serve the same intelligent manufacturing Task;

sending the next step operation request to the corresponding network device;

receiving the next-step operation instruction information corresponding to the operation instruction information sent by the network device based on the next-step operation request;

Sending the next step operation instruction information to the first user equipment corresponding to the operation instruction information.

According to one aspect of the present application, there is provided a computer-readable medium comprising instructions that, when executed, cause a system to:

obtaining the first position information of the target object in the operation area in the first coordinate system corresponding to the operation area;

determining the second position information of the target object in the second coordinate system according to the coordinate mapping relationship between the second coordinate system corresponding to the first user equipment and the first coordinate system;

obtaining the operation instruction information corresponding to the target object;

The operation instruction information corresponding to the target object is superimposed and displayed on the target object according to the second position information.

According to another aspect of the present application, there is provided a computer-readable medium comprising instructions that, when executed, cause a system to:

The next step operation instruction information corresponding to the operation instruction information is sent to the corresponding first user equipment.

According to yet another aspect of the present application, there is provided a computer-readable medium comprising instructions that, when executed, cause a system to:

Obtain the next operation request submitted by the user through the second user equipment and corresponding to the work instruction information in the corresponding first user equipment, wherein the first user equipment and the second user equipment serve the same intelligent manufacturing Task;

sending the next step operation request to the corresponding network device;

receiving the next-step operation instruction information corresponding to the operation instruction information sent by the network device based on the next-step operation request;

Sending the next step operation instruction information to the first user equipment corresponding to the operation instruction information.

Compared with the prior art, the present application guides the worker to complete the assembly of the workpiece by presenting some auxiliary information related to the assembly of the workpiece in the current operating area to the user holding the augmented reality device, such as the installation position of the workpiece, the installation sequence, etc., effectively. It can greatly reduce the cost of assembly and maintenance, reduce the probability of errors during the installation process, improve production efficiency, and achieve rapid response and precise intervention in the production process.

Description of drawings

Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

FIG. 1 shows a system topology diagram of an augmented reality-based smart manufacturing method according to an embodiment of the present application;

FIG. 2 shows a method flowchart of an augmented reality-based smart manufacturing method at the first user equipment end according to an embodiment of the present application;

Figure 3 shows the basic imaging model or basic pinhole model of the camera;

Fig. 4 shows a three-dimensional point conversion relationship between the world coordinate system and the camera screen coordinate system;

FIG. 5 shows a method flowchart of an augmented reality-based smart manufacturing method at a network device end according to another embodiment of the present application;

FIG. 6 shows a method flowchart of an augmented reality-based smart manufacturing method at a second user equipment end according to yet another embodiment of the present application;

7 shows a method flow chart of an augmented reality-based smart manufacturing method according to an aspect of the present application;

8 shows a method flow chart of an augmented reality-based smart manufacturing method according to yet another aspect of the present application;

FIG. 9 shows a device structure diagram of an augmented reality-based smart manufacturing method at a first user equipment end according to an embodiment of the present application;

10 shows a device structure diagram of an augmented reality-based smart manufacturing method at a network device end according to another embodiment of the present application;

FIG. 11 shows a device structure diagram of an augmented reality-based smart manufacturing method according to an aspect of the present application;

12 shows a system example diagram of an augmented reality-based smart manufacturing method according to an aspect of the present application;

13 shows a system example diagram of an augmented reality-based smart manufacturing method according to yet another aspect of the present application;

14 illustrates an exemplary system that may be used to implement various embodiments described in this application.

The same or similar reference numbers in the drawings represent the same or similar parts.

Detailed ways

The present application will be described in further detail below with reference to the accompanying drawings.

In a typical configuration of the present application, the terminal, the device serving the network, and the trusted party each include one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

Memory may include non-persistent memory in computer readable media, random access memory (RAM) and/or non-volatile memory in the form of, for example, read only memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

Computer-readable media includes both persistent and non-permanent, removable and non-removable media, and storage of information may be implemented by any method or technology. Information may be computer readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Flash Memory or other memory technology, Compact Disc Read Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic tape cartridges, magnetic tape disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

The equipment referred to in this application includes, but is not limited to, user equipment, network equipment, or equipment formed by integrating user equipment and network equipment through a network. The user equipment includes, but is not limited to, any mobile electronic product that can perform human-computer interaction with the user (for example, human-computer interaction through a touchpad), such as a smart phone, a tablet computer, etc., and the mobile electronic product can use any operation. system, such as android operating system, iOS operating system, etc. Wherein, the network device includes an electronic device that can automatically perform numerical calculation and information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), programmable logic Device (PLD), Field Programmable Gate Array (FPGA), Digital Signal Processor (DSP), Embedded Devices, etc. The network device includes, but is not limited to, a computer, a network host, a single network server, multiple network server sets, or a cloud composed of multiple servers; here, a cloud is composed of a large number of computers or network servers based on cloud computing, Among them, cloud computing is a kind of distributed computing, a virtual supercomputer composed of a group of loosely coupled computer sets. The network includes, but is not limited to, the Internet, a wide area network, a metropolitan area network, a local area network, a VPN network, a wireless ad hoc network (Ad Hoc network), and the like. Preferably, the device may also be a program running on the user equipment, network equipment, or a device formed by user equipment and network equipment, network equipment, touch terminal or network equipment and touch terminal integrated through a network.

Of course, those skilled in the art should understand that the above-mentioned devices are only examples, and other existing or possible devices that may appear in the future, if applicable to this application, should also be included within the protection scope of this application, and are included in this application by reference. this.

In the description of this application, "plurality" means two or more, unless expressly and specifically defined otherwise.

FIG. 1 shows a typical scenario of the present application. The user holds a first user equipment (such as an augmented reality device, etc.), and the first user equipment superimposes and displays the operation instruction information on the corresponding position of the target object in the operation area to assist the user. Complete the assembly of the target object, etc.; wherein, the solution can be completed by the first user equipment independently, or can be completed by the first user equipment and network equipment (such as MES production execution system) in cooperation, or the first user equipment, the second The user equipment (such as human-computer interaction equipment) and network equipment are coordinated to complete; in addition, the first user equipment can also send the current work instruction information to a third user equipment (such as a PC device) for the third user equipment to be superimposed and displayed on the target object corresponding location. The augmented reality devices include, but are not limited to, augmented reality glasses, augmented reality helmets and other augmented reality devices. Herein, the following embodiments are described by taking the augmented reality glasses as an example. Those skilled in the art should understand that these embodiments are also applicable to augmented reality. Other existing or future augmented reality devices such as helmets; network devices include but are not limited to cloud servers and MES (Manufacturing execution system) production execution systems. The MES production execution system is used as an example to illustrate the following embodiments. Those skilled in the art It should be understood that these embodiments are also applicable to other existing or future network devices such as cloud servers; third user devices include but are not limited to augmented reality devices, tablet computers, and , mobile devices, PC devices and other devices, the following embodiments are described here by taking PC devices as an example, and those skilled in the art should understand that these embodiments are also applicable to other existing or existing augmented reality devices, tablet computers, mobile devices, etc. A third user device that may appear in the future.

FIG. 2 shows an augmented reality-based smart manufacturing method at the first user equipment end according to an aspect of the present application, wherein the method includes step S11 , step S12 , step S13 and step S14 . In step S11, the first user equipment obtains the first position information of the target object in the operation area in the first coordinate system corresponding to the operation area; in step S12, the first user equipment corresponds to the first user equipment according to the first user equipment. The coordinate mapping relationship between the second coordinate system and the first coordinate system, to determine the second position information of the target object in the second coordinate system; in step S13, the first user equipment obtains the target object corresponding work instruction information; in step S14, the first user equipment superimposes and displays the work instruction information corresponding to the target object on the target object according to the second position information.

Specifically, in step S11, the first user equipment acquires first position information of the target object in the operation area in the first coordinate system corresponding to the operation area. Wherein, the operation area includes, but is not limited to, a workbench for operation, an operation area on an assembly line, a plane area that can be used for operation, and the like. For example, a two-dimensional identification map that is horizontal to the desktop is placed in the operation area, the first user equipment scans the identification map, and establishes a world coordinate system with the center of the identification map as the origin, and takes the center of the identification map as the whole world The center of the coordinate system, and the scale of the world coordinate system is established by using the identification map. The first user equipment uses the world coordinate system as the first coordinate system, and uses the computer vision algorithm to calculate the first coordinate system of the target object in the first coordinate system. Location information (such as coordinate location information, etc.).

Of course, those skilled in the art should understand that the above operating regions are only examples, and other existing or possible operating regions that may appear in the future, if applicable to the present application, should also be included within the protection scope of the present application, and are hereby referred to by way of reference Included here.

In step S12, the first user equipment determines the second coordinate of the target object in the second coordinate system according to the coordinate mapping relationship between the second coordinate system corresponding to the first user equipment and the first coordinate system location information. For example, we can assume that the target object has a first coordinate system (assumed to be the world coordinate system), the glasses entity camera has a three-dimensional camera coordinate system, the human eye and the glasses screen form a virtual camera coordinate system, and the glasses screen has a first coordinate system. A two-dimensional coordinate system (such as a two-dimensional coordinate system). The coordinates of the target object itself in the world coordinate system are known. First, it is transformed into the three-dimensional camera coordinate system of the glasses entity camera, and the transformation matrix is obtained through the identification and tracking algorithm. Then convert the external parameters (known) of the calibration parameters to the virtual camera coordinate system composed of the human eye and the glasses screen, and finally obtain the coordinates of the second coordinate system on the glasses screen according to the virtual camera internal parameters (known). Therefore, the final second position information of the target object in the second coordinate system can be calculated.

In step S13, the first user equipment acquires operation instruction information corresponding to the target object. The work instruction information includes information such as the assembly position and assembly sequence of the target object. For example, the first user equipment obtains the operation instruction information corresponding to the target object by identifying the target object and matching it in a local or cloud database; or the first user equipment reads the operation corresponding to the target object according to the user's operation, etc. Guidance information; for another example, the first user equipment captures an image corresponding to the target object, sends the image to the network device, and receives operation guidance information corresponding to the target object identified by the network device based on the image. There is no necessary sequence relationship between step S11 and step S13 here.

In step S14, the first user equipment superimposes and displays the operation instruction information corresponding to the target object on the target object according to the second position information. For example, according to the position information of the target object in the second coordinate system, and according to the positional relationship between the virtual superimposed information (such as information such as parts assembled in the target object) and the target object superimposed and displayed in the operation instruction information, the first user equipment can The corresponding position superimposes and displays the virtual superimposed information in the operation instruction information.

For example, the first user holds augmented reality glasses, and the augmented reality glasses capture image information related to the current operating area through a camera. The augmented reality glasses establish a world coordinate system according to the corresponding two-dimensional identification map in the operation area, and calculate the position of the target object in the world coordinate system. We can assume that the physical camera of the glasses has a three-dimensional camera coordinate system, the human eye and the glasses screen form a virtual camera coordinate system, and the glasses screen has a two-dimensional coordinate system. The coordinates of the target object (such as the outer box of the electric control box and the circuit board) are known in the world coordinate system. First, it is converted into the three-dimensional camera coordinate system of the glasses entity camera. The transformation matrix is obtained through the identification and tracking algorithm. out. Then convert the external parameters (known) of the calibration parameters to the virtual camera coordinate system composed of the human eye and the glasses screen, and finally obtain the coordinates of the second coordinate system on the glasses screen according to the virtual camera internal parameters (known). Therefore, the final second position information of the target object in the glasses screen can be calculated. The augmented reality glasses query the database for operation instruction information related to the object according to the information related to the target object, or the augmented reality glasses read the operation instruction information of the target object based on the selection operation of the first user, etc., wherein the operation instruction information includes However, it is not limited to the installation position and installation sequence of the circuit board in the outer box of the electric control box. According to the real-time position of the target object on the glasses screen, the augmented reality glasses superimpose and display the corresponding operation instruction information of the target object in real time at the position, such as the virtual circuit board in the recommended installation position of the circuit board in the outer box of the electric control box. Wait.

Of course, those skilled in the art should understand that the above-mentioned work instruction information is only an example. If other existing or future work instruction information is applicable to this application, it should also be included in the protection scope of this application, and is hereby referred to as References are included here.

In some embodiments, the method further includes step S15 (not shown). In step S15, the first user equipment sends the work instruction information to the corresponding third user equipment, so that the third user equipment superimposes and displays the work instruction information on the image about the target object. For example, the third user equipment includes a third camera, and the image information related to the operation area is captured by the third camera. The first user equipment sends the operation instruction information to the third camera device, and the third equipment device superimposes the received operation instruction information on the currently presented image of the target object in real time.

For example, the third user holds a PC device, the PC device is installed with a relatively fixed camera, and the PC device captures image information about the outer box of the electric control box in the operation area through the camera. After the augmented reality glasses acquire the work instruction information about the outer box of the electric control box, the work instruction information is sent to the PC device. The PC device receives the work instruction information, and superimposes and displays the virtual circuit board and other work instruction information at the corresponding position in the currently presented image information about the outer box of the electric control box through the computer vision algorithm. Here, we assume that the outer box of the electric control box in the operation area has a world coordinate system, the camera of the PC device has a three-dimensional camera coordinate system, the screen of the PC device has a two-dimensional image coordinate system, and the outer box of the electric control box is in The coordinates in the world coordinate system are known. We first convert the coordinate information of the outer box of the electric control box in the world coordinate system to obtain the outer box of the electric control box in the camera coordinate system according to the coordinate transformation between the camera coordinate system and the world coordinate system. Then, according to the coordinate change of the camera coordinate system and the image coordinate system, the position information of the outer box of the electric control box on the screen is determined; The corresponding position inside the box is superimposed to display the installation position and installation sequence of parts such as circuit boards. The installation sequence can be distinguished by different colors or different shades of the same color.

In some embodiments, the method further includes method S16 (not shown). In step S16, the first user equipment captures the image information of the target object in the operation area through the photographing device of the first user equipment; wherein, in step S11, the first user equipment obtains the image information through the image information The first position information of the target object in the first coordinate system corresponding to the operation area. In some embodiments, in step S13, the first user equipment acquires the work instruction information corresponding to the target object according to the image information. For example, the first user equipment captures the image information of the target object in the operation area through the camera, establishes a first coordinate system through the image information, and calculates the first position information of the target object in the first coordinate system; the first user equipment according to The image information is used for image recognition of the target object, and the computer vision algorithm is used to match the model in the pre-stored database, and the operation instruction information corresponding to the matched model is used as the operation instruction information corresponding to the target object.

For example, the augmented reality glasses use the camera to capture image information about the outer box of the electric control box on the current console, and the image information also includes the plane where the console is located. There is a two-dimensional identification map placed by the first user in the plane, and the augmented reality glasses establish a three-dimensional world coordinate system with the center of the two-dimensional identification map as the origin according to the captured image information and the two-dimensional identification map in the image information, and according to The distance between the outer box of the electric control box and the center of the two-dimensional identification map, and the current position information of the outer box of the electric control box in the world coordinate system is calculated. The augmented reality glasses perform matching in the database according to the image information of the outer box of the electric control box, and determine the outer box model of the electric control box that matches the image information, and the operation instruction information corresponding to the model.

In some embodiments, the method further includes step S17 (not shown) and step S18 (not shown). In step S17, the first user equipment acquires the next work instruction information corresponding to the work instruction information; in step S18, the first user equipment superimposes and displays the next work instruction information on the target object. For example, after the first user completes the work related to the current work instruction information, the first user equipment obtains the next work instruction information corresponding to the work instruction information. Step-by-step work instructions. The first user equipment superimposes and displays the next step operation instruction information on the target object at the position according to the real-time position of the target object on the device screen.

For example, the first user installs the electric control box on the console, and the first user has installed the circuit board in the corresponding position according to the presentation position of the virtual circuit board in the operation instruction information. According to the operation, the reality glasses read the next step operation instruction information corresponding to the target object in the database, such as fixing the corresponding screws in the corresponding positions of the circuit board.

Of course, those skilled in the art should understand that the above-mentioned next-step operation instruction information is only an example, and other existing or possible future next-step operation instruction information, if applicable to this application, should also be included within the protection scope of this application, and is hereby incorporated by reference.

In some embodiments, in step S17, the first user equipment receives the next work instruction information corresponding to the work instruction information sent by the corresponding network device. In some embodiments, the next step operation instruction information is sent by the network device based on the received next step operation request. In some embodiments, the next step operation instruction information is sent by the network device based on the received next step operation request. For example, after the user completes the current work instruction information, when the first user equipment determines that the current work has been completed based on the user's operation or by identifying the current target object, the first user equipment sends a next operation request to the network device. The network device receives the next step operation request, determines corresponding next step operation instruction information in the database based on the next step operation request, and sends the next step operation instruction information to the first user equipment.

For example, the first user installs the electric control box on the console, and the first user has installed the circuit board in the corresponding position according to the presentation position of the virtual circuit board in the operation instruction information. According to the operation, the reality glasses send the next operation request to the MES production execution system, wherein the next operation request includes the relevant information of the current work instruction information (such as the current work instruction information or the corresponding image information when the work instruction is completed, etc.). The MES production execution system receives the next step operation request, and determines the next step operation instruction information corresponding to the target object in the database according to the next step operation request. For example, the next step is determined according to the image information of the circuit board installed in the outer box of the electric control box. Work instruction information, such as fixing the corresponding screws in the corresponding positions of the circuit board, etc. The MES production execution system sends the next step operation instruction information to the augmented reality glasses.

Of course, those skilled in the art should understand that the above-mentioned next-step operation requests are only examples, and other existing or future next-step operation requests, if applicable to this application, should also be included in the protection scope of this application, and in the This is incorporated herein by reference.

In some embodiments, in step S17, the first user equipment receives the next work instruction information sent by the corresponding second user equipment and corresponding to the work instruction information, wherein the next work instruction information is sent by the The network device sends to the second user equipment. For example, the second user equipment includes a human-computer interaction device that establishes a communication connection with the first user equipment, the second user equipment may further establish a communication connection with the network device, and the second user equipment may present the target object current about the operation of the first user Based on the work instruction information, the second user can select to enter the next step operation information on the second user equipment based on the work instruction information. For another example, the second user equipment receives the next step guidance information of the first user equipment, and sends the next step operation guidance information to the first user equipment, where the next step guidance information includes the network equipment according to the information sent by the second user equipment. Returned by the next operation request.

For example, the second user holds a human-computer interaction device (such as a tablet computer, etc.), the human-computer interaction device establishes a communication connection with augmented reality glasses and a network device, and the first user and the second user are performing the same production execution task. The human-computer interaction device and the augmented reality glasses present the same work instruction information. For example, the human-computer interaction device presents the work instruction information on the screen, and the augmented reality superimposes the corresponding work instruction information on the screen. The first user installs the electric control box on the console, and the first user has installed the circuit board in the corresponding position according to the presentation position of the virtual circuit board in the work instruction information, and the first user clicks to proceed to the next step, augmented reality glasses According to the operation, a next operation request is sent to the human-computer interaction device, wherein the next operation request includes relevant information of the current work instruction information (such as current work instruction information or image information corresponding to the completion of the work instruction, etc.). The human-computer interaction device receives the next step operation request, and sends the next step operation request to the MES production execution system, and the MES production execution system determines the next step operation instruction information corresponding to the target object in the database according to the next step operation request, For example, according to the image information that the circuit board has been installed in the outer box of the electric control box, the next step operation instruction information, such as fixing the corresponding screws in the corresponding position of the circuit board, is determined. The MES production execution system sends the next step operation guidance device to the human-computer interaction device, and forwards it to the augmented reality glasses via the human-computer interaction device.

For another example, the first user installs the electric control box on the console, the first user has installed the circuit board in the corresponding position according to the position where the virtual circuit board is presented in the work instruction information, and the second user selects on the screen of the human-computer interaction device. Carry out the next operation, and the human-computer interaction device sends the next operation request to the network device according to the operation, wherein the next operation request includes the relevant information of the current work instruction information (such as the current work instruction information or the corresponding information when the work instruction is completed). image information, etc.). The MES production execution system receives the next step operation request, and determines the next step operation instruction information corresponding to the target object in the database according to the next step operation request. For example, the next step is determined according to the image information of the circuit board installed in the outer box of the electric control box. Work instruction information, such as fixing the corresponding screws in the corresponding positions of the circuit board, etc. The MES production execution system sends the next step operation guidance device to the human-computer interaction device, and forwards it to the augmented reality glasses via the human-computer interaction device.

In some embodiments, in step S17, the first user equipment acquires a next operation request regarding the work instruction information submitted by the user through the first user equipment, and sends the next operation request to the network A device that receives the next-step operation instruction information corresponding to the operation instruction information and sent by the network device based on the next-step operation request. For example, after the user completes the current job instruction information, the user selects an operation to enter the next job on the first user equipment, and the first user equipment sends a next operation request to the network device based on the user's operation. The network device receives the next step operation request, determines corresponding next step operation instruction information in the database based on the next step operation request, and sends the next step operation instruction information to the first user equipment.

For example, the first user installs the electric control box on the console, and the first user has installed the circuit board in the corresponding position according to the presentation position of the virtual circuit board in the operation instruction information. According to the operation, the reality glasses send the next operation request to the MES production execution system, wherein the next operation request includes the relevant information of the current work instruction information (such as the current work instruction information or the corresponding image information when the work instruction is completed, etc.). The MES production execution system receives the next step operation request, and determines the next step operation instruction information corresponding to the target object in the database according to the next step operation request. For example, the next step is determined according to the image information of the circuit board installed in the outer box of the electric control box. Work instruction information, such as fixing the corresponding screws in the corresponding positions of the circuit board, etc. The MES production execution system sends the next step operation instruction information to the augmented reality glasses.

In some embodiments, the method includes step S19 (not shown). In step S19, the first user equipment photographs the target object being operated by the photographing device in the first user equipment, and sends the photographed image information to the corresponding network equipment; wherein, in step S17, The first user equipment receives the next work instruction information corresponding to the work instruction information and sent by the network device after it is determined according to the image information that the work instruction information has been completed. For example, the first user device captures image information related to the target object when the current user is working in real time, and sends the image information to the network device. The network device receives the image information and matches the preset image of the job completion according to the computer vision algorithm. If the image related to the target object in the captured image information matches the preset image, the network device determines that the current job related to the target object has been completed. Determine the next work instruction information corresponding to the current work instruction information, and send the next work instruction information to the first user equipment.

For example, the first user installs the electric control box on the console, the first user has installed the circuit board in the corresponding position according to the presentation position of the virtual circuit board in the work instruction information, and the augmented reality glasses photograph that the circuit board has been installed on the electric circuit board. Control the image information of the corresponding position in the outer box of the box, and send the image information to the MES production execution system. The MES production execution system receives the image information, and matches the preset completed model in the execution task in the database, matches the preset model of the completed job instruction information, determines that the current job instruction information has been completed, and the MES produces According to the completion of the current work instruction, the execution system obtains the corresponding next work instruction information, such as fixing the corresponding screws in the corresponding position of the circuit board. Subsequently, the MES production execution system sends the next step operation instruction information to the augmented reality device.

In some embodiments, the method includes step S20 (not shown). In step S20, the first user equipment determines the current position information of the target object after the operation in the second coordinate system; wherein, in step S18, the first user equipment determines the current position information of the target object according to the current position information. The next operation instruction information is superimposed and displayed on the target object. For example, the first user equipment determines the current position information of the target object in the second coordinate system in real time according to the captured image information after the operation, and superimposes and displays the next operation guidance information on the target object at the position according to the position information. .

For example, the augmented reality glasses determine the position of the outer box of the electric control box in the world coordinate system in real time, and obtain the position of the outer box of the electric control box on the glasses screen according to the coordinate transformation relationship. The augmented reality glasses receive the instructions for the next step of fixing the screws at the corresponding positions of the circuit boards installed in the outer box of the electric control box, and superimpose the corresponding position of the circuit boards in the outer box of the electric control box on the current screen to display the corresponding screws to be fixed, etc. Information such as where each screw will be fastened, in what order, and what type.

In some embodiments, the method includes step S01 (not shown). In step S01 , the first user equipment photographs the target object being operated by the photographing device in the first user equipment, and determines whether the operation instruction information in the first user equipment has been completed according to the image information. For example, the first user equipment captures image information related to the target object when the current user is working in real time, and determines whether the current job has been completed according to the image information. The image related to the target object in the image information matches the preset image, and the first user equipment determines that the current operation related to the target object has been completed, and reads the next operation instruction information corresponding to the current operation instruction information in the database.

For example, the first user installs the electric control box on the console, the first user has installed the circuit board in the corresponding position according to the presentation position of the virtual circuit board in the work instruction information, and the augmented reality glasses photograph that the circuit board has been installed on the electric circuit board. The image information of the corresponding position in the outer box of the control box is matched with the preset completed model in the execution task in the database, and the preset model of the completed operation instruction information is matched to determine that the current operation instruction information has been completed. According to the completion of the current work instruction, the augmented reality glasses obtain the corresponding next work instruction information, such as fixing the corresponding screws in the corresponding positions of the circuit board.

In some embodiments, in step S12, the first user equipment is based on the first coordinate mapping relationship between the second coordinate system corresponding to the first user equipment and the human eye coordinate system, the human eye coordinate system and the camera coordinate system The second coordinate mapping relationship and the third coordinate mapping relationship between the camera coordinate system and the first coordinate system, determine the coordinate mapping relationship between the second coordinate system corresponding to the first user and the first coordinate system, And based on the coordinate mapping relationship between the second coordinate system and the first coordinate system, the second position information of the target object in the second coordinate system is determined.

For example, we can assume that the target object itself has a first coordinate system (assumed to be the world coordinate system), the augmented reality glasses entity camera has a three-dimensional camera coordinate system, the human eye and the glasses screen form a virtual camera coordinate system, the glasses screen There is a two-dimensional second coordinate system (image coordinate system). The coordinates of the target object itself in the world coordinate system are known. First, it is transformed into the three-dimensional camera coordinate system of the glasses entity camera, and the transformation matrix is obtained through the identification and tracking algorithm. Then, according to the external parameters (known) of the calibration parameters, it is converted to the virtual camera coordinate system composed of the human eye and the glasses screen, and finally the coordinates on the glasses screen are obtained according to the internal parameters of the virtual camera (known). Therefore, the final position of the target object in the glasses screen can be calculated. Here, we transform the coordinates of the target object in the world coordinate system to the physical camera coordinate system on the glasses (the coordinate system corresponding to the glasses camera), where the transformation matrix is obtained through the identification and tracking algorithm. After the augmented reality glasses determine the coordinate mapping relationship between the world coordinate system and the image coordinate system, the position information of the target object on the glasses screen is obtained according to the position of the target object in the world coordinate system. Among them, the specific camera model converted from world coordinates to image coordinates is as follows:

1. Ideal model

Figure 3 shows the basic imaging model of the camera, commonly referred to as the basic pinhole model, given by the central projection transformation of the three-dimensional space to the plane.

As shown in Figure 3, O _c is the center of the camera, and its distance from the camera image plane π is f, and f is called the focal length of the camera. The projection (or image) m of the spatial point X _c on the plane π is the intersection of the ray with the point O _c as the endpoint and passing through the point X _c and the plane π, and the ray with the point O _c as the endpoint and perpendicular to the image plane is called The optical axis or principal axis, the intersection p of the principal axis and the image plane is called the principal point of the camera. In order to describe this projection relationship algebraically, the camera coordinate system and the image plane coordinate system need to be established. On the image plane, the image coordinate system o-xy is established with the principal point p as the coordinate origin of the image plane coordinate system, and the horizontal line and the vertical line as the x-axis and y-axis respectively. In space, the camera coordinate system Ox _c y _c z _c is established by taking the camera center O _c as the coordinate origin of the camera coordinate system, as shown in Figure 3(b). The homogeneous coordinates of the spatial point X _c in the camera coordinate system are marked as X _c =(x _c , y _c , z _c , 1) ^T , and the homogeneous coordinates of its image point m in the image coordinate system are marked m = ( x,y,1) ^T , call (X _c ,m) a set of pairs. According to the triangle similarity principle, it can be inferred that the space point X _c and its image point m satisfy the following relationship:

Convert to matrix form:

Among them, X _c =(x _c , y _c , z _c , 1) ^T , m = (x, y, 1) ^T are the homogeneous coordinates of the space point and the image point, respectively, and are a homogeneous coordinate from space to the image plane. Sublinear transformation. Denote P=diag(f,f,1)(I,0), then the above formula can be expressed in a simpler form:

m=PX _c (3)

Note: (3) is a homogeneous equation, meaning equal in the sense of differing by a non-zero constant factor. The matrix P is usually called the camera matrix. This is the algebraic representation of the basic imaging model.

2. Actual model

The above discussion is the theoretical situation (the principal point is the origin of the image coordinate system), but in practice,

1. The coordinate origin of the image plane may not be on the principal point;

Second, the images used for computer processing are usually digital images obtained by a CCD camera, and the points on the image plane are digitally discretized.

In the ideal model derived above, it is assumed that the scales of the image coordinates on the two axes are not equal, so that the pixels after digital discretization of the CCD camera are not a square, so it is necessary to introduce a non-equivalent scale factor; third, the general camera There are distortion parameters.

Under the constraints of the above three conditions, the ideal central projection model can be rewritten as a model with five parameters:

Similarly, the projection relationship of the camera can be written as:

m=K(I,0)X _c =PX _c (5)

是摄像机内参数矩阵，f_x,f_y称为CCD摄像机在u轴和v轴方向上的尺度因子，(u₀,v₀)^T称为CCD摄像机的主点。s称为CCD摄像机的畸变因子或倾斜因子。摄像机的内部参数共有五个。in:

is the parameter matrix of the camera, f _x , f _y are called the scale factors of the CCD camera in the u-axis and v-axis directions, and (u ₀ , v ₀ ) ^T is called the principal point of the CCD camera. s is called the distortion factor or tilt factor of the CCD camera. There are five internal parameters of the camera.

3. General model

We generally describe a three-dimensional point. Since the camera may be moving all the time, we do not describe it based on the camera coordinate system, but in the world coordinate system. The relationship between the world coordinate system and the camera coordinate system can be described by a rotation matrix and a translation vector, as shown in Figure 4.

Let the coordinates of the space point in the world coordinate system and the camera coordinate system be X=(x,y,z,1) ^T , X _c =(x _c ,y _c ,z _c ,1) ^T , then the difference between the two Relationship:

Bringing (6) into (5), we get:

表示摄像机中心在世界坐标系中的坐标，摄像机矩阵

为摄像机的外参数矩阵。其中，R＝(α,β,γ)是旋转矩阵，α,β,γ分别是绕摄像机坐标系x,y,z轴的旋转角度。in,

Represents the coordinates of the camera center in the world coordinate system, the camera matrix

is the extrinsic parameter matrix of the camera. Among them, R=(α, β, γ) is the rotation matrix, and α, β, γ are the rotation angles around the camera coordinate system x, y, and z axes, respectively.

is the translation matrix, T _x , T _y , T _z are the translations around the camera coordinate system x, y, z axes, respectively, so the camera external parameter consists of 6 parameters (α, β, γ, T _x , T _y , T _z ).

Fig. 5 shows an augmented reality-based smart manufacturing method at the network device side according to another aspect of the present application, wherein the method includes step S21. In step S21, the network device sends the next step operation instruction information corresponding to the operation instruction information to the corresponding first user equipment.

For example, the first user holds augmented reality glasses, the augmented reality glasses use a camera to capture image information related to the current operating area, and the augmented reality glasses establish a communication connection with the network device. The network device determines the current work instruction information of the current first user based on the communication, determines the corresponding next work instruction information in the database, and sends the next work instruction information to the first user equipment. The first user equipment receives the next operation instruction information, and superimposes the next operation instruction information on the target object in real time at the position according to the position of the target object on the current screen.

In some embodiments, in step S21, the network device receives a next step operation request sent by the corresponding second user equipment and corresponding to the work instruction information in the corresponding first user equipment, wherein the first user equipment and the The second user equipment serves the same intelligent manufacturing task, determines the next work instruction information corresponding to the work instruction information, and sends the next work instruction information to the second user equipment. For example, the second user equipment includes a human-computer interaction device that establishes a communication connection with the first user equipment, and the second user equipment may also establish a communication connection with the network device. The second user equipment obtains the next operation request corresponding to the job instruction information of the current first user equipment. For example, the second user obtains the next operation request corresponding to the current job information of the target object based on the operation of the second user. The user equipment receives the next operation request sent by the first user equipment; the second user equipment sends the next operation request to the network device. The network device determines corresponding next-step operation instruction information according to the received next-step operation request, and sends the next-step operation instruction information to the second user equipment.

For example, the second user holds a human-computer interaction device (such as a tablet computer, etc.), the human-computer interaction device establishes a communication connection with augmented reality glasses and a network device, and the first user and the second user are performing unified production execution tasks. The human-computer interaction device and the augmented reality glasses present the same work instruction information. For example, the human-computer interaction device presents the work instruction information on the screen, and the augmented reality superimposes the corresponding work instruction information on the screen. The first user installs the electric control box on the console, and the first user has installed the circuit board in the corresponding position according to the presentation position of the virtual circuit board in the work instruction information, and the first user clicks to proceed to the next step, augmented reality glasses According to the operation, a next operation request is sent to the human-computer interaction device, wherein the next operation request includes relevant information of the current work instruction information (such as current work instruction information or image information corresponding to the completion of the work instruction, etc.). The human-computer interaction device receives the next step operation request, and sends the next step operation request to the MES production execution system, and the MES production execution system determines the next step operation instruction information corresponding to the target object in the database according to the next step operation request, For example, according to the image information that the circuit board has been installed in the outer box of the electric control box, the next step operation instruction information, such as fixing the corresponding screws in the corresponding position of the circuit board, is determined. The MES production execution system sends the next step operation guidance device to the human-computer interaction device, and forwards it to the augmented reality glasses via the human-computer interaction device.

For another example, the first user installs the electric control box on the console, the first user has installed the circuit board in the corresponding position according to the position where the virtual circuit board is presented in the work instruction information, and the second user selects on the screen of the human-computer interaction device. Carry out the next operation, and the human-computer interaction device sends the next operation request to the network device according to the operation, wherein the next operation request includes the relevant information of the current work instruction information (such as the current work instruction information or the corresponding information when the work instruction is completed). image information, etc.). The MES production execution system receives the next step operation request, and determines the next step operation instruction information corresponding to the target object in the database according to the next step operation request. For example, the next step is determined according to the image information of the circuit board installed in the outer box of the electric control box. Work instruction information, such as fixing the corresponding screws in the corresponding positions of the circuit board, etc. The MES production execution system sends the next step operation guidance device to the human-computer interaction device, and forwards it to the augmented reality glasses via the human-computer interaction device.

In some embodiments, in step S21, the network device receives the next operation request on the job instruction information sent by the corresponding first user equipment, determines the next step operation instruction information corresponding to the job instruction information, and sends the next operation instruction information to the next step. One-step operation instruction information is sent to the second user equipment. For example, after the user completes the current work instruction information, the user selects an operation to enter the next work on the first user device, or the first user determines that the current work instruction has been completed through the captured image of the work instruction. The operation sends the next operation request to the network device. The network device receives the next step operation request, determines corresponding next step operation instruction information in the database based on the next step operation request, and sends the next step operation instruction information to the second user equipment.

For example, the first user installs the electric control box on the console, and the first user has installed the circuit board in the corresponding position according to the presentation position of the virtual circuit board in the operation instruction information. According to the operation, the reality glasses send the next operation request to the MES production execution system; for another example, the augmented reality glasses use the camera to capture the image information related to the operation instruction of the electric control box on the current console, and determine the current operation according to the operation instruction information. The instruction information has been completed, and the augmented reality glasses send the next operation request to the MES production execution system, wherein the next operation request includes the relevant information of the current operation instruction information (such as the current operation instruction information or the corresponding image information when the operation instruction is completed, etc. ). The MES production execution system receives the next step operation request, and determines the next step operation instruction information corresponding to the target object in the database according to the next step operation request. For example, the next step is determined according to the image information of the circuit board installed in the outer box of the electric control box. Work instruction information, such as fixing the corresponding screws in the corresponding positions of the circuit board, etc. The MES production execution system sends the next step operation instruction information to the augmented reality glasses.

In some embodiments, the method further includes step S22 (not shown). In step S22, the network device receives image information about the target object sent by the corresponding first user equipment, wherein the image information is obtained by photographing the target object being operated by the photographing device in the first user equipment, according to The image information determines whether the work instruction information in the first user equipment has been completed; wherein, in step S21, if the work instruction information has been completed, the network device sends the work instruction to the first user equipment The information corresponding to the next work instruction information. For example, the first user device captures image information related to the target object when the current user is working in real time, and sends the image information to the network device. The network device receives the image information and matches the preset image of the job completion according to the computer vision algorithm. If the image related to the target object in the captured image information matches the preset image, the network device determines that the current job related to the target object has been completed. Determine the next work instruction information corresponding to the current work instruction information, and send the next work instruction information to the first user equipment.

For example, the first user installs the electric control box on the console, the first user has installed the circuit board in the corresponding position according to the presentation position of the virtual circuit board in the work instruction information, and the augmented reality glasses photograph that the circuit board has been installed on the electric circuit board. Control the image information of the corresponding position in the outer box of the box, and send the image information to the MES production execution system. The MES production execution system receives the image information, and matches the preset completed model in the execution task in the database, matches the preset model of the completed job instruction information, determines that the current job instruction information has been completed, and the MES produces According to the completion of the current work instruction, the execution system obtains the corresponding next work instruction information, such as fixing the corresponding screws in the corresponding position of the circuit board. Subsequently, the MES production execution system sends the next step operation instruction information to the augmented reality device.

FIG. 6 shows an augmented reality-based smart manufacturing method at the second user equipment end according to yet another aspect of the present application, wherein the method includes step S31 , step S32 , step S33 and step S34 . In step S31, the second user equipment acquires the next operation request submitted by the user through the second user equipment and corresponding to the work instruction information in the corresponding first user equipment, wherein the first user equipment and the The second user equipment serves the same intelligent manufacturing task; in step S32, the second user equipment sends the next step operation request to the corresponding network device; in step S33, the second user equipment receives the the next step operation instruction information corresponding to the operation instruction information sent by the next operation request; in step S34, the second user equipment sends the next operation instruction information to the first user corresponding to the operation instruction information equipment. For example, the first user holds the first user equipment, the second user holds the computer interaction equipment (such as PC equipment, etc.), the first user equipment captures image information related to the target object in the current operation area through the camera, and the first user equipment A communication connection is established with the second user equipment and the network equipment. If the user's current work instruction has been completed, the second user equipment sends a next operation request to the network device based on the operation of the second user, and the network device receives the request, determines the next work instruction information corresponding to the request in the database, and sends the request to the network device. The next step operation instruction information is sent to the second user equipment. The second user receives the next step operation instruction information, and sends the next step operation instruction information to the first user equipment. The first user equipment receives the next operation instruction information, and superimposes the next operation instruction information on the target object in real time at the position according to the position of the target object on the current screen.

For example, the first user holds augmented reality glasses, the second user equipment holds a machine interaction device, the augmented reality device establishes a communication connection with the human-computer interaction device, and the human-machine interaction device establishes a communication connection with the MES production execution system. Human-computer interaction equipment and augmented reality glasses present the same operation guidance information, such as installation guidance information related to the outer box of the electric control box on the current console. The first user installs the electric control box on the console, the first user has installed the circuit board in the corresponding position according to the position of the virtual circuit board in the operation instruction information, and the second user selects the next step on the screen of the human-computer interaction device operation, the human-computer interaction device sends the next operation request to the network device according to the operation, wherein the next operation request includes the relevant information of the current work instruction information (such as the current work instruction information or the corresponding image information when the work instruction is completed). Wait). The MES production execution system receives the next step operation request, and determines the next step operation instruction information corresponding to the target object in the database according to the next step operation request. For example, the next step is determined according to the image information of the circuit board installed in the outer box of the electric control box. Work instruction information, such as fixing the corresponding screws in the corresponding positions of the circuit board, etc. The MES production execution system sends the next step operation guidance device to the human-computer interaction device, and forwards it to the augmented reality glasses via the human-computer interaction device. The augmented reality glasses receive the instructions for the next step of fixing the screws at the corresponding positions of the circuit boards installed in the outer box of the electric control box, and superimpose the corresponding position of the circuit boards in the outer box of the electric control box on the current screen to display the corresponding screws to be fixed, etc. Information such as where each screw will be fastened, in what order, and what type.

In some embodiments, in step S31, the second user equipment receives a next operation request sent by the first user equipment and corresponding to the work instruction information in the first user equipment, wherein the first user equipment and the The second user equipment serves the same intelligent manufacturing task.

For example, the first user installs the electric control box on the console, and the first user has installed the circuit board in the corresponding position according to the presentation position of the virtual circuit board in the operation instruction information. According to the operation, the reality glasses send a next operation request to the human-computer interaction device, wherein the next operation request includes the relevant information of the current work instruction information (such as the current work instruction information or the corresponding image information when the work instruction is completed, etc.). The human-computer interaction device receives the next step operation request, and the human-computer interaction device sends the next step operation request to the MES production execution system to determine the corresponding next step operation instruction information, and forwards the next step operation instruction information by the MES production execution system. to augmented reality glasses.

FIG. 7 shows an augmented reality-based smart manufacturing method according to an aspect of the present application, wherein the method includes:

The first user equipment obtains the first position information of the target object in the operation area in the first coordinate system corresponding to the operation area, according to the second coordinate system corresponding to the first user equipment and the coordinates of the first coordinate system mapping relationship, determine the second position information of the target object in the second coordinate system, obtain the work instruction information corresponding to the target object, and assign the work instruction corresponding to the target object according to the second position information The information is superimposed and displayed on the target object;

The network device sends the next step operation instruction information corresponding to the operation instruction information to the first user equipment;

The first user equipment receives the next-step operation instruction information, and superimposes and displays the next-step operation instruction information on the target object.

FIG. 8 shows an augmented reality-based smart manufacturing method according to another aspect of the present application, wherein the method includes:

The first user equipment obtains the first position information of the target object in the operation area in the first coordinate system corresponding to the operation area, according to the second coordinate system corresponding to the first user equipment and the coordinates of the first coordinate system mapping relationship, determine the second position information of the target object in the second coordinate system, obtain the work instruction information corresponding to the target object, and assign the work instruction corresponding to the target object according to the second position information The information is superimposed and displayed on the target object;

The second user equipment acquires the next operation request submitted by the user through the second user equipment and corresponding to the work instruction information in the first user equipment, wherein the first user equipment and the second user equipment Serving the same intelligent manufacturing task;

sending, by the second user equipment, the next step operation request to the corresponding network device;

The network device receives the next operation request, determines the next operation instruction information corresponding to the operation instruction information, and sends the next operation instruction information to the second user equipment.

The second user equipment receives the next step operation instruction information, and sends the next step operation instruction information to the first user equipment.

The first user equipment receives the next-step operation instruction information, and superimposes and displays the next-step operation instruction information on the target object.

FIG. 9 shows an augmented reality-based smart manufacturing first user equipment according to an aspect of the present application, wherein the device includes a first location acquisition module 11 , a second location acquisition module 12 , a guidance information acquisition module 13 and a guidance Information overlay module 14 . The first position obtaining module 11 is used to obtain the first position information of the target object in the operation area in the first coordinate system corresponding to the operation area; the second position obtaining module 12 is used to obtain the first position information according to the corresponding first user equipment The coordinate mapping relationship between the second coordinate system and the first coordinate system, to determine the second position information of the target object in the second coordinate system; the guidance information acquisition module 13 is used to obtain the corresponding target object. The instruction information superimposing module 14 is configured to superimpose and display the operation instruction information corresponding to the target object on the target object according to the second position information.

Specifically, the first position obtaining module 11 is configured to obtain the first position information of the target object in the operation area in the first coordinate system corresponding to the operation area. Wherein, the operation area includes, but is not limited to, a workbench for operation, an operation area on an assembly line, a plane area that can be used for operation, and the like. For example, a two-dimensional identification map that is horizontal to the desktop is placed in the operation area, the first user equipment scans the identification map, and establishes a world coordinate system with the center of the identification map as the origin, and takes the center of the identification map as the whole world The center of the coordinate system, and the scale of the world coordinate system is established by using the identification map. The first user equipment uses the world coordinate system as the first coordinate system, and uses the computer vision algorithm to calculate the first coordinate system of the target object in the first coordinate system. Location information (such as coordinate location information, etc.).

Of course, those skilled in the art should understand that the above operating regions are only examples, and other existing or possible operating regions that may appear in the future, if applicable to the present application, should also be included within the protection scope of the present application, and are hereby referred to by way of reference Included here.

The second position acquisition module 12 is configured to determine the second position of the target object in the second coordinate system according to the coordinate mapping relationship between the second coordinate system corresponding to the first user equipment and the first coordinate system location information. For example, we can assume that the target object has a first coordinate system (assumed to be the world coordinate system), the glasses entity camera has a three-dimensional camera coordinate system, the human eye and the glasses screen form a virtual camera coordinate system, and the glasses screen has a first coordinate system. A two-dimensional coordinate system (such as a two-dimensional coordinate system). The coordinates of the target object itself in the world coordinate system are known. First, it is transformed into the three-dimensional camera coordinate system of the glasses entity camera, and the transformation matrix is obtained through the identification and tracking algorithm. Then convert the external parameters (known) of the calibration parameters to the virtual camera coordinate system composed of the human eye and the glasses screen, and finally obtain the coordinates of the second coordinate system on the glasses screen according to the virtual camera internal parameters (known). Therefore, the final second position information of the target object in the second coordinate system can be calculated.

The guidance information acquisition module 13 is configured to acquire the operation guidance information corresponding to the target object. The work instruction information includes information such as the assembly position and assembly sequence of the target object. For example, the first user equipment obtains the operation instruction information corresponding to the target object by identifying the target object and matching it in a local or cloud database; or the first user equipment reads the operation corresponding to the target object according to the user's operation, etc. Guidance information; for another example, the first user equipment captures an image corresponding to the target object, sends the image to the network device, and receives operation guidance information corresponding to the target object identified by the network device based on the image.

The instruction information superimposing module 14 superimposes and displays the operation instruction information corresponding to the target object on the target object according to the second position information. For example, according to the position information of the target object in the second coordinate system, and according to the positional relationship between the virtual superimposed information (such as information such as parts assembled in the target object) and the target object superimposed and displayed in the operation instruction information, the first user equipment can The corresponding position superimposes and displays the virtual superimposed information in the operation instruction information.

For example, the first user holds augmented reality glasses, and the augmented reality glasses capture image information related to the current operating area through a camera. The augmented reality glasses establish a world coordinate system according to the corresponding two-dimensional identification map in the operation area, and calculate the position of the target object in the world coordinate system. We can assume that the physical camera of the glasses has a three-dimensional camera coordinate system, the human eye and the glasses screen form a virtual camera coordinate system, and the glasses screen has a two-dimensional coordinate system. The coordinates of the target object (such as the outer box of the electric control box and the circuit board) are known in the world coordinate system. First, it is converted into the three-dimensional camera coordinate system of the glasses entity camera. The transformation matrix is obtained through the identification and tracking algorithm. out. Then convert the external parameters (known) of the calibration parameters to the virtual camera coordinate system composed of the human eye and the glasses screen, and finally obtain the coordinates of the second coordinate system on the glasses screen according to the virtual camera internal parameters (known). Therefore, the final second position information of the target object in the glasses screen can be calculated. The augmented reality glasses query the database for operation instruction information related to the object according to the information related to the target object, or the augmented reality glasses read the operation instruction information of the target object based on the selection operation of the first user, etc., wherein the operation instruction information includes However, it is not limited to the installation position and installation sequence of the circuit board in the outer box of the electric control box. According to the real-time position of the target object on the glasses screen, the augmented reality glasses superimpose and display the corresponding operation instruction information of the target object in real time at the position, such as the virtual circuit board in the recommended installation position of the circuit board in the outer box of the electric control box. Wait.

Of course, those skilled in the art should understand that the above-mentioned work instruction information is only an example. If other existing or future work instruction information is applicable to this application, it should also be included in the protection scope of this application, and is hereby referred to as References are included here.

In some embodiments, the apparatus further includes a work instruction sending module 15 (not shown). The work instruction sending module 15 is configured to send the work instruction information to the corresponding third user equipment, so that the third user equipment can superimpose and display the work instruction information on the image about the target object. For example, the third user equipment includes a third camera, and the image information related to the operation area is captured by the third camera. The first user equipment sends the operation instruction information to the third camera device, and the third equipment device superimposes the received operation instruction information on the currently presented image of the target object in real time.

For example, the third user holds a PC device, the PC device is installed with a relatively fixed camera, and the PC device captures image information about the outer box of the electric control box in the operation area through the camera. After the augmented reality glasses acquire the work instruction information about the outer box of the electric control box, the work instruction information is sent to the PC device. The PC device receives the work instruction information, and superimposes and displays the virtual circuit board and other work instruction information at the corresponding position in the currently presented image information about the outer box of the electric control box through the computer vision algorithm. Here, we assume that the outer box of the electric control box in the operation area has a world coordinate system, the camera of the PC device has a three-dimensional camera coordinate system, the screen of the PC device has a two-dimensional image coordinate system, and the outer box of the electric control box is in The coordinates in the world coordinate system are known. We first convert the coordinate information of the outer box of the electric control box in the world coordinate system to obtain the outer box of the electric control box in the camera coordinate system according to the coordinate transformation between the camera coordinate system and the world coordinate system. Then, according to the coordinate change of the camera coordinate system and the image coordinate system, the position information of the outer box of the electric control box on the screen is determined; The corresponding position inside the box is superimposed to display the installation position and installation sequence of parts such as circuit boards. The installation sequence can be distinguished by different colors or different shades of the same color.

In some embodiments, the device also includes an image capture module 16 (not shown). The image capturing module 16 is configured to capture the image information of the target object in the operation area through the capturing device of the first user equipment; wherein, the first position obtaining module is configured to obtain the target through the image information The first position information of the object in the first coordinate system corresponding to the operation area. In some embodiments, the instruction information acquisition module 13 is configured to acquire operation instruction information corresponding to the target object according to the image information. For example, the first user equipment captures the image information of the target object in the operation area through the camera, establishes a first coordinate system through the image information, and calculates the first position information of the target object in the first coordinate system; the first user equipment according to The image information is used for image recognition of the target object, and the computer vision algorithm is used to match the model in the pre-stored database, and the operation instruction information corresponding to the matched model is used as the operation instruction information corresponding to the target object.

For example, the augmented reality glasses use the camera to capture image information about the outer box of the electric control box on the current console, and the image information also includes the plane where the console is located. There is a two-dimensional identification map placed by the first user in the plane, and the augmented reality glasses establish a three-dimensional world coordinate system with the center of the two-dimensional identification map as the origin according to the captured image information and the two-dimensional identification map in the image information, and according to The distance between the outer box of the electric control box and the center of the two-dimensional identification map, and the current position information of the outer box of the electric control box in the world coordinate system is calculated. The augmented reality glasses perform matching in the database according to the image information of the outer box of the electric control box, and determine the outer box model of the electric control box that matches the image information, and the operation instruction information corresponding to the model.

In some embodiments, the device further includes a next-step guidance acquisition module 17 (not shown) and a next-step guidance overlay module 18 (not shown). It includes a next-step guidance acquisition module 17 for obtaining the next-step operation guidance information corresponding to the operation guidance information; a next-step guidance overlay module 18 for superimposing and displaying the next-step operation guidance information on the target object. For example, after the first user completes the work related to the current work instruction information, the first user equipment obtains the next work instruction information corresponding to the work instruction information. Step-by-step work instructions. The first user equipment superimposes and displays the next step operation instruction information on the target object at the position according to the real-time position of the target object on the device screen.

For example, the first user installs the electric control box on the console, and the first user has installed the circuit board in the corresponding position according to the presentation position of the virtual circuit board in the operation instruction information. According to the operation, the reality glasses read the next step operation instruction information corresponding to the target object in the database, such as fixing the corresponding screws in the corresponding positions of the circuit board.

Of course, those skilled in the art should understand that the above-mentioned next-step operation instruction information is only an example, and other existing or possible future next-step operation instruction information, if applicable to this application, should also be included within the protection scope of this application, and is hereby incorporated by reference.

In some embodiments, the next step guide obtaining module 17 is configured to receive next step guide information sent by the corresponding network device and corresponding to the work guide information. In some embodiments, the next step operation instruction information is sent by the network device based on the received next step operation request. In some embodiments, the next step operation instruction information is sent by the network device based on the received next step operation request. For example, after the user completes the current work instruction information, when the first user equipment determines that the current work has been completed based on the user's operation or by identifying the current target object, the first user equipment sends a next operation request to the network device. The network device receives the next step operation request, determines corresponding next step operation instruction information in the database based on the next step operation request, and sends the next step operation instruction information to the first user equipment.

For example, the first user installs the electric control box on the console, and the first user has installed the circuit board in the corresponding position according to the presentation position of the virtual circuit board in the operation instruction information. According to the operation, the reality glasses send the next operation request to the MES production execution system, wherein the next operation request includes the relevant information of the current work instruction information (such as the current work instruction information or the corresponding image information when the work instruction is completed, etc.). The MES production execution system receives the next step operation request, and determines the next step operation instruction information corresponding to the target object in the database according to the next step operation request. For example, the next step is determined according to the image information of the circuit board installed in the outer box of the electric control box. Work instruction information, such as fixing the corresponding screws in the corresponding positions of the circuit board, etc. The MES production execution system sends the next step operation instruction information to the augmented reality glasses.

Of course, those skilled in the art should understand that the above-mentioned next-step operation requests are only examples, and other existing or future next-step operation requests, if applicable to this application, should also be included in the protection scope of this application, and in the This is incorporated herein by reference.

In some embodiments, the next step guide obtaining module 17 is configured to receive next step guide information sent by the corresponding second user equipment and corresponding to the work guide information, wherein the next step guide information is obtained by the The network device sends to the second user equipment. For example, the second user equipment includes a human-computer interaction device that establishes a communication connection with the first user equipment, the second user equipment may further establish a communication connection with the network device, and the second user equipment may present the target object current about the operation of the first user Based on the work instruction information, the second user can select to enter the next step operation information on the second user equipment based on the work instruction information. For another example, the second user equipment receives the next step guidance information of the first user equipment, and sends the next step operation guidance information to the first user equipment, where the next step guidance information includes the network equipment according to the information sent by the second user equipment. Returned by the next operation request.

For example, the second user holds a human-computer interaction device (such as a tablet computer, etc.), the human-computer interaction device establishes a communication connection with augmented reality glasses and a network device, and the first user and the second user are performing the same production execution task. The human-computer interaction device and the augmented reality glasses present the same work instruction information. For example, the human-computer interaction device presents the work instruction information on the screen, and the augmented reality superimposes the corresponding work instruction information on the screen. The first user installs the electric control box on the console, and the first user has installed the circuit board in the corresponding position according to the presentation position of the virtual circuit board in the work instruction information, and the first user clicks to proceed to the next step, augmented reality glasses According to the operation, a next operation request is sent to the human-computer interaction device, wherein the next operation request includes relevant information of the current work instruction information (such as current work instruction information or image information corresponding to the completion of the work instruction, etc.). The human-computer interaction device receives the next step operation request, and sends the next step operation request to the MES production execution system, and the MES production execution system determines the next step operation instruction information corresponding to the target object in the database according to the next step operation request, For example, according to the image information that the circuit board has been installed in the outer box of the electric control box, the next step operation instruction information, such as fixing the corresponding screws in the corresponding position of the circuit board, is determined. The MES production execution system sends the next step operation guidance device to the human-computer interaction device, and forwards it to the augmented reality glasses via the human-computer interaction device.

For another example, the first user installs the electric control box on the console, the first user has installed the circuit board in the corresponding position according to the position where the virtual circuit board is presented in the work instruction information, and the second user selects on the screen of the human-computer interaction device. Carry out the next operation, and the human-computer interaction device sends the next operation request to the network device according to the operation, wherein the next operation request includes the relevant information of the current work instruction information (such as the current work instruction information or the corresponding information when the work instruction is completed). image information, etc.). The MES production execution system receives the next step operation request, and determines the next step operation instruction information corresponding to the target object in the database according to the next step operation request. For example, the next step is determined according to the image information of the circuit board installed in the outer box of the electric control box. Work instruction information, such as fixing the corresponding screws in the corresponding positions of the circuit board, etc. The MES production execution system sends the next step operation guidance device to the human-computer interaction device, and forwards it to the augmented reality glasses via the human-computer interaction device.

In some embodiments, the next step guide obtaining module 17 is configured to obtain the next step operation request submitted by the user through the first user equipment regarding the work guide information, and send the next step operation request to the network A device that receives the next-step operation instruction information corresponding to the operation instruction information and sent by the network device based on the next-step operation request. For example, after the user completes the current job instruction information, the user selects an operation to enter the next job on the first user equipment, and the first user equipment sends a next operation request to the network device based on the user's operation. The network device receives the next step operation request, determines corresponding next step operation instruction information in the database based on the next step operation request, and sends the next step operation instruction information to the first user equipment.

For example, the first user installs the electric control box on the console, and the first user has installed the circuit board in the corresponding position according to the presentation position of the virtual circuit board in the operation instruction information. According to the operation, the reality glasses send the next operation request to the MES production execution system, wherein the next operation request includes the relevant information of the current work instruction information (such as the current work instruction information or the corresponding image information when the work instruction is completed, etc.). The MES production execution system receives the next step operation request, and determines the next step operation instruction information corresponding to the target object in the database according to the next step operation request. For example, the next step is determined according to the image information of the circuit board installed in the outer box of the electric control box. Work instruction information, such as fixing the corresponding screws in the corresponding positions of the circuit board, etc. The MES production execution system sends the next step operation instruction information to the augmented reality glasses.

In some embodiments, the apparatus includes a filming job module 19 (not shown). The photographing operation module 19 is used to photograph the target object being operated by the photographing device in the first user equipment, and send the photographed image information to the corresponding network equipment; wherein, the next step is to instruct the acquisition module 17, It is used for receiving the next work instruction information corresponding to the work instruction information and sent by the network device after it is determined according to the image information that the work instruction information has been completed. For example, the first user device captures image information related to the target object when the current user is working in real time, and sends the image information to the network device. The network device receives the image information and matches the preset image of the job completion according to the computer vision algorithm. If the image related to the target object in the captured image information matches the preset image, the network device determines that the current job related to the target object has been completed. Determine the next work instruction information corresponding to the current work instruction information, and send the next work instruction information to the first user equipment.

For example, the first user installs the electric control box on the console, the first user has installed the circuit board in the corresponding position according to the presentation position of the virtual circuit board in the work instruction information, and the augmented reality glasses photograph that the circuit board has been installed on the electric circuit board. Control the image information of the corresponding position in the outer box of the box, and send the image information to the MES production execution system. The MES production execution system receives the image information, and matches the preset completed model in the execution task in the database, matches the preset model of the completed job instruction information, determines that the current job instruction information has been completed, and the MES produces According to the completion of the current work instruction, the execution system obtains the corresponding next work instruction information, such as fixing the corresponding screws in the corresponding position of the circuit board. Subsequently, the MES production execution system sends the next step operation instruction information to the augmented reality device.

In some embodiments, the device includes a location determination module 20 (not shown). The position determination module 20 is used to determine the current position information of the target object after the operation in the second coordinate system; wherein, the next step guide superposition module 18 is used to determine the next step according to the current position information The operation instruction information is superimposed and displayed on the target object. For example, the first user equipment determines the current position information of the target object in the second coordinate system in real time according to the captured image information after the operation, and superimposes and displays the next operation guidance information on the target object at the position according to the position information. .

For example, the augmented reality glasses determine the position of the outer box of the electric control box in the world coordinate system in real time, and obtain the position of the outer box of the electric control box on the glasses screen according to the coordinate transformation relationship. The augmented reality glasses receive the instructions for the next step of fixing the screws at the corresponding positions of the circuit boards installed in the outer box of the electric control box, and superimpose the corresponding position of the circuit boards in the outer box of the electric control box on the current screen to display the corresponding screws to be fixed, etc. Information such as where each screw will be fastened, in what order, and what type.

In some embodiments, the apparatus includes a next step reading module 01 (not shown). The next step is to read the module 01, which is used for photographing the target object being operated by the photographing device in the first user equipment, and determining whether the operation instruction information in the first user equipment has been completed according to the image information. For example, the first user equipment captures image information related to the target object when the current user is working in real time, and determines whether the current job has been completed according to the image information. The image related to the target object in the image information matches the preset image, and the first user equipment determines that the current operation related to the target object has been completed, and reads the next operation instruction information corresponding to the current operation instruction information in the database.

For example, the first user installs the electric control box on the console, the first user has installed the circuit board in the corresponding position according to the presentation position of the virtual circuit board in the work instruction information, and the augmented reality glasses photograph that the circuit board has been installed on the electric circuit board. The image information of the corresponding position in the outer box of the control box is matched with the preset completed model in the execution task in the database, and the preset model of the completed operation instruction information is matched to determine that the current operation instruction information has been completed. According to the completion of the current work instruction, the augmented reality glasses obtain the corresponding next work instruction information, such as fixing the corresponding screws in the corresponding positions of the circuit board.

In some embodiments, the second position acquisition module 12 is configured to use the first coordinate mapping relationship between the second coordinate system corresponding to the first user equipment and the human eye coordinate system, the human eye coordinate system and the camera coordinate system The second coordinate mapping relationship and the third coordinate mapping relationship between the camera coordinate system and the first coordinate system, determine the coordinate mapping relationship between the second coordinate system corresponding to the first user and the first coordinate system, And based on the coordinate mapping relationship between the second coordinate system and the first coordinate system, the second position information of the target object in the second coordinate system is determined.

For example, we can assume that the target object itself has a first coordinate system (assumed to be the world coordinate system), the augmented reality glasses entity camera has a three-dimensional camera coordinate system, the human eye and the glasses screen form a virtual camera coordinate system, the glasses screen There is a two-dimensional second coordinate system (image coordinate system). The coordinates of the target object itself in the world coordinate system are known. First, it is transformed into the three-dimensional camera coordinate system of the glasses entity camera, and the transformation matrix is obtained through the identification and tracking algorithm. Then, according to the external parameters (known) of the calibration parameters, it is converted to the virtual camera coordinate system composed of the human eye and the glasses screen, and finally the coordinates on the glasses screen are obtained according to the internal parameters of the virtual camera (known). Therefore, the final position of the target object in the glasses screen can be calculated. Here, we transform the coordinates of the target object in the world coordinate system to the physical camera coordinate system on the glasses (the coordinate system corresponding to the glasses camera), where the transformation matrix is obtained through the identification and tracking algorithm. After the augmented reality glasses determine the coordinate mapping relationship between the world coordinate system and the image coordinate system, the position information of the target object on the glasses screen is obtained according to the position of the target object in the world coordinate system. Among them, the specific camera model converted from world coordinates to image coordinates is as follows:

1. Ideal model

Figure 3 shows the basic imaging model of the camera, commonly referred to as the basic pinhole model, given by the central projection transformation of the three-dimensional space to the plane.

As shown in Figure 3, O _c is the center of the camera, and its distance from the camera image plane π is f, and f is called the focal length of the camera. The projection (or image) m of the spatial point X _c on the plane π is the intersection of the ray with the point O _c as the endpoint and passing through the point X _c and the plane π, and the ray with the point O _c as the endpoint and perpendicular to the image plane is called The optical axis or principal axis, the intersection p of the principal axis and the image plane is called the principal point of the camera. In order to describe this projection relationship algebraically, the camera coordinate system and the image plane coordinate system need to be established. On the image plane, the image coordinate system o-xy is established with the principal point p as the coordinate origin of the image plane coordinate system, and the horizontal line and the vertical line as the x-axis and y-axis respectively. In space, the camera coordinate system Ox _c y _c z _c is established by taking the camera center O _c as the coordinate origin of the camera coordinate system, as shown in Figure 3(b). The homogeneous coordinates of the spatial point X _c in the camera coordinate system are marked as X _c =(x _c , y _c , z _c , 1) ^T , and the homogeneous coordinates of its image point m in the image coordinate system are marked m = ( x,y,1) ^T , call (X _c ,m) a set of pairs. According to the triangle similarity principle, it can be inferred that the space point X _c and its image point m satisfy the following relationship:

Convert to matrix form:

Among them, X _c =(x _c , y _c , z _c , 1) ^T , m = (x, y, 1) ^T are the homogeneous coordinates of the space point and the image point, respectively, and are a homogeneous coordinate from space to the image plane. Sublinear transformation. Denote P=diag(f,f,1)(I,0), then the above formula can be expressed in a simpler form:

m=PX _c (11)

Note: (11) is a homogeneous equation, meaning equal in the sense of differing by a non-zero constant factor. The matrix P is usually called the camera matrix. This is the algebraic representation of the basic imaging model.

2. Actual model

The above discussion is the theoretical situation (the principal point is the origin of the image coordinate system), but in practice,

1. The coordinate origin of the image plane may not be on the principal point;

Second, the images used for computer processing are usually digital images obtained by a CCD camera, and the points on the image plane are digitally discretized.

In the ideal model derived above, it is assumed that the scales of the image coordinates on the two axes are not equal, so that the pixels after digital discretization of the CCD camera are not a square, so it is necessary to introduce a non-equivalent scale factor; third, the general camera There are distortion parameters.

Under the constraints of the above three conditions, the ideal central projection model can be rewritten as a model with five parameters:

Similarly, the projection relationship of the camera can be written as:

m=K(I,0)X _c =PX _c (13)

是摄像机内参数矩阵，f_x,f_y称为CCD摄像机在u轴和v轴方向上的尺度因子，(u₀,v₀)^T称为CCD摄像机的主点。s称为CCD摄像机的畸变因子或倾斜因子。摄像机的内部参数共有五个。in:

is the parameter matrix of the camera, f _x , f _y are called the scale factors of the CCD camera in the u-axis and v-axis directions, and (u ₀ , v ₀ ) ^T is called the principal point of the CCD camera. s is called the distortion factor or tilt factor of the CCD camera. There are five internal parameters of the camera.

3. General model

We generally describe a three-dimensional point. Since the camera may be moving all the time, we do not describe it based on the camera coordinate system, but in the world coordinate system. The relationship between the world coordinate system and the camera coordinate system can be described by a rotation matrix and a translation vector, as shown in Figure 4.

Let the coordinates of the space point in the world coordinate system and the camera coordinate system be X=(x,y,z,1) ^T , X _c =(x _c ,y _c ,z _c ,1) ^T , then the difference between the two Relationship:

Bringing (14) into (13), we get:

表示摄像机中心在世界坐标系中的坐标，摄像机矩阵

为摄像机的外参数矩阵。其中，R＝(α,β,γ)是旋转矩阵，α,β,γ分别是绕摄像机坐标系x,y,z轴的旋转角度。in,

Represents the coordinates of the camera center in the world coordinate system, the camera matrix

is the extrinsic parameter matrix of the camera. Among them, R=(α, β, γ) is the rotation matrix, and α, β, γ are the rotation angles around the camera coordinate system x, y, and z axes, respectively.

is the translation matrix, T _x , T _y , T _z are the translations around the camera coordinate system x, y, z axes, respectively, so the camera external parameter consists of 6 parameters (α, β, γ, T _x , T _y , T _z ).

FIG. 10 shows an augmented reality-based intelligent manufacturing network device according to another aspect of the present application, wherein the device includes a next step guidance sending module 21 . The next step instruction sending module 21 is configured to send the next step operation instruction information corresponding to the operation instruction information to the corresponding first user equipment.

For example, the first user holds augmented reality glasses, the augmented reality glasses use a camera to capture image information related to the current operating area, and the augmented reality glasses establish a communication connection with the network device. The network device determines the current work instruction information of the current first user based on the communication, determines the corresponding next work instruction information in the database, and sends the next work instruction information to the first user equipment. The first user equipment receives the next operation instruction information, and superimposes the next operation instruction information on the target object in real time at the position according to the position of the target object on the current screen.

In some embodiments, the next step instruction sending module 21 is configured to receive a next step operation request sent by the corresponding second user equipment and corresponding to the work instruction information in the corresponding first user equipment, wherein the first user equipment Serving the same intelligent manufacturing task as the second user equipment, determining the next work instruction information corresponding to the work instruction information, and sending the next work instruction information to the second user equipment. For example, the second user equipment includes a human-computer interaction device that establishes a communication connection with the first user equipment, and the second user equipment may also establish a communication connection with the network device. The second user equipment obtains the next operation request corresponding to the job instruction information of the current first user equipment. For example, the second user obtains the next operation request corresponding to the current job information of the target object based on the operation of the second user. The user equipment receives the next operation request sent by the first user equipment; the second user equipment sends the next operation request to the network device. The network device determines corresponding next-step operation instruction information according to the received next-step operation request, and sends the next-step operation instruction information to the second user equipment.

For example, the second user holds a human-computer interaction device (such as a tablet computer, etc.), the human-computer interaction device establishes a communication connection with augmented reality glasses and a network device, and the first user and the second user are performing unified production execution tasks. The human-computer interaction device and the augmented reality glasses present the same work instruction information. For example, the human-computer interaction device presents the work instruction information on the screen, and the augmented reality superimposes the corresponding work instruction information on the screen. The first user installs the electric control box on the console, and the first user has installed the circuit board in the corresponding position according to the presentation position of the virtual circuit board in the work instruction information, and the first user clicks to proceed to the next step, augmented reality glasses According to the operation, a next operation request is sent to the human-computer interaction device, wherein the next operation request includes relevant information of the current work instruction information (such as current work instruction information or image information corresponding to the completion of the work instruction, etc.). The human-computer interaction device receives the next step operation request, and sends the next step operation request to the MES production execution system, and the MES production execution system determines the next step operation instruction information corresponding to the target object in the database according to the next step operation request, For example, according to the image information that the circuit board has been installed in the outer box of the electric control box, the next step operation instruction information, such as fixing the corresponding screws in the corresponding position of the circuit board, is determined. The MES production execution system sends the next step operation guidance device to the human-computer interaction device, and forwards it to the augmented reality glasses via the human-computer interaction device.

For another example, the first user installs the electric control box on the console, the first user has installed the circuit board in the corresponding position according to the position where the virtual circuit board is presented in the work instruction information, and the second user selects on the screen of the human-computer interaction device. Carry out the next operation, and the human-computer interaction device sends the next operation request to the network device according to the operation, wherein the next operation request includes the relevant information of the current work instruction information (such as the current work instruction information or the corresponding information when the work instruction is completed). image information, etc.). The MES production execution system receives the next step operation request, and determines the next step operation instruction information corresponding to the target object in the database according to the next step operation request. For example, the next step is determined according to the image information of the circuit board installed in the outer box of the electric control box. Work instruction information, such as fixing the corresponding screws in the corresponding positions of the circuit board, etc. The MES production execution system sends the next step operation guidance device to the human-computer interaction device, and forwards it to the augmented reality glasses via the human-computer interaction device.

In some embodiments, the next-step instruction sending module 21 is configured to receive a next-step operation request on the work instruction information sent by the corresponding first user equipment, determine the next-step work instruction information corresponding to the work instruction information, and send the requested information to the next step. The next step operation instruction information is sent to the second user equipment. For example, after the user completes the current work instruction information, the user selects an operation to enter the next work on the first user device, or the first user determines that the current work instruction has been completed through the captured image of the work instruction. The operation sends the next operation request to the network device. The network device receives the next step operation request, determines corresponding next step operation instruction information in the database based on the next step operation request, and sends the next step operation instruction information to the second user equipment.

For example, the first user installs the electric control box on the console, and the first user has installed the circuit board in the corresponding position according to the presentation position of the virtual circuit board in the operation instruction information. According to the operation, the reality glasses send the next operation request to the MES production execution system; for another example, the augmented reality glasses use the camera to capture the image information related to the operation instruction of the electric control box on the current console, and determine the current operation according to the operation instruction information. The instruction information has been completed, and the augmented reality glasses send the next operation request to the MES production execution system, wherein the next operation request includes the relevant information of the current operation instruction information (such as the current operation instruction information or the corresponding image information when the operation instruction is completed, etc. ). The MES production execution system receives the next step operation request, and determines the next step operation instruction information corresponding to the target object in the database according to the next step operation request. For example, the next step is determined according to the image information of the circuit board installed in the outer box of the electric control box. Work instruction information, such as fixing the corresponding screws in the corresponding positions of the circuit board, etc. The MES production execution system sends the next step operation instruction information to the augmented reality glasses.

In some embodiments, the apparatus also includes a job completion determination module 22 (not shown). The job completion determination module 22 is configured to receive image information about the target object sent by the corresponding first user equipment, wherein the image information is obtained by photographing the target object being operated by the photographing device in the first user equipment, Determine whether the work instruction information in the first user equipment has been completed according to the image information; wherein, the next step instruction sending module 21, if the work instruction information has been completed, is configured to send the first user equipment all the information. Next step operation instruction information corresponding to the above operation instruction information. For example, the first user device captures image information related to the target object when the current user is working in real time, and sends the image information to the network device. The network device receives the image information and matches the preset image of the job completion according to the computer vision algorithm. If the image related to the target object in the captured image information matches the preset image, the network device determines that the current job related to the target object has been completed. Determine the next work instruction information corresponding to the current work instruction information, and send the next work instruction information to the first user equipment.

For example, the first user installs the electric control box on the console, the first user has installed the circuit board in the corresponding position according to the presentation position of the virtual circuit board in the work instruction information, and the augmented reality glasses photograph that the circuit board has been installed on the electric circuit board. Control the image information of the corresponding position in the outer box of the box, and send the image information to the MES production execution system. The MES production execution system receives the image information, and matches the preset completed model in the execution task in the database, matches the preset model of the completed job instruction information, determines that the current job instruction information has been completed, and the MES produces According to the completion of the current work instruction, the execution system obtains the corresponding next work instruction information, such as fixing the corresponding screws in the corresponding position of the circuit board. Subsequently, the MES production execution system sends the next step operation instruction information to the augmented reality device.

FIG. 11 shows a second user equipment of augmented reality-based smart manufacturing according to yet another aspect of the present application, wherein the device includes a next step request obtaining module 31 , a next step request sending module 32 , and a next step instruction receiving module 33 and the next step instructing the sending module 34 . The next step request obtaining module 31 is configured to obtain the next step operation request submitted by the user through the second user equipment and corresponding to the work instruction information in the corresponding first user equipment, wherein the first user equipment and the The second user equipment serves the same intelligent manufacturing task; the next step request sending module 32 is configured to send the next step operation request to the corresponding network device; the next step instruction receiving module 33 is configured to receive the network device based on the the next step operation instruction information corresponding to the operation instruction information sent by the next operation request; the next step instruction sending module 34 is configured to send the next step operation instruction information to the first user corresponding to the operation instruction information equipment. For example, the first user holds the first user equipment, the second user holds the computer interaction equipment (such as PC equipment, etc.), the first user equipment captures image information related to the target object in the current operation area through the camera, and the first user equipment A communication connection is established with the second user equipment and the network equipment. If the user's current work instruction has been completed, the second user equipment sends a next operation request to the network device based on the operation of the second user, and the network device receives the request, determines the next work instruction information corresponding to the request in the database, and sends the request to the network device. The next step operation instruction information is sent to the second user equipment. The second user receives the next step operation instruction information, and sends the next step operation instruction information to the first user equipment. The first user equipment receives the next operation instruction information, and superimposes the next operation instruction information on the target object in real time at the position according to the position of the target object on the current screen.

For example, the first user holds augmented reality glasses, the second user equipment holds a machine interaction device, the augmented reality device establishes a communication connection with the human-computer interaction device, and the human-computer interaction device establishes a communication connection with the MES production execution system. Human-computer interaction equipment and augmented reality glasses present the same operation guidance information, such as installation guidance information related to the outer box of the electric control box on the current console. The first user installs the electric control box on the console, the first user has installed the circuit board in the corresponding position according to the position of the virtual circuit board in the operation instruction information, and the second user selects the next step on the screen of the human-computer interaction device operation, the human-computer interaction device sends the next operation request to the network device according to the operation, wherein the next operation request includes the relevant information of the current work instruction information (such as the current work instruction information or the corresponding image information when the work instruction is completed). Wait). The MES production execution system receives the next step operation request, and determines the next step operation instruction information corresponding to the target object in the database according to the next step operation request. For example, the next step is determined according to the image information of the circuit board installed in the outer box of the electric control box. Work instruction information, such as fixing the corresponding screws in the corresponding positions of the circuit board, etc. The MES production execution system sends the next step operation guidance device to the human-computer interaction device, and forwards it to the augmented reality glasses via the human-computer interaction device. The augmented reality glasses receive the instructions for the next step of fixing the screws at the corresponding positions of the circuit boards installed in the outer box of the electric control box, and superimpose the corresponding position of the circuit boards in the outer box of the electric control box on the current screen to display the corresponding screws to be fixed, etc. Information such as where each screw will be fastened, in what order, and what type.

In some embodiments, the next step request obtaining module 31 is configured to receive a next step operation request sent by the first user equipment and corresponding to the work instruction information in the first user equipment, wherein the first user equipment and the The second user equipment serves the same intelligent manufacturing task.

For example, the first user installs the electric control box on the console, and the first user has installed the circuit board in the corresponding position according to the presentation position of the virtual circuit board in the operation instruction information. According to the operation, the reality glasses send a next operation request to the human-computer interaction device, wherein the next operation request includes the relevant information of the current work instruction information (such as the current work instruction information or the corresponding image information when the work instruction is completed, etc.). The human-computer interaction device receives the next step operation request, and the human-computer interaction device sends the next step operation request to the MES production execution system to determine the corresponding next step operation instruction information, and forwards the next step operation instruction information by the MES production execution system. to augmented reality glasses.

FIG. 12 shows an augmented reality-based smart manufacturing system according to an aspect of the present application, wherein the system has the above-mentioned first user equipment and the above-mentioned network equipment.

FIG. 13 shows an augmented reality-based intelligent manufacturing system according to an aspect of the present application, wherein the system includes the above-mentioned first user equipment, the above-mentioned network equipment, and the above-mentioned second user equipment.

The present application also provides a computer-readable storage medium, where the computer-readable storage medium stores computer code, and when the computer code is executed, the method described in any preceding item is executed.

The present application also provides a computer program product, when the computer program product is executed by a computer device, the method according to any one of the preceding items is executed.

The present application also provides a computer device, the computer device comprising:

one or more processors;

memory for storing one or more computer programs;

The one or more computer programs, when executed by the one or more processors, cause the one or more processors to implement the method of any preceding item.

Figure 14 illustrates an exemplary system that may be used to implement various embodiments described in this application;

As shown in FIG. 14 , in some embodiments, the system 300 can be used as an augmented reality-based smart manufacturing device in any of the described embodiments. In some embodiments, system 300 may include one or more computer-readable media (eg, system memory or NVM/storage device 320 ) having instructions and be coupled to the one or more computer-readable media and configured to execute Instructions to implement a module to perform one or more processors (eg, processor(s) 305 ) to perform the actions described herein.

For one embodiment, the system control module 310 may include any suitable interface controller to provide at least one of the processor(s) 305 and/or any suitable device or component in communication with the system control module 310 any appropriate interface.

The system control module 310 may include a memory controller module 330 to provide an interface to the system memory 315 . The memory controller module 330 may be a hardware module, a software module, and/or a firmware module.

System memory 315 may be used, for example, to load and store data and/or instructions for system 300 . For one embodiment, system memory 315 may include any suitable volatile memory, eg, suitable DRAM. In some embodiments, system memory 315 may include double data rate type quad synchronous dynamic random access memory (DDR4 SDRAM).

For one embodiment, system control module 310 may include one or more input/output (I/O) controllers to provide interfaces to NVM/storage device 320 and communication interface(s) 325 .

For example, NVM/storage device 320 may be used to store data and/or instructions. NVM/storage device 320 may include any suitable non-volatile memory (eg, flash memory) and/or may include any suitable non-volatile storage device(s) (eg, one or more hard drives ( HDD), one or more compact disc (CD) drives and/or one or more digital versatile disc (DVD) drives).

NVM/storage device 320 may include storage resources that are physically part of the device on which system 300 is installed, or it may be accessed by the device without necessarily being part of the device. For example, the NVM/storage device 320 is accessible via the communication interface(s) 325 over a network.

Communication interface(s) 325 may provide an interface for system 300 to communicate over one or more networks and/or with any other suitable device. System 300 may wirelessly communicate with one or more components of a wireless network in accordance with any of one or more wireless network standards and/or protocols.

For one embodiment, at least one of the processor(s) 305 may be packaged with the logic of one or more controllers of the system control module 310 (eg, the memory controller module 330 ). For one embodiment, at least one of the processor(s) 305 may be packaged with logic of one or more controllers of the system control module 310 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 305 may be integrated on the same die with the logic of one or more controllers of the system control module 310 . For one embodiment, at least one of the processor(s) 305 may be integrated on the same die with logic of one or more controllers of the system control module 310 to form a system on a chip (SoC).

In various embodiments, system 300 may be, but is not limited to, a server, workstation, desktop computing device, or mobile computing device (eg, laptop computing device, handheld computing device, tablet computer, netbook, etc.). In various embodiments, system 300 may have more or fewer components and/or different architectures. For example, in some embodiments, system 300 includes one or more cameras, keyboards, liquid crystal display (LCD) screens (including touchscreen displays), non-volatile memory ports, multiple antennas, graphics chips, application specific integrated circuits ( ASIC) and speakers.

It should be noted that the present application may be implemented in software and/or a combination of software and hardware, eg, an application specific integrated circuit (ASIC), a general purpose computer, or any other similar hardware device. In one embodiment, the software program of the present application may be executed by a processor to implement the steps or functions described above. Likewise, the software programs of the present application (including associated data structures) may be stored on a computer-readable recording medium, such as RAM memory, magnetic or optical drives or floppy disks, and the like. In addition, some steps or functions of the present application may be implemented in hardware, for example, as a circuit that cooperates with a processor to perform various steps or functions.

In addition, a part of the present application can be applied as a computer program product, such as computer program instructions, which when executed by a computer, through the operation of the computer, can invoke or provide methods and/or technical solutions according to the present application. Those skilled in the art should understand that the existing forms of computer program instructions in computer-readable media include but are not limited to source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include but are not limited to Limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding post-installation program. program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium that can be accessed by a computer.

Communication media includes media by which communication signals containing, for example, computer readable instructions, data structures, program modules or other data are transmitted from one system to another. Communication media may include conducted transmission media such as cables and wires (eg, fiber optic, coaxial, etc.) and wireless (unconducted transmission) media capable of propagating energy waves, such as acoustic, electromagnetic, RF, microwave, and infrared . Computer readable instructions, data structures, program modules or other data may be embodied, for example, as a modulated data signal in a wireless medium such as a carrier wave or similar mechanism such as embodied as part of spread spectrum technology. The term "modulated data signal" refers to a signal whose one or more characteristics are altered or set in a manner that encodes information in the signal. Modulation can be analog, digital or hybrid modulation techniques.

By way of example and not limitation, computer-readable storage media may include volatile and non-volatile, readable storage media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Removable and non-removable media. For example, computer-readable storage media include, but are not limited to, volatile memory, such as random access memory (RAM, DRAM, SRAM); and non-volatile memory, such as flash memory, various read-only memories (ROM, PROM, EPROM) , EEPROM), magnetic and ferromagnetic/ferroelectric memory (MRAM, FeRAM); and magnetic and optical storage devices (hard disks, tapes, CDs, DVDs); or other media now known or later developed capable of storing data for computer systems Computer readable information/data used.

Here, an embodiment according to the present application includes an apparatus comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein, when the computer program instructions are executed by the processor, a trigger is The apparatus operates based on the aforementioned methods and/or technical solutions according to various embodiments of the present application.

It will be apparent to those skilled in the art that the present application is not limited to the details of the above-described exemplary embodiments, but that the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Accordingly, the embodiments are to be regarded in all respects as illustrative and not restrictive, and the scope of the application is to be defined by the appended claims rather than the foregoing description, which is therefore intended to fall within the scope of the claims. All changes within the meaning and scope of the equivalents of , are included in this application. Any reference signs in the claims shall not be construed as limiting the involved claim. Furthermore, it is clear that the word "comprising" does not exclude other units or steps and the singular does not exclude the plural. Several units or means recited in the device claims can also be realized by one unit or means by means of software or hardware. The terms first, second, etc. are used to denote names and do not denote any particular order.

Claims (33)

1. A smart manufacturing method based on augmented reality at a first user equipment side, wherein the method comprises: obtaining the first position information of the target object in the operation area in the first coordinate system corresponding to the operation area; According to the first coordinate mapping relationship between the second coordinate system corresponding to the first user equipment and the human eye coordinate system, the second coordinate mapping relationship between the human eye coordinate system and the camera coordinate system, and the camera coordinate system and the The third coordinate mapping relationship of the first coordinate system, determining the coordinate mapping relationship between the second coordinate system corresponding to the first user equipment and the first coordinate system, wherein the second coordinate system includes a coordinate mapping relationship based on the first coordinate system. A two-dimensional coordinate system established by the screen of the user equipment, the human eye coordinate system includes a three-dimensional coordinate system established based on the human eye and the screen; determining the second position information of the target object in the second coordinate system based on the first position information, the coordinate mapping relationship between the second coordinate system and the first coordinate system; obtaining the operation instruction information corresponding to the target object; The operation instruction information corresponding to the target object is superimposed and displayed on the target object according to the second position information. 2. The method of claim 1, wherein the method further comprises: The work instruction information is sent to the corresponding third user equipment, so that the third user equipment superimposes and displays the work instruction information on the image about the target object. 3. The method of claim 1 or 2, wherein the method further comprises: Photograph the image information of the target object in the operation area by the photographing device of the first user equipment; Wherein, the obtaining the first position information of the target object in the operation area in the first coordinate system corresponding to the operation area includes: The first position information of the target object in the first coordinate system corresponding to the operation area is acquired by using the image information. 4. The method according to claim 3, wherein the acquiring operation instruction information corresponding to the target object comprises: Acquire work instruction information corresponding to the target object according to the image information. 5. The method of claim 1, wherein the method further comprises: Acquiring the next work instruction information corresponding to the work instruction information; The next step operation instruction information is superimposed and displayed on the target object. 6. The method according to claim 5, wherein the acquiring the next step operation instruction information corresponding to the operation instruction information comprises: Receive the next work instruction information sent by the corresponding network device and corresponding to the work instruction information. 7. The method according to claim 6, wherein the next step operation instruction information is sent by the network device based on the received next step operation request. 8. The method according to claim 7, wherein the receiving the next work instruction information corresponding to the work instruction information sent by the corresponding network device comprises: Receive next-step work instruction information sent by a corresponding second user equipment and corresponding to the work instruction information, wherein the next-step work instruction information is sent by the network device to the second user equipment. 9. The method according to claim 7, wherein the receiving the next work instruction information corresponding to the work instruction information sent by the corresponding network device comprises: obtaining a next operation request submitted by the user through the first user equipment regarding the work instruction information; sending the next operation request to the network device; Receive the next work instruction information corresponding to the work instruction information sent by the network device based on the next operation request. 10. The method of claim 6, wherein the method further comprises: Photograph the target object being operated by the photographing device in the first user equipment, and send the photographed image information to the corresponding network device; Wherein, the receiving the next work instruction information corresponding to the work instruction information sent by the corresponding network device includes: Receive next work instruction information corresponding to the work instruction information and sent by the network device after it is determined according to the image information that the work instruction information has been completed. 11. The method of claim 5, wherein the method further comprises: determining the current position information of the target object in the second coordinate system after the operation; Wherein, the superimposing and displaying the next step operation instruction information on the target object includes: The next step operation instruction information is superimposed and displayed on the target object according to the current position information. 12. The method of claim 5, wherein the method further comprises: Photograph the real-time image information of the target object being operated by the photographing device in the first user equipment; Determine whether the work instruction information in the first user equipment has been completed according to the real-time image information; Wherein, the obtaining the next job instruction information corresponding to the job instruction information includes: If the operation instruction information has been completed, read the next operation instruction information corresponding to the operation instruction information. 13. A smart manufacturing method based on augmented reality at a network device side, wherein the method comprises: Send the next step operation instruction information corresponding to the operation instruction information to the corresponding first user equipment, wherein the next step operation instruction information includes the next operation of the operation instruction information corresponding to the target object, and the operation instruction information is used according to the second operation instruction information. The position information is superimposed and displayed on the target object, the second position information includes the position information of the target object in the corresponding second coordinate system, and the second position information is based on the first position corresponding to the target object in the operation area. The first position information in the coordinate system, the coordinate mapping relationship between the second coordinate system and the first coordinate system are determined, and the coordinate mapping relationship is based on the first coordinate mapping between the second coordinate system and the human eye coordinate system. relationship, the second coordinate mapping relationship between the human eye coordinate system and the camera coordinate system, and the third coordinate mapping relationship between the camera coordinate system and the first coordinate system, the second coordinate system includes A two-dimensional coordinate system established by a screen of a user equipment, and the human eye coordinate system includes a three-dimensional coordinate system established based on the human eye and the screen. 14. The method according to claim 13, wherein the sending the next operation instruction information corresponding to the operation instruction information to the corresponding first user equipment comprises: receiving a next step operation request sent by the corresponding second user equipment and corresponding to the work instruction information in the corresponding first user equipment, wherein the first user equipment and the second user equipment serve the same intelligent manufacturing task; determining the next work instruction information corresponding to the work instruction information; Sending the next step operation instruction information to the second user equipment. 15. The method according to claim 13, wherein the sending the next operation instruction information corresponding to the operation instruction information to the corresponding first user equipment comprises: Receive a next operation request about the work instruction information sent by the corresponding first user equipment; determining the next work instruction information corresponding to the work instruction information; The next step operation instruction information is sent to the corresponding second user equipment, wherein the first user equipment and the second user equipment serve the same intelligent manufacturing task. 16. The method of claim 13, wherein the method further comprises: receiving image information about the target object sent by the corresponding first user equipment, wherein the image information is obtained by photographing the target object being operated by a photographing device in the first user equipment; determining, according to the image information, whether the work instruction information in the first user equipment has been completed; Wherein, the sending the next step operation instruction information corresponding to the operation instruction information to the corresponding first user equipment includes: If the work instruction information has been completed, the next work instruction information corresponding to the work instruction information is sent to the first user equipment. 17. An augmented reality-based smart manufacturing method on a second user equipment side, wherein the method comprises: Obtain the next operation request submitted by the user through the second user equipment and corresponding to the work instruction information in the corresponding first user equipment, wherein the first user equipment and the second user equipment serve the same intelligent manufacturing Task; sending the next step operation request to the corresponding network device; Receive next-step operation instruction information corresponding to the operation instruction information sent by the network device based on the next-step operation request, wherein the next-step operation instruction information includes the next step of the operation instruction information corresponding to the target object operation, the work instruction information is used to be superimposed and displayed on the target object according to the second position information, the second position information includes the position information of the target object in the corresponding second coordinate system, the second position information It is determined based on the first position information of the target object in the first coordinate system corresponding to the operation area, the coordinate mapping relationship between the second coordinate system and the first coordinate system, and the coordinate mapping relationship is based on the second coordinate mapping relationship. The first coordinate mapping relationship between the coordinate system and the human eye coordinate system, the second coordinate mapping relationship between the human eye coordinate system and the camera coordinate system, and the third coordinate mapping relationship between the camera coordinate system and the first coordinate system are determined , the second coordinate system includes a two-dimensional coordinate system established based on the screen of the first user equipment, and the human eye coordinate system includes a three-dimensional coordinate system established based on the human eye and the screen; Sending the next step operation instruction information to the first user equipment corresponding to the operation instruction information. 18. The method according to claim 17, wherein the acquiring the next operation request submitted by the user through the second user equipment and corresponding to the work instruction information in the corresponding first user equipment, wherein the first A user equipment and the second user equipment serve the same intelligent manufacturing task, including: A next operation request corresponding to the work instruction information in the first user equipment is received and sent by the first user equipment, wherein the first user equipment and the second user equipment serve the same intelligent manufacturing task. 19. An augmented reality-based smart manufacturing method, wherein the method comprises: The first user equipment obtains the first position information of the target object in the operation area in the first coordinate system corresponding to the operation area, according to the second coordinate system corresponding to the first user equipment and the first coordinate of the human eye coordinate system The mapping relationship, the second coordinate mapping relationship between the human eye coordinate system and the camera coordinate system, and the third coordinate mapping relationship between the camera coordinate system and the first coordinate system, determine the second coordinate mapping relationship corresponding to the first user equipment The coordinate mapping relationship between the coordinate system and the first coordinate system, wherein the second coordinate system includes a two-dimensional coordinate system established based on the screen of the first user equipment, and the human eye coordinate system includes The three-dimensional coordinate system established by the screen; based on the first position information, the coordinate mapping relationship between the second coordinate system and the first coordinate system, determine the first position of the target object in the second coordinate system. Second position information, obtaining the operation instruction information corresponding to the target object, and superimposing and displaying the operation instruction information corresponding to the target object on the target object according to the second position information; The network device sends the next step operation instruction information corresponding to the operation instruction information to the first user equipment; The first user equipment receives the next-step operation instruction information, and superimposes and displays the next-step operation instruction information on the target object. 20. An augmented reality-based smart manufacturing method, wherein the method comprises: The first user equipment obtains the first position information of the target object in the operation area in the first coordinate system corresponding to the operation area, according to the second coordinate system corresponding to the first user equipment and the first coordinate of the human eye coordinate system The mapping relationship, the second coordinate mapping relationship between the human eye coordinate system and the camera coordinate system, and the third coordinate mapping relationship between the camera coordinate system and the first coordinate system, determine the second coordinate mapping relationship corresponding to the first user equipment The coordinate mapping relationship between the coordinate system and the first coordinate system, wherein the second coordinate system includes a two-dimensional coordinate system established based on the screen of the first user equipment, and the human eye coordinate system includes The three-dimensional coordinate system established by the screen; based on the first position information, the coordinate mapping relationship between the second coordinate system and the first coordinate system, determine the first position of the target object in the second coordinate system. Second position information, obtaining the operation instruction information corresponding to the target object, and superimposing and displaying the operation instruction information corresponding to the target object on the target object according to the second position information; The second user equipment acquires the next operation request submitted by the user through the second user equipment and corresponding to the work instruction information in the first user equipment, wherein the first user equipment and the second user equipment Serving the same intelligent manufacturing task; sending, by the second user equipment, the next step operation request to the corresponding network device; The network device receives the next operation request, determines the next operation instruction information corresponding to the operation instruction information, and sends the next operation instruction information to the second user equipment; The second user equipment receives the next step operation instruction information, and sends the next step operation instruction information to the first user equipment; The first user equipment receives the next-step operation instruction information, and superimposes and displays the next-step operation instruction information on the target object. 21. An augmented reality-based smart manufacturing first user equipment, wherein the device comprises: a first position acquisition module, configured to acquire the first position information of the target object in the operation area in the first coordinate system corresponding to the operation area; The second position acquisition module is configured to, according to the first coordinate mapping relationship between the second coordinate system corresponding to the first user equipment and the human eye coordinate system, the second coordinate mapping relationship between the human eye coordinate system and the camera coordinate system, and The third coordinate mapping relationship between the camera coordinate system and the first coordinate system determines the coordinate mapping relationship between the second coordinate system corresponding to the first user equipment and the first coordinate system, wherein the second coordinate system The coordinate system includes a two-dimensional coordinate system established based on the screen of the first user equipment, and the human eye coordinate system includes a three-dimensional coordinate system established based on the human eye and the screen; The coordinate mapping relationship between the second coordinate system and the first coordinate system determines the second position information of the target object in the second coordinate system; a guidance information acquisition module, used for acquiring the operation guidance information corresponding to the target object; The instruction information superimposing module is configured to superimpose and display the operation instruction information corresponding to the target object on the target object according to the second position information. 22. The apparatus of claim 21, wherein the apparatus further comprises: A work instruction sending module, configured to send the work instruction information to a corresponding third user equipment, so that the third user equipment superimposes and displays the work instruction information on the image about the target object. 23. The apparatus of claim 21 or 22, wherein the apparatus further comprises: a next-step guidance acquisition module, configured to obtain the next-step operation guidance information corresponding to the operation guidance information; The next step guidance overlay module is used to superimpose and display the next step operation guidance information on the target object. 24. The device according to claim 23, wherein the next step guide obtaining module is used for: Receive the next work instruction information sent by the corresponding network device and corresponding to the work instruction information. 25. The device according to claim 24, wherein the next step guide acquisition module is used to: obtaining a next operation request submitted by the user through the first user equipment regarding the work instruction information; sending the next operation request to the network device; Receive the next work instruction information corresponding to the work instruction information sent by the network device based on the next operation request. 26. The apparatus of claim 24, wherein the apparatus further comprises: a photographing operation module, configured to photograph the target object being operated by the photographing device in the first user equipment, and send the photographed image information to the corresponding network device; Wherein, the next step guidance acquisition module is used for: Receive next work instruction information corresponding to the work instruction information and sent by the network device after it is determined according to the image information that the work instruction information has been completed. 27. The device according to claim 24, wherein the device further comprises a next step reading module, which is used for: Photograph the real-time image information of the target object being operated by the photographing device in the first user equipment; Determine whether the work instruction information in the first user equipment has been completed according to the real-time image information; Wherein, the next step guidance acquisition module is used for: If the operation instruction information has been completed, read the next operation instruction information corresponding to the operation instruction information. 28. An augmented reality-based intelligent manufacturing network device, wherein the device comprises: A next step instruction sending module, configured to send the next step operation instruction information corresponding to the operation instruction information to the corresponding first user equipment, wherein the next step operation instruction information includes the next operation of the operation instruction information corresponding to the target object, the The operation instruction information is used to be superimposed and displayed on the target object according to second position information, the second position information includes the position information of the target object in the corresponding second coordinate system, and the second position information is based on the target The first position information of the object in the first coordinate system corresponding to the operation area, and the coordinate mapping relationship between the second coordinate system and the first coordinate system are determined, and the coordinate mapping relationship is determined according to the second coordinate system and the human The first coordinate mapping relationship of the eye coordinate system, the second coordinate mapping relationship between the human eye coordinate system and the camera coordinate system, and the third coordinate mapping relationship between the camera coordinate system and the first coordinate system are determined, and the first coordinate mapping relationship is determined. The two coordinate system includes a two-dimensional coordinate system established based on the screen of the first user equipment, and the human eye coordinate system includes a three-dimensional coordinate system established based on the human eye and the screen. 29. A second user equipment based on augmented reality intelligent manufacturing, wherein the equipment comprises: The next step request acquisition module is configured to acquire the next step operation request submitted by the user through the second user equipment and corresponding to the work instruction information in the corresponding first user equipment, wherein the first user equipment and the first user equipment Two user equipments serve the same intelligent manufacturing task; The next step request sending module is used to send the next step operation request to the corresponding network device; A next step instruction receiving module, configured to receive the next step operation instruction information corresponding to the operation instruction information sent by the network device based on the next step operation request, wherein the next step operation instruction information includes the corresponding target object. The next step of the operation instruction information, the operation instruction information is used to be superimposed and displayed on the target object according to the second position information, and the second position information includes the position of the target object in the corresponding second coordinate system information, the second position information is determined based on the first position information of the target object in the first coordinate system corresponding to the operation area, and the coordinate mapping relationship between the second coordinate system and the first coordinate system, and the The coordinate mapping relationship is based on the first coordinate mapping relationship between the second coordinate system and the human eye coordinate system, the second coordinate mapping relationship between the human eye coordinate system and the camera coordinate system, and the camera coordinate system and the first coordinate system. The third coordinate mapping relationship of the system is determined, the second coordinate system includes a two-dimensional coordinate system established based on the screen of the first user equipment, and the human eye coordinate system includes three-dimensional coordinates established based on the human eye and the screen Tie; The next step instruction sending module is configured to send the next step operation instruction information to the first user equipment corresponding to the operation instruction information. 30. An augmented reality-based intelligent manufacturing system, wherein the system comprises the first user equipment according to any one of claims 21 to 27 and the network equipment according to claim 28. 31. The system of claim 30, wherein the system further comprises the second user equipment of claim 29. 32. A smart manufacturing device based on augmented reality, wherein the device comprises: processor; and a memory arranged to store computer-executable instructions which, when executed, cause the processor to perform the operations of the method of any of claims 1 to 18. 33. A computer readable medium comprising instructions which, when executed, cause a system to operate the method of any one of claims 1 to 18.

Images