KR102210803B1 - System, method and apparatus for smart management based on augmented reality - Google Patents

Abstract

The present invention relates to a method, a device, and a system for smart management based on augmented reality. The method of the present invention comprises the steps of: rendering an augmented reality image representing a pre-designed 3D modeling of a first device when a first tag corresponding to the first device is identified by a user terminal; displaying at least one real-time data related to the first device together with an augmented reality image of the first device; and synchronizing a pattern and timing of an operation of the augmented reality image with a pattern and timing of an actual operation of the first device as the at least one real-time data is reflected to the pre-designed 3D modeling of the first device. According to the present invention, more natural and realistic augmented reality can be provided.

Description

Augmented reality-based smart management method, device, and system {SYSTEM, METHOD AND APPARATUS FOR SMART MANAGEMENT BASED ON AUGMENTED REALITY}

The present invention relates to a smart management method, apparatus, and system based on Augmented Reality. More specifically, it relates to an augmented reality-based smart management method, apparatus, and system capable of rendering an augmented reality image that intuitively displays the state of equipment installed in a management place.

Conventionally, a system for managing agricultural and fishery equipment has transmitted an image of the equipment operation in real time to a user terminal located in a remote place, or provided data indicating the current state of the equipment, and receives the operation signal of the equipment from the user terminal. It was limited to operating the equipment remotely.

In such a conventional system for managing agricultural and fishery equipment, even if the user watches the video of the equipment through the user terminal, it is not possible to check detailed conditions such as a failure of internal parts of the equipment or a failure of software, and even if receiving data related to the equipment. An additional process was needed to directly check the data, analyze the data, and determine whether the current equipment is normal or abnormal. For example, life management data of lateral sensors and detailed information such as calibration are highly difficult tasks that the user running the farm cannot perform the data analysis process, so the data is practically useless to the user. It was being handled. Accordingly, the user could not intuitively grasp whether the equipment is operating normally through the user terminal, and users who manage agricultural and fishery equipment with limited time and space restrictions due to the working environment can properly utilize the system for managing the conventional agricultural and fishery equipment. There was no.

In particular, in the case of farms, they are distributed in islands or remote areas, but there are many cases where it is difficult for a professional engineer to actually visit. In many cases, sensitive organisms such as shrimp are farmed in aquaculture farms. In the case of organisms raised in aquaculture farms, a large amount of damage occurs in a short period of time even when the environment changes, so prompt action is required. However, in the event of equipment failure in the farm, it was difficult for experts to visit, so there were frequent cases where the farm was quickly damaged without any assistance from experts.

Accordingly, even for general users who are not experts in the agricultural and fishery industry, there is a need for a technology that enables easy management of equipment and easy provision of specialized technologies from external experts.

Republic of Korea Patent Publication 10-2089005 B1

The technical problem to be solved by the present invention is to provide an augmented reality-based smart management method, apparatus, and system capable of rendering an augmented reality image that can intuitively express the state of the equipment using data of equipment installed in a management place. will be.

Another technical problem to be solved by the present invention is to provide an augmented reality-based smart management method, device, and system capable of displaying an augmented reality image of equipment installed in a management place and various information related to the equipment at once.

Another technical problem to be solved by the present invention is to provide an augmented reality-based smart management method, apparatus, and system capable of simultaneously rendering information for managing or repairing equipment installed in a management place together with an augmented reality image. .

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

In an augmented reality-based smart management method according to an embodiment of the present invention for solving the technical problem, in the method performed by a computing device, the step of taking a management place by a user terminal, and by the user terminal Rendering a pre-designed augmented reality image representing 3D modeling of the first equipment when the first equipment installed in the management place is photographed and a first tag corresponding to the first equipment is identified by the user terminal, Displaying at least one or more real-time data related to the first device together with an augmented reality image of the first device, and the augmentation by reflecting the at least one real-time data to the 3D modeling of the pre-designed first device Synchronizing the pattern of the motion of the real image and the timing of the motion with the pattern of the actual motion of the first device and the timing of the motion, wherein the real-time data is related to a first object managed by the first device. Including data and failure data of the first equipment, wherein the data related to the first object is data obtained by a first sensor that detects the first object, and the failure data of the first equipment is transmitted to a second sensor. It is data related to the operating state of the first device obtained by the augmented reality image of the first device, the augmented reality image to perform a function of an indicator to display the current state of the first device, the indicator Is to highlight the augmented reality image of the first device according to whether the at least one real-time data matches a reference rule, and the reference rule is, the data measured by the second sensor is determined for each of the real-time data items. If it is out of the preset normal range data, it may be a rule for highlighting the augmented reality image of the first device.

In one embodiment, the displaying of at least one or more real-time data related to the first device together with the augmented reality image of the first device comprises: decomposing the augmented reality image of the first device into parts of the first device, , Comprising the step of displaying each part apart, and highlighting the part violating the standard rule among the disassembled parts, and the pre-designed first equipment with the at least one real-time data Synchronizing the operation pattern and the operation timing of the augmented reality image with the actual operation pattern and operation timing of the first device as reflected in the 3D modeling of the augmented reality image, the reference rule using at least one real-time data Analyzing the operation pattern and the timing of the operation of the part violating the reference rule, and rendering the operation pattern and the operation timing of the part violating the reference rule into the augmented reality image of the first device. I can.

In one embodiment, displaying an A/S button associated with the augmented reality image of the highlighted first equipment, and when the A/S button is input, maintaining the first equipment as a repair server for maintenance Further comprising the step of requesting maintenance, wherein the step of requesting maintenance to the repair server comprises: self-repairing the highlighted parts in case of the first level according to the level in which the first equipment violates the reference rule. Receiving a remote support indicator for the repair server from the repair server, displaying the remote support indicator on a real-time image of the first equipment, comparing the remote support indicator with an operation displayed on the real-time image of the first equipment If it is determined that the first equipment has actually moved according to the remote support indicator, transmitting operation completion information to the repair server, and receiving a second remote support indicator from the repair server in response to the operation completion information It may include the step of.

The augmented reality-based smart management method, device, and system according to an embodiment of the present invention renders the state of the device as an augmented reality image using data of the device installed in the management place, so that information that can intuitively identify the state of the device Can provide.

In addition, the augmented reality-based smart management method, device, and system according to an embodiment of the present invention provides a more natural and realistic augmented reality by synchronizing the augmented reality image of the device with the operation of the actual device using real-time data. can do.

In addition, since the augmented reality-based smart management method, device, and system according to an embodiment of the present invention displays which parts of the equipment currently operate in the same manner as actual operations, maintenance, management and management of the smart management system It can provide intuitive information about pay.

In addition, the augmented reality-based smart management method, device, and system according to an embodiment of the present invention is a remote display displayed on the device without calling a remote expert in case of a malfunction of the device that is only a level that can be repaired by the user. By looking at the support indicator, you can assist users to repair the equipment themselves.

1 is an exemplary diagram of an augmented reality-based smart management system according to an embodiment of the present invention.
2 is a flow chart of a smart management method based on an augmented reality according to an embodiment of the present invention.
3 to 5 are diagrams for explaining an example of displaying an augmented reality image of a first device and at least one or more real-time data related to the first device on a user terminal.
6 is a flow chart for describing in more detail some of the steps described with reference to FIG. 2.
7 is a view for explaining an example in which some defective parts are highlighted to perform an indicator function according to FIG. 5.
8 is a diagram for explaining a UI displayed on a user terminal that can be referred to in some embodiments of the present invention.
9 is a flowchart of an augmented reality-based smart management method according to another embodiment of the present invention.
10 is a diagram illustrating a remote assistance indicator displayed on a real-time image of a first device.
11 is a hardware configuration diagram of a smart management device based on an augmented reality according to another embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the technical idea of the present invention is not limited to the following embodiments, but may be implemented in various different forms, and only the following embodiments complete the technical idea of the present invention, and in the technical field to which the present invention pertains. It is provided to completely inform the scope of the present invention to those of ordinary skill in the art, and the technical idea of the present invention is only defined by the scope of the claims.

In adding reference numerals to elements of each drawing, it should be noted that the same elements are assigned the same numerals as possible even if they are indicated on different drawings. In addition, in describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the subject matter of the present invention, a detailed description thereof will be omitted.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used with meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically. The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase.

In addition, in describing the constituent elements of the present invention, terms such as first, second, A, B, (a) and (b) may be used. These terms are only used to distinguish the component from other components, and the nature, order, or order of the component is not limited by the term. When a component is described as being "connected", "coupled" or "connected" to another component, the component may be directly connected or connected to that other component, but another component between each component It should be understood that elements may be “connected”, “coupled” or “connected”.

As used in the specification, "comprises" and/or "comprising" refers to the presence of one or more other components, steps, actions and/or elements, and/or elements, steps, actions and/or elements mentioned. Or does not exclude additions.

Hereinafter, some embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is an exemplary diagram of an augmented reality-based smart management system according to an embodiment of the present invention.

Referring to FIG. 1, a smart management system based on an augmented reality according to an embodiment of the present invention may include a smart management device 100 and a user terminal 200. Each component of the smart management system disclosed in FIG. 1 represents functional elements that are functionally divided, and any one or more components may be integrated and implemented with each other in an actual physical environment. Hereinafter, each component will be described in detail.

The smart management device 100 may be a device that monitors the equipment (1-1 to 1-8) installed in the management place (A) and manages the operation and state of the equipment (1-1 to 1-8). In the present specification, the management place (A) may be a place where management related to agriculture, fishery, livestock industry, or three forestry is performed, but is not limited thereto. In one embodiment, the management site A may be a farm where living things are raised. The equipment (1-1 to 1-8) may be equipment such as a pumping device installed in the management place (A), a seawater pump, a seawater tank, an oxygen generator, an oxygen dissolver, a power device, a switchboard, a boiler, a boiler pipe, and the like, The types of equipment (1-1 to 1-8) are not limited thereto, and may be applied to various facilities that may be installed in agriculture, fishing, livestock industry, or forestry. In addition, the number of the equipment (1-1 to 1-8) is not limited.

The smart management device 100 may generate real-time data on the equipment (1-1 to 1-8) installed in the management location (A) and the surrounding environment of the management location (A). The smart management device 100 may collect real-time data through various sensors installed in the management place A. For example, the smart management device 100 may collect real-time data using a camera, a temperature sensor, a humidity sensor, a gyro sensor, and a water quality sensor. The smart management device 100 may transmit various types of real-time information detected in the management place A to the user terminal 200.

The user terminal 200 that may be referred to in some embodiments of the present invention may be any device capable of transmitting and receiving information to and from the smart management device 100 through a network and outputting an augmented reality image. For example, the user terminal 200 that can be referred to in an embodiment of the present invention is a desktop (Desktop), a workstation (Workstation), a server (Server), a laptop (Laptop), a tablet (Tablet), a smartphone ( Smart Phone) or a phablet (Phablet), but is not limited thereto, and consists of a device in the form of a portable multimedia player (PMP), personal digital assistants (PDA), etc. May be.

The user terminal 200 may photograph the management place A and display an augmented reality image. Specifically, the user terminal 200 displays the management location A and at the same time uses various pieces of information received from the smart management device 100 to the management location A on the equipment 1-1 to 1-8. It is possible to render an augmented reality image for the installed equipment (1-1 to 1-8). In this case, the user terminal 200 may display an augmented reality image of a different color for each state according to the state of the equipment 1-1 to 1-8. For example, the user terminal 200 displays a green augmented reality image when the equipment (1-1 to 1-8) is normal, and a red augmented reality image when the equipment (1-1 to 1-8) is broken. Can be displayed.

Here, the augmented reality image may be an image generated using 3D modeling of pre-designed equipment (1-1 to 1-8). The augmented reality image is an image generated by the smart management device 100 and then transmitted to the user terminal 200, or the user terminal 200 uses 3D modeling of pre-designed equipment (1-1 to 1-8). Although it may be a generated image, it should be noted that it does not limit a subject generating or rendering an augmented reality image.

The user terminal 200 may identify tags corresponding to the equipment 1-1 to 1-8 installed in the management place A. More specifically, the user terminal 200 can take a picture of the management place (A) using a camera, and when identifying a tag corresponding to the equipment (1-1 to 1-8) installed in the management place (A), the corresponding Augmented reality images of the devices 1-1 to 1-8 may be rendered and displayed on the display.

The user terminal 200 renders an augmented reality image using pre-designed 3D modeling, and the first equipment (1) pre-designed various real-time data received from the smart management device 100 to display a more realistic augmented reality image. -1 to 1-8) can be reflected in the 3D modeling. In order to express the augmented reality image in the same motion as the real one, the user terminal 200 determines the motion pattern and the motion timing of the augmented reality image as the actual motion pattern and motion of the first equipment (1-1 to 1-8). Can be synchronized to the timing of.

In addition, the user terminal 200 may display a UI button for maintenance, management or maintenance of the equipment 1-1 to 1-8, and may receive an input of the UI button from the user. When a UI for maintenance, management, and maintenance is input to the user terminal 200, an audio or video call may be connected to a repair server for repairing the equipment 1-1 to 1-8. In addition, the user terminal 200 may be provided with a guide for operating the equipment (1-1 to 1-8) from the repair server.

The augmented reality-based smart management system according to an embodiment of the present invention uses the data of the equipment (1-1 to 1-8) installed in the management place (A) to determine the status of the equipment (1-1 to 1-8). Since the is rendered as an augmented reality image, information for intuitively grasping the state of the equipment (1-1 to 1-8) can be provided.

So far, a smart management system according to an embodiment of the present invention has been schematically described with reference to FIG. 1. Hereinafter, a smart management method according to an embodiment of the present invention will be described in detail with reference to FIGS. 2 to 8.

The smart management method according to the present embodiment may be performed by a computing device. In addition, the method according to the present embodiment may be performed by being divided by the first computing device and the second computing device. Hereinafter, in performing each operation of the method according to the present embodiment, if the description of the subject is omitted, the subject may be interpreted as being the computing device.

In step S100 of FIG. 2, a management place may be photographed by the user terminal. In this case, a management location may be photographed using a smart management application of the user terminal or a smart management web. Thereafter, when the first tag corresponding to the first device is identified by the user terminal in step S200, the augmented reality image of the first device may be rendered in step S300. In this case, the first tag is a tag corresponding to the first device and may be formed of a QR code. When the first tag is not identified in step S200, only an image photographing a management location may be displayed. The first tag can be attached to the first device or at a location adjacent to the first device. In another embodiment, if the outline of the object being photographed is obtained using image processing without identifying the first device using tagging such as the first tag, and the outline corresponds to a predesigned 3D modeling, the corresponding device An augmented reality image of may be generated.

In one embodiment, the augmented reality image may function as an indicator displaying the current state of the first device. Here, the indicator may be a function of highlighting the augmented reality image of the first device. The indicator may highlight the augmented reality image by whether at least one real-time data matches the reference rule.

In this case, the reference rule may be a rule for highlighting the indicator when the data measured by the sensor is out of the normal range data set for each real-time data item. In one embodiment, the indicator may be a means for indicating a failure of the first equipment. For example, the indicator may highlight an augmented reality image when data related to a first object managed by the first device is out of normal range data or failure data of the first device is out of normal range data. In this case, real-time data sensed by various sensors may be dynamically reflected in the rendered augmented reality image. Here, the failure data refers to data that serves as a criterion for determining whether the first equipment has failed.

Here, the data related to the first object is data obtained by a first sensor that detects the first object, and the failure data of the first equipment is data related to the operation state of the first equipment obtained by the second sensor. Can be

In step S500, the pattern of the operation of the augmented reality image and the timing of the operation may be synchronized with the pattern of the actual operation of the first device and the timing of the operation. For example, assuming that the first equipment is a water pump, the augmented reality image may be rendered as an augmented reality image of the water pump according to 3D modeling of the water pump, which is designed in advance. Thereafter, the augmented reality image of the water pump can be moved according to a predesigned motion. Through this, the speed at which the water pump operates, the angle at which various parts move, and the amount of water discharged from the water pump are detected, and such real-time data can be reflected in 3D modeling. The pattern of the operation of the water pump and the timing of the operation are determined according to the real-time data, and the pattern and the timing of the operation of the augmented reality image of the water pump may be synchronized with the pattern of the actual operation of the water pump and the timing of the operation. .

The augmented reality-based smart management system according to an embodiment of the present invention can provide a more natural and realistic augmented reality by synchronizing the augmented reality image of the device with the operation of the actual device using real-time data. In addition, the augmented reality-based smart management system according to the present invention provides intuitive information on maintenance, management, and maintenance to the user of the smart management system because it displays which parts of the equipment currently operate and how they operate in the same manner as the actual operation. can do.

According to the present embodiment, in the augmented reality-based smart management method, a real-time image and real-time data are displayed to a user, and an augmented reality image of the first device is displayed together, and the augmented reality image is highlighted to perform an indicator function, It is possible to provide information for intuitively and easily recognizing whether the first equipment has failed or the status of a management location. For example, the user can intuitively recognize the device in question by viewing only the augmented reality image.

3 is a diagram for describing an example of displaying an augmented reality image of a first device and at least one or more real-time data related to the first device on a user terminal.

Referring to FIG. 3, an augmented reality image 20 and real-time data 30 and 40 of a first device may be displayed together in a real-time image 10 displayed on a user terminal. In FIG. 3, the first equipment may be a filter that fuses water and oxygen and puts it back into the water.

The augmented reality image 20 of the first device may be displayed overlaid on the real-time image 10. In addition, the augmented reality image 20 of the first device may be displayed overlaid in the same size and direction as the shape of the first device in an area where the first device is located in a real-time image.

If the first equipment is a filter, the data 30 and 40 related to the first object managed by the first equipment may refer to data related to water in a tank filtered by the filter. Specifically, the data 30 and 40 related to the first object may be displayed as data 30 related to oxygen and data 40 related to the degree of oxidation of water.

The oxygen-related data 30 can display the identifier 30a providing the dissolved oxygen in the water in the tank and the dissolved oxygen amount 30b, and a green circle indicating that the dissolved oxygen amount is normal can be displayed. That is, the first real-time data item 30 identified by the identifier 30a is dissolved oxygen in water in the tank, and the data 30b of the displayed first real-time data item is 54 mg/L.

The graph 30c displayed for the first real-time data item 30 corresponding to the preset normal range data may be displayed in green. On the other hand, the graph 40c displayed for the second real-time data item 40 out of the preset normal range data may be displayed in yellow. Colors may be expressed depending on the degree of deviation from the normal range data.

In the data 40 on the degree of oxidation of water, the hydrogen ion concentration of the water in the tank may be displayed in the size of the circle 40a, and whether or not it is normal may be indicated through the color 40b. In addition, the data 40 on the degree of oxidation of water may display a PH 40c and a concentration 40d. The data 40 on the degree of oxidation of water may be expressed as 72.7% when the normal standard data is set to 100% and the concentration measured by the sensor is expressed as a percentage (40d).

In addition, the concentration of individual ions dissolved in water, the temperature of the water in the water tank, etc. can also be displayed. In addition, the amount of dissolved oxygen in the plastic house may be displayed on a real-time image 10 displayed on the user terminal as shown in FIG. 4.

Referring to FIG. 5, all real-time data related to a management place may be displayed on the user terminal 200. Referring to FIG. 7, a hydrogen ion concentration 60 of water in a water tank, a dissolved oxygen amount 70 of water, and a nitrite ion concentration 80 of water may be simultaneously displayed on the user terminal 200. According to the present embodiment, the user can monitor all real-time data on the management place on one screen.

In an embodiment, in relation to the normal range data preset for each real-time data item, a category of the real-time data item may be set and displayed. In this case, the category 60e may be displayed in color. The hydrogen ion concentration 60 may be displayed in a category 60e. In addition, the amount of dissolved oxygen 70 of water and the concentration of nitrite ions 80 of water may also be divided into categories and displayed. According to the present embodiment, by dividing the data of the real-time data item into categories, the user can easily determine whether the real-time data item is normal. In addition, the real-time data may indicate 52 components such as nitrate, ammonia, magnesium, calcium, potassium, and sodium, but is not limited thereto.

According to an embodiment of the present disclosure, a method of displaying a real-time data item as a graph may display a degree of deviation in color when the data is out of a preset normal range data related to the real-time data item. In addition, normal standard data may be set to 100%, and data of real-time data items measured by a sensor may be displayed as a percentage.

In one embodiment, it is determined that a real-time data item with a high priority is displayed in the first area, but the priority is given priority as the degree to which the data measured by the sensor deviates from the normal range data previously set for each real-time data item is greater. It may be that the ranking is determined high. That is, when the first equipment is a malfunctioning equipment or an object managed by the first equipment is defective, the user can intuitively determine whether the equipment is malfunctioning or the object is defective by displaying the related data in the first area where the user can easily recognize. Can be recognized as. According to the present embodiment, by determining a real-time data item having a high degree of deviation from the normal range data as the highest priority, the user can recognize the problem real-time data and the user can quickly take a corresponding action.

6 is a flow chart for explaining in more detail some steps described with reference to FIG. 2, and FIG. 7 is a view for explaining an example in which some defective parts are highlighted according to FIG. 6 to perform an indicator function.

Referring to FIG. 6, steps S410 and S420 may be performed in step S400, and steps S510 and S520 may be performed in step S500.

Specifically, when real-time data is displayed together with the augmented reality image of the first device in step S400, the augmented reality image of the first device is decomposed into parts of the first device in step S410 so that each part can be displayed at a distance. have. Among the parts disassembled in step S420, a part violating the standard rule is determined through real-time data, and the part violating the standard rule may be highlighted.

As shown in FIG. 7, highlighting may be displayed on some defective parts of the augmented reality image. The indicator 20a may be highlighted in red and displayed on the faulty part of the first equipment. That is, a partial area of the augmented reality image may function as an indicator. Here, the failure site may be determined using failure data obtained from the second sensor installed in the first equipment. Here, when the fault data is out of a preset normal range, it may be determined that a fault has occurred.

Referring back to FIG. 6, in step S500, the operation pattern and the operation timing of the augmented reality image may be synchronized with the actual operation pattern and operation timing of the first device. In step S510, at least one real-time data Using this, the pattern of the operation of the part violating the reference rule and the timing of the operation may be analyzed. Subsequently, in step S520, a pattern of an operation of a component violating the reference rule and a timing of the operation may be rendered in the augmented reality image of the first device.

According to the present embodiment, the augmented reality-based smart management method can display a structure in a state in which a broken component is disassembled, and display how the broken component is operating. Accordingly, the user can intuitively and easily grasp which parts are broken and how even by looking only at the rendered augmented reality image.

8 is a diagram for explaining a UI displayed on a user terminal that can be referred to in some embodiments of the present invention.

Referring to FIG. 8, a monitoring button 51, an AR view button 53, an A/S button 55, and an option button 57 may be displayed on the smart management application of the user terminal 200.

The monitoring button 51 is a button that displays the status of equipment installed in the management place using real-time data, and the AR view button 53 may be a button that converts to a shooting screen and renders an augmented reality image when a tag is identified. .

The A/S button 55 may be a button for performing A/S related to a faulty part of the first equipment. When the A/S button is input, a corresponding signal may be output according to the degree of failure data. Here, the corresponding signal may be classified according to the degree of failure data.

The option button 57 may be a button for changing various settings of a smart management application and changing the quality of an augmented reality image, a monitoring period, and the like.

9 is a flow chart of a smart management method based on an augmented reality according to another embodiment of the present invention, and FIG. 10 is a diagram for explaining a remote assistance indicator displayed on a real-time image of a first device.

Referring to FIG. 9, the augmented reality-based smart management method according to another embodiment of the present invention may include operations of steps S600 and S700 additionally performed in the augmented reality-based smart management method according to the above-described embodiment. . Accordingly, the description of the redundant steps will be omitted and different operations will be mainly described.

When an A/S button for performing A/S associated with a faulty part of the first equipment is input, maintenance may be requested to a repair server that maintains the first equipment.

Specifically, an A/S button associated with the augmented reality image of the first equipment highlighted in step S600 is displayed, and in step S700, maintenance may be requested to a repair server that maintains the first equipment.

Subsequently, a first remote support indicator for self-repairing the part highlighted in step S710 may be received from the repair server. At this time, the stage of failure may be classified into a first level and a second level according to a level in which the first equipment violates the standard rule, and the first level is a stage determined as a case where self-repair is possible without replacement of parts, The second level may be a step in which a replacement of a component is required. When the stage of the failure is the first level according to the level in which the first equipment violates the reference rule, a first remote support indicator may be received from the repair server.

In step S720, a first first remote assistance indicator may be displayed on a real-time image of the first device. As shown in FIG. 10, the first remote assistance indicator may be formed of a shape including a color indicating the position of the button.

Referring back to FIG. 9, the remote support indicator may guide an operation capable of separating, detaching, or repairing parts of the equipment while steps S730 to S750 are performed.

In operation S730, the first first remote assistance indicator and the operation displayed on the real-time image of the first device may be compared. Thereafter, in step S740, if it is determined that the first equipment has actually moved according to the first remote support indicator, operation completion information may be transmitted to the repair server. In step S750, a second remote support indicator may be received from the repair server in response to the operation completion information.

For a specific example, in the case of including a first step of turning off the power and a second step of turning on the power to repair a specific part of the filter of the farm, the first step on the button for turning off the power of the augmented reality image of the filter A remote assistance indicator may be displayed. As it is determined that the user has input the button for turning off the power indicated by the first remote assistance indicator, the second remote assistance indicator is displayed on the button for turning on the power among the augmented reality images of the filter corresponding to the subsequent second step. I can.

The augmented reality-based smart management method according to an embodiment of the present invention may guide the user to operate the equipment according to the remote support indicator while repeatedly performing the steps S730 to S750, and even in the case of a repair operation of high difficulty, It is possible to provide information that allows inexperienced users to easily maintain, repair and repair equipment.

In another embodiment, information in which parts related to the first equipment are automatically ordered may be transmitted to the repair server when the stage of failure is the second level according to the level in which the first equipment violates the reference rule.

In the augmented reality-based smart management method according to an embodiment of the present invention, in the case of a malfunction of equipment that is only a level that the user can repair by himself, the remote support indicator displayed on the equipment is viewed without calling a remote expert, Users can assist themselves in repairing equipment. Furthermore, in the case of a failure requiring replacement of consumables, an order of consumables is automatically made so that user convenience may be further increased.

In an augmented reality-based smart management method according to an embodiment of the present invention, a real-time image of a management place is displayed, an augmented reality image of a device existing in the displayed image is displayed, and an object managed by the device and the device itself By displaying real-time data of the user, the user can intuitively recognize comprehensive information related to the management place.

In addition, the augmented reality-based smart management method according to an embodiment of the present invention can be generated in the management location by dynamically outputting a signal corresponding to the failure location and degree when a failure of equipment existing in the management location occurs. You can quickly deal with the problem of equipment failure. Convenience of a user controlling a management location may be increased.

The augmented reality-based smart management method according to another embodiment of the present invention may determine the number of faulty equipment using current consumption.

The smart management device can classify equipment of the same type for which the equipment's deterioration state information corresponds to a predefined determination condition, and obtain the rated current of the equipment of the same type. The smart management device may obtain the current supplied to the equipment from the information of each equipment, and obtain the current consumption of one equipment.

The smart management device can determine the number of faulty equipment by dividing the difference between the total rated current of the same type of equipment and the current supplied to the actual equipment by the current consumption of one equipment.

The predefined discrimination condition is that the change of current according to the power supplied to the equipment when the same type of equipment is operated, and the change of current according to the power supplied to the equipment that actually failed or deteriorated in the same type of equipment in the past. If it is similar to any one within a predefined error range, it may be a condition for determining that there is an abnormality according to the deterioration state.

The operation of analyzing the deterioration state information by the smart management device may include an operation of analyzing a change trend of current according to power supplied to the equipment when the equipment is driven, and applying the analyzed change trend to a predefined determination condition.

In another embodiment, the smart management device obtains the number of equipment of the same type, and defines the total rated current of the same equipment and the total current supplied to the same equipment divided by the current consumption of one equipment as the number of possible failures of the equipment. And, based on the ratio of the number of failure candidates and the number of equipment, it is possible to calculate the expected noise of the current supplied to the equipment. The smart management device determines the number of fault candidates as the number of faulty equipment when the error between the measured noise and the predicted noise of the current supplied to the device is within a predefined error range, and when the error exceeds the predefined error range. , It is possible to determine the number of equipment that is a failure of the equipment by subtracting from the number of failure candidates as many as an integer rounded off the value obtained by dividing the error by a predefined unit error. In this case, the smart management device may determine twice the integer as the number of equipment that is degraded in performance.

Up to now, an augmented reality-based smart management method and an application field thereof according to an embodiment of the present invention have been described with reference to FIGS. 1 to 10. Hereinafter, a computing device 1500 capable of implementing the augmented reality-based smart management device 100 according to an embodiment of the present invention may be described.

11 is a hardware configuration diagram illustrating an exemplary computing device 1500 capable of implementing the augmented reality-based smart management device 100 according to an embodiment of the present invention.

As shown in FIG. 11, the computing device 1500 loads one or more processors 1510, a bus 1550, a communication interface 1570, and a computer program 1591 executed by the processor 1510. The memory 1530 may include a memory 1530 and a storage 1590 for storing the computer program 1591. However, only the components related to the embodiment of the present invention are shown in FIG. 11. Accordingly, those of ordinary skill in the art to which the present invention belongs may recognize that other general-purpose components may be further included in addition to the components illustrated in FIG. 11.

The processor 1510 may control the overall operation of each component of the computing device 1500. The processor 1510 includes a CPU (Central Processing Unit), MPU (Micro Processor Unit), MCU (Micro Controller Unit), GPU (Graphic Processing Unit), or any type of processor well known in the technical field of the present invention. Can be. In addition, the processor 1510 may perform an operation on at least one application or program for executing the method according to the embodiments of the present invention. The computing device 1500 may include one or more processors.

The memory 1530 may store various types of data, commands, and/or information. The memory 1530 may load one or more programs 1591 from the storage 1590 in order to execute the method according to the embodiments of the present invention. The memory 1530 may be implemented as a volatile memory such as RAM, but the technical scope of the present invention is not limited thereto.

The bus 1550 may provide a communication function between components of the computing device 1500. The bus 1550 may be implemented as various types of buses such as an address bus, a data bus, and a control bus.

The communication interface 1570 may support wired/wireless Internet communication of the computing device 1500. Also, the communication interface 1570 may support various communication methods other than Internet communication. To this end, the communication interface 1570 may be configured to include a communication module well known in the technical field of the present invention.

According to some embodiments, the communication interface 1570 may be omitted.

The storage 1590 may non-temporarily store the one or more programs 1591 and various data.

The storage 1590 is a nonvolatile memory such as a ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), flash memory, etc., a hard disk, a removable disk, or well in the technical field to which the present invention belongs. It may be configured to include any known computer-readable recording medium.

The computer program 1591 may include one or more instructions that when loaded into the memory 1530 cause the processor 1510 to perform a method/operation in accordance with various embodiments of the present invention. That is, the processor 1510 may perform methods/operations according to various embodiments of the present disclosure by executing the one or more instructions.

In the above case, the augmented reality-based smart management device 100 according to an embodiment of the present invention may be implemented through the computing device 1500.

So far, various embodiments of the present invention and effects according to the embodiments have been mentioned with reference to FIGS. 1 to 11. The effects according to the technical idea of the present invention are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

The technical idea of the present invention described with reference to FIGS. 1 to 11 so far may be implemented as computer-readable codes on a computer-readable medium. The computer-readable recording medium is, for example, a removable recording medium (CD, DVD, Blu-ray disk, USB storage device, removable hard disk) or a fixed recording medium (ROM, RAM, computer-equipped hard disk). I can. The computer program recorded in the computer-readable recording medium may be transmitted to another computing device through a network such as the Internet and installed in the other computing device, thereby being used in the other computing device.

In the above, the technical idea of the present invention is not necessarily limited to these embodiments, even though it has been described that all the constituent elements constituting the embodiments of the present invention are combined into one or operated in combination. That is, as long as it is within the scope of the object of the present invention, one or more of the components may be selectively combined and operated.

Although the operations are illustrated in a specific order in the drawings, it should not be understood that the operations must be executed in the specific order shown or in a sequential order, or all illustrated operations must be executed to obtain a desired result. In certain situations, multitasking and parallel processing may be advantageous. Moreover, the separation of the various components in the above-described embodiments should not be understood as necessitating such separation, and the program components and systems described may generally be integrated together into a single software product or packaged into multiple software products. It should be understood that there is.

Although the embodiments of the present invention have been described above with reference to the accompanying drawings, those of ordinary skill in the art to which the present invention pertains may implement the present invention in other specific forms without changing the technical idea or essential features. I can understand that there is. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. The scope of protection of the present invention should be interpreted by the claims below, and all technical ideas within the scope equivalent thereto should be interpreted as being included in the scope of the technical ideas defined by the present invention.

Claims (3)

In the method performed by the computing device,
A step of photographing a management location by a user terminal; And
An augmented reality image representing a predesigned 3D modeling of the first equipment when the first equipment installed in the management location is photographed by the user terminal and a first tag corresponding to the first equipment is identified by the user terminal Rendering the;
Displaying at least one real-time data related to the first device together with an augmented reality image of the first device; And
As the at least one real-time data is reflected in the pre-designed 3D modeling of the first device, synchronizing the operation pattern and the operation timing of the augmented reality image with the actual operation pattern and operation timing of the first equipment Including steps,
The real-time data,
Including data related to a first object managed by the first device and failure data of the first device, wherein the data related to the first object is data obtained by a first sensor that detects the first object , The failure data of the first equipment is data related to the operation state of the first equipment obtained by a second sensor,
The augmented reality image of the first device,
Performs the function of an indicator for displaying the current state of the first device through the augmented reality image, wherein the indicator displays an augmented reality image of the first device according to whether the at least one real-time data matches a reference rule. Highlighting,
The above standard rules are:
If the data measured by the second sensor is out of the normal range data preset for each item of the real-time data, this is a rule for highlighting the augmented reality image of the first device,
Displaying at least one or more real-time data related to the first device together with an augmented reality image of the first device,
Disassembling the augmented reality image of the first equipment into parts of the first equipment, including displaying each part apart; And
Highlighting a part violating the standard rule among the disassembled parts,
As the at least one real-time data is reflected in the pre-designed 3D modeling of the first device, synchronizing the operation pattern and the operation timing of the augmented reality image with the actual operation pattern and operation timing of the first equipment The steps are,
Analyzing an operation pattern and an operation timing of a part violating the reference rule by using at least one real-time data; And
Including the step of rendering the pattern of the operation of the part violating the reference rule and the timing of the operation in the augmented reality image of the first device,
Smart management method based on augmented reality. delete The method of claim 1,
Displaying an A/S button associated with the highlighted augmented reality image of the first device; And
When the A/S button is input, further comprising the step of requesting maintenance to a repair server that maintains the first equipment,
The step of requesting maintenance to the repair server,
Receiving, from the repair server, a remote assistance indicator for self-repairing the highlighted part in the case of a first level by the first equipment according to a level in violation of the reference rule;
Displaying the remote assistance indicator on a real-time image of the first device;
Comparing an operation displayed on a real-time image of the first device with the remote assistance indicator;
Transmitting operation completion information to the repair server when it is determined that the first equipment has actually moved according to the remote assistance indicator; And
Receiving a second remote assistance indicator from the repair server in response to the operation completion information,
Smart management method based on augmented reality.

Images