JP6970774B2 - Mobile terminals, methods and programs - Google Patents

Description

The present invention relates to mobile terminals, methods and programs.

Japanese Unexamined Patent Publication No. 2014-164482 (Patent Document 1) discloses a head mounted display (HMD). This head-mounted display is used at the time of maintenance of a substrate processing apparatus that processes a substrate such as a semiconductor wafer. This head-mounted display displays support information related to maintenance. The support information is displayed superimposed on the real space as augmented reality (AR). The operator can perform the maintenance work while referring to the support information displayed on the head-mounted display (see Patent Document 1).

Japanese Unexamined Patent Publication No. 2014-164482

However, depending on the work content of the user (worker), an AR image such as support information displayed on a mobile terminal such as a head-mounted display may be an obstacle.

The present invention has been made to solve such a problem, and an object of the present invention is to provide a mobile terminal, a method, and a program that are easy to use regardless of the work content of the user.

A mobile terminal according to a certain aspect of the present invention includes a display unit, a control unit, and a reception unit. The display unit is configured to display a first AR image showing the inside of a predetermined portion of the industrial equipment. The control unit is configured to control the display unit. The reception unit is configured to accept the selection of either the first mode or the second mode. The control unit controls the display unit so that the first AR image is displayed when the first mode is selected, while the display unit hides the first AR image when the second mode is selected. Control. When the first mode is selected, at least a part of the first AR image is displayed at a position overlapping the predetermined portion on the display unit.

In this mobile terminal, at least a part of the first AR image is displayed at a position overlapping a predetermined part of the industrial device on the display unit. Depending on the work content of the user, it may be convenient that the first AR image is not displayed. In this mobile terminal, it is possible to switch between the first mode in which the first AR image is displayed and the second mode in which the first AR image is hidden. Therefore, according to this mobile terminal, the user can appropriately switch whether or not to display the first AR image on the display unit according to his / her current work content.

Further, the mobile terminal further includes a detection unit configured to detect a value related to the operation of the user of the mobile terminal, and the control unit determines whether or not the user is operating based on the detection result of the detection unit. It is configured to make a determination, and the control unit controls the display unit to display the first AR image when it is determined that the user is not operating when the first mode is selected. When it is determined that the user is operating, the display unit may be controlled so as to hide the first AR image.

When the first AR image is displayed on the display unit, the first AR image covers a predetermined portion of the industrial equipment. As a result, the area covered by the first AR image becomes difficult for the user to see. If the first AR image is continuously displayed on the display unit even during the operation of the user, a situation may occur in which the user unintentionally touches a surrounding object. In this mobile terminal, when it is determined that the user is operating, the first AR image is not displayed on the display unit. Therefore, according to this mobile terminal, it is possible to suppress the occurrence of a situation in which the user unintentionally comes into contact with a surrounding object.

Further, the mobile terminal further includes a communication unit configured to communicate with the industrial equipment, and the control unit acquires operation state information indicating the current operation state of the industrial equipment via the communication unit. It may be configured in.

According to this mobile terminal, the mobile terminal can accurately grasp the current operating state of the industrial device.

Further, in the mobile terminal, the control unit may be configured to generate a first AR image based on the operating state information.

According to this mobile terminal, a first AR image showing an internal operating state of an industrial device can be displayed on a display unit. As a result, the user can accurately grasp the internal state of the industrial equipment by visually recognizing the first AR image.

Further, in the mobile terminal, the display unit is configured to further display a second AR image showing the correct operating state of the industrial device, and the control unit is when the first AR image is displayed on the display unit. , The display unit may be controlled so that the second AR image is displayed side by side with the first AR image.

In this mobile terminal, the first AR image and the second AR image are displayed side by side. Therefore, the user can determine whether or not the current operation of the industrial equipment is correct by comparing the first AR image and the second AR image.

Further, the mobile terminal may be a head-mounted display.

According to this mobile terminal, the user can perform various tasks without blocking both hands.

Further, a method according to another aspect of the present invention is a method of displaying an AR image showing the inside of a predetermined portion of an industrial device on a display unit of a mobile terminal. This method has a step of accepting the selection of either the first mode or the second mode, a step of displaying an AR image on the display unit when the first mode is selected, and a step when the second mode is selected. Includes a step to hide the AR image. When the first mode is selected, at least a part of the AR image is displayed at a position overlapping a predetermined part on the display unit.

According to this method, it is possible to appropriately switch whether or not to display the first AR image on the display unit according to the current work content of the user.

Further, a program according to another aspect of the present invention is a program for causing a computer to execute a process of displaying an AR image showing the inside of a predetermined portion of an industrial device on a display unit of a mobile terminal. This program has a step of accepting the selection of either the first mode or the second mode, a step of displaying an AR image on the display unit when the first mode is selected, and a step when the second mode is selected. Have the computer perform the steps to hide the AR image. When the first mode is selected, at least a part of the AR image is displayed at a position overlapping a predetermined part on the display unit.

According to this program, it is possible to appropriately switch whether or not to display the first AR image on the display unit according to the current work content of the user.

According to the present invention, it is possible to provide a mobile terminal, a method and a program that are easy to use regardless of the work content of the user.

It is a figure including the figure which shows the plane which shows the plane when the industrial equipment is seen from the top, and the figure which shows the side surface of a part of the industrial equipment schematically. It is a figure which shows an example of 3D AR image displayed in HMD. It is a figure which shows typically the system including HMD. It is a figure which shows the hardware structure of HMD schematically. It is a figure which shows the hardware configuration of a server schematically. It is a sequence diagram which shows the operation which HMD acquires the operation state information from the industrial equipment. It is a figure which shows typically the 1st display example of the display part in the state which communication between HMD and industrial equipment is established. It is a flowchart which shows the mode setting processing procedure about display of the 1st AR image in HMD. It is a flowchart which shows the processing procedure which is executed first after the setting mode is changed to the display mode. It is a flowchart which shows the processing procedure executed in the display mode. It is a figure which shows typically the 2nd display example of the display part in the state which communication between HMD and industrial equipment is established. It is a flowchart which shows the switching processing procedure of the display state of the 2nd AR image which shows the normal operation state of the part corresponding to the 1st AR image. It is a flowchart which shows the display processing procedure of the 2nd AR image executed in S52 of FIG.

Hereinafter, an embodiment according to one aspect of the present invention (hereinafter, also referred to as “the present embodiment”) will be described in detail with reference to the drawings. The same or corresponding parts in the drawings are designated by the same reference numerals and the description thereof will not be repeated. Moreover, the present embodiment described below is merely an example of the present invention in all respects. The present embodiment can be variously improved or modified within the scope of the present invention. That is, in carrying out the present invention, a specific configuration can be appropriately adopted according to the embodiment.

[1. Overview]
FIG. 1 is a diagram including a diagram schematically showing a plane when the industrial equipment 30 is viewed from above, and a diagram schematically showing a side surface of a part of the industrial equipment 30. In the present embodiment, the industrial equipment 30 is a semiconductor manufacturing apparatus. The industrial device 30 does not necessarily have to be a semiconductor manufacturing device. The industrial equipment 30 may be, for example, another substrate processing apparatus such as a MEMS (Micro Electro Mechanical Systems) manufacturing apparatus or a liquid crystal panel manufacturing apparatus. Further, the industrial equipment 30 may be, for example, a reaction tank of a chemical plant, a distillation column, a heat exchanger, a pipe thereof, or a mold for an injection molding device. Further, the industrial device 30 may be a device having a power mechanism section, in which at least a part of the power mechanism section is covered, and the operation of the power mechanism section cannot be visually recognized from the outside of the device. ..

The industrial equipment 30 includes a vacuum transfer chamber 5 having a substantially hexagonal shape in a plan view, two load locks 3 each having a substrate storage unit 4 capable of storing a plurality of boards, and two vacuum chambers 31. Each load lock 3 and each vacuum chamber 31 are connected to a vacuum transfer chamber 5 via a gate valve 7 that can be opened and closed. The vacuum transfer chamber 5 includes an alignment unit 2 for adjusting the position of the substrate and a vacuum robot 6. The vacuum robot 6 transfers the substrate between the substrate accommodating portion 4 and the vacuum chamber 31 by rotating and expanding / contracting.

In the vacuum chamber 31, the semiconductor wafer (substrate) W1 is subjected to a process such as etching. A stage 34 is provided in the vacuum chamber 31. Further, a viewing window 32 is formed in the vacuum chamber 31. The transfer mechanism 33 is a part of the vacuum robot 6. The semiconductor wafer W1 is conveyed onto the stage 34 by the transfer mechanism 33.

The stage 34 includes a stage body 34A and three lift pins 34B. The three lift pins 34B are arranged on the same circumference in a plan view. The transport mechanism 33 includes a transport arm 33A and a fork 33B. For example, the transfer arm 33A arranges the semiconductor wafer W1 on the fork 33B on the lift pin 34B. The lift pin 34B is housed in the stage main body 34A after the transport mechanism 33 is retracted. As a result, the semiconductor wafer W1 is placed on the stage 34.

The viewing window 32 is, for example, a window for checking the inside of the vacuum chamber 31, and is made of, for example, quartz or sapphire glass. A maintenance worker of the industrial equipment 30 (hereinafter, also simply referred to as a “worker”) confirms the state inside the vacuum chamber 31 through, for example, the viewing window 32. However, it is difficult to increase the size of the viewing window 32 from the viewpoint of the strength required for the viewing window 32 and the arrangement of each component in the vacuum chamber 31. Therefore, the operator cannot sufficiently grasp the state inside the vacuum chamber 31 through the viewing window 32. In FIG. 1, the viewing window 32 is located slightly above the semiconductor wafer W1, but the position of the viewing window 32 is not limited to this.

In the present embodiment, an operator (an example of a user) performs maintenance work on the industrial equipment 30 with the HMD 10 included in the system 1 described later attached. Inside the industrial equipment 30, a certain space exists as in the case of the other industrial equipment described above. In the space, for example, the transport mechanism 33 and the stage 34 operate. The inside of the space is a part that the operator wants to visually check for maintenance work or the like. In the HMD 10, for example, an AR image showing the inside of a predetermined portion P1 of the industrial equipment 30 is superimposed and displayed on the industrial equipment 30. The AR image showing the inside of the predetermined portion P1 in FIG. 1 is represented in a plane (2D) for easy understanding, but is actually a three-dimensional (3D). This also applies to FIGS. 7 and 11 described later. However, the AR image displayed on the HMD 10 does not necessarily have to be a 3D image, and may be a 2D image.

The operator can fully grasp the state in the industrial equipment 30 by referring to the AR image displayed on the HMD 10. Examples of the inside of the industrial equipment 30 shown by the AR image include various parts included in the vacuum chamber 31 and the inner wall of the vacuum chamber 31.

FIG. 2 is a diagram showing an example of a 3D AR image displayed on the HMD 10. The AR image shown in FIG. 2 shows a certain state inside the vacuum chamber 31. This AR image shows the stage 34 existing in the vacuum chamber 31, the inner wall I1 of the vacuum chamber 31, and the like. The portion T1 indicates the thickness of the wall of the vacuum chamber 31, and the ridge line L1 indicates the ridge line on the worker side formed by each surface (side surface and upper surface) of the inner wall I1 of the vacuum chamber 31. Since the ridge line L1 is a portion in the vacuum chamber 31 that cannot be seen at the time of display, it is displayed as a dotted line, for example. In FIG. 2, some configurations of the gate valve 7 and the like are omitted for ease of understanding. For example, the AR image showing the inside of the vacuum chamber 31 does not have to completely represent all the parts actually existing in the vacuum chamber 31 as an AR image.

As shown in FIG. 2, when the AR image is a 3D image, for example, when the operator views the actual vacuum chamber 31 from diagonally above, the operator can see the stage existing in the vacuum chamber 31. The AR image showing 34 mag can be seen from diagonally above. That is, by looking at the AR image displayed on the HMD 10, the operator has the illusion that, for example, the AR image showing the stage 34 exists at the originally existing position in the vacuum chamber 31. When the AR image showing the stage 34 or the like is viewed from diagonally above, the operator can recognize the three-dimensional structure of the stage 34 or the like. Further, for example, by displaying a 3D image of the inner wall I1 of the vacuum chamber 31 in addition to the parts such as the stage 34, the operator can perform the parts such as the stage 34 in addition to the three-dimensional structure of the parts such as the stage 34. The positional relationship between and the inner wall I1 can also be recognized. For example, the distance between the parts such as the stage 34 and the inner wall I1 on the back side can be examined by using a means for calculating the dimensions of the AR image such as an AR measure.

However, depending on the work content of the worker, the AR image displayed on the HMD 10 may be an obstacle. That is, depending on the work content of the user, it may be convenient that the AR image is not displayed. In the HMD 10, it is possible to switch between a display mode for displaying an AR image showing the state inside the industrial equipment 30 and a non-display mode for hiding the AR image showing the state inside the industrial equipment 30. Therefore, according to the HMD 10, the user can appropriately switch whether or not to display the AR image on the display unit according to his / her current work content. Hereinafter, the system 1 including the HMD 10 according to the present embodiment will be described in detail.

[2. composition]
<2-1. Overall system configuration>
FIG. 3 is a diagram schematically showing a system 1 including the HMD 10 according to the present embodiment. As shown in FIG. 3, the system 1 includes an HMD 10, a server 20, and an industrial device 30. In the system 1, the HMD 10, the server 20, and the industrial equipment 30 can communicate with each other via the network N1.

HMD10 is, for example, HoloLens® manufactured by Microsoft Corporation and has an augmented reality function. The HMD 10 requests the server 20 for information about the industrial equipment 30 that the worker is visually recognizing through the HMD 10, and receives the information from the server 20. Further, the HMD 10 continuously receives the operating state information indicating the current operating state of the industrial device 30 from the industrial device 30. In the HMD 10, an AR image showing the state in the industrial equipment 30 is displayed based on the information received from the server 20 and the industrial equipment 30.

<2-2. HMD hardware configuration>
FIG. 4 is a diagram schematically showing a hardware configuration of the HMD 10. As shown in FIG. 4, the HMD 10 includes a control unit 11, a camera 13, a display unit 14, various sensors 15, a communication unit (communication I / F) 16, and a storage unit 12, and each configuration thereof is included. Is electrically connected via a bus.

The control unit 11 includes a CPU (Central Processing Unit) 11A, a RAM (Random Access Memory) 11B, a ROM (Read Only Memory) 11C, and the like, and is configured to control each component according to information processing. ..

The camera 13 is composed of a camera module including an image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The camera 13 captures, for example, a moving image corresponding to the line of sight of the operator.

The display unit 14 includes, for example, a display such as a liquid crystal display or an organic EL (Electro Luminescence) display. If the HMD 10 is a so-called optical see-through type, the display unit 14 further includes an optical system such as a half mirror. On the other hand, if the HMD 10 is a video see-through type, the display unit 14 is composed of a display. The shape of the display unit 14 is the shape of a lens of glasses. The display unit 14 displays, for example, various AR images. Regarding the information in the real world, the display unit 14 may display a moving image taken by the camera 13 (video see-through type) or may transmit the information in the real world (optical see-through type).

The various sensors 15 include, for example, a distance image sensor, a speed sensor, an acceleration sensor, and a gyro sensor. In addition, the various sensors 15 may include, for example, a geomagnetic sensor (direction sensor). The distance image sensor is configured to detect the distance between the HMD 10 and the industrial equipment 30. The speed sensor is configured to detect the speed of the HMD 10. The accelerometer is configured to detect the acceleration of the HMD 10. The gyro sensor is configured to detect the angular velocity of the HMD 10. For example, it can be said that each of the speed sensor, the acceleration sensor, and the gyro sensor included in the various sensors 15 detects the value related to the operation of the user of the HMD 10.

The communication unit 16 is configured to communicate with an external device (for example, a server 20 and an industrial device 30) provided outside the HMD 10 via the network N1. The communication unit 16 is composed of, for example, a wired LAN (Local Area Network) module or a wireless LAN module.

The storage unit 12 is an auxiliary storage device such as a hard disk drive or a solid state drive. The storage unit 12 is configured to store, for example, the control program 121. By executing the control program 12A by the control unit 11, various controls of the HMD 10 are performed.

The control unit 11 is configured to perform SLAM (Simultaneous Localization And Mapping), for example. That is, the control unit 11 can perform spatial recognition and self-position estimation by using the captured image by the camera 13 and the detection results of various sensors 15.

For example, with the user wearing the HMD 10, the distance image sensor measures the distance to a plurality of feature points in the surrounding environment, and the control unit 11 spatially recognizes the distance based on the measurement result. After that, when the user moves the viewpoint while wearing the HMD 10, the acceleration sensor, the gyro sensor, and the like detect the movement amount and the movement direction, and the control unit 11 estimates the position of the HMD 10 (self-position estimation) based on the detection result. To do. The control unit 11 further recognizes the space of the surrounding environment. The control unit 11 can always grasp the three-dimensional space by repeating these space recognition and self-position estimation.

Further, the control unit 11 is configured to receive input from the operator through, for example, a gesture. For example, when a selection palette containing a plurality of options is displayed on the display unit 14, and a gesture for selecting one of the options is recognized by the camera 13 or various sensors 15 (for example, a distance image sensor), the control unit 11 executes the control corresponding to the selected option. That is, in the present embodiment, the camera 13 for recognizing gestures and various sensors 15 function as the "reception unit" in the present invention. Other functions realized by the control unit 11 will be described in detail later.

<2-3. Server hardware configuration>
FIG. 5 is a diagram schematically showing a hardware configuration of the server 20. In the present embodiment, the server 20 is realized by, for example, a general-purpose computer.

As shown in FIG. 5, the server 20 includes a control unit 21, a communication unit (communication I / F) 23, and a storage unit 22, and each configuration is electrically connected via a bus. ..

The control unit 21 includes a CPU 21A, a RAM 21B, a ROM 21C, and the like, and is configured to control each component according to information processing.

The communication unit 23 is configured to communicate with an external device (for example, HMD 10) provided outside the server 20 via the network N1. The communication unit 23 is composed of, for example, a wired LAN module or a wireless LAN module.

The storage unit 22 is an auxiliary storage device such as a hard disk drive or a solid state drive. The storage unit 22 is configured to store, for example, the control program 22A and the databases D1 and D2.

The database D1 manages, for example, the three-dimensional shape of each component (for example, the transport mechanism 33 and the stage 34) and the position information of each component for each of the plurality of industrial devices 30. More specifically, the database D1 manages the reference point of each industrial device 30 and the three-dimensional shape and position information of each component included in the industrial device 30 with respect to the reference point. In the present embodiment, the reference point is a reference point in the three-dimensional space for superimposing the AR image on the real space, and is recognized after grasping the three-dimensional space.

An individual AR marker is attached to each industrial device 30, and the AR marker indicates a reference point of the industrial device 30. The database D1 is included in, for example, a) AR marker shape data for each industrial device 30, b) 3D drawing data of each industrial device 30 including coordinate information based on a reference point, and c) industrial device 30. It manages the coordinate information with respect to the reference point of the origin of each drive unit.

The database D2 manages, for example, information on the normal operation of each component for each of the plurality of industrial devices 30.

[3. motion]
<3-1. Acquisition of operation status information from industrial equipment Operation>
FIG. 6 is a sequence diagram showing an operation in which the HMD 10 acquires operation state information from the industrial equipment 30. The control unit 11 performs space recognition and self-position estimation by using the captured image by the camera 13 and the detection results of various sensors 15, and subsequently continuously grasps the three-dimensional space (S1). In this case, for example, an AR marker (not shown) attached to a fixed position of the industrial equipment 30 is photographed by the camera 13. The control unit 11 identifies the industrial equipment 30 existing in front of the worker based on the AR marker photographed by the camera 13 (S2).

When the control unit 11 recognizes the positional relationship between the industrial device 30 and its reference point, the control unit 11 executes a process for establishing communication between the HMD 10 and the industrial device 30 (S3). ). When the communication between the HMD 10 and the industrial device 30 is established, the transmission of the operation state information from the industrial device 30 to the HMD 10 is started (S4). The operation state information is continuously transmitted at predetermined time intervals. The operating state information includes the position information of each component in the industrial equipment 30. The control unit 11 can recognize the current position of each component in the industrial equipment 30 by referring to the acquired operation state information. That is, the control unit 11 can recognize the operation of each component in the industrial equipment 30 by referring to the operation state information acquired at predetermined time intervals.

<3-2. Operation related to the display of AR images showing the inside of industrial equipment>
FIG. 7 is a diagram schematically showing a first display example of the display unit 14 in a state where communication between the HMD 10 and the industrial equipment 30 is established. As shown in FIG. 7, options 17A and 17B are displayed on the display unit 14.

Option 17A is a UI (User Interface) for switching the mode related to the display of the first AR image A1 in the HMD 10. The first AR image A1 is an AR image showing the inside of a predetermined portion of the industrial equipment 30. In the HMD 10, the first AR image A1 corresponding to a predetermined portion existing in front of the line of sight of the operator is displayed. In FIG. 7, the industrial equipment 30 is omitted. By looking at the first AR image A1 displayed on the HMD 10, the operator can see, for example, as if an AR image showing each component in the industrial equipment 30 exists at the originally existing position in the industrial equipment 30. I have an illusion. It should be noted that the display area of the first AR image A1 is slightly deviated from the area (hereinafter, also referred to as “predetermined area”) in which the operator has the illusion that the first AR image A1 exists in the area inside the vacuum chamber 31 that should originally exist. There is. For example, the display area of at least a part of the first AR image A1 may overlap with a predetermined area. Further, for example, the first AR image A1 may be enlarged and displayed more than it should be. That is, the first AR image A1 may be enlarged and displayed with the central portion of the predetermined region as the center. The center of the enlargement of the first AR image A1 may be slightly deviated from the central portion of the predetermined region. Even in this case, it is sufficient that the display area of at least a part of the enlarged first AR image A1 overlaps the predetermined area. The "predetermined area" overlaps with the position where the predetermined portion of the industrial equipment 30 can be seen as a planar position on the display unit 14. The HMD 10 is provided with a "display mode" for displaying the first AR image A1 and a "non-display mode" for hiding the first AR image A1.

Option 17B is a UI for switching the display state of the second AR image A2 (described later) showing the normal operating state of the component corresponding to the first AR image A1. The switching of the display state of the second AR image A2 will be described in detail later.

FIG. 8 is a flowchart showing a mode setting processing procedure regarding the display of the first AR image A1 in the HMD 10. The process shown in this flowchart is repeatedly executed by the control unit 11 in a state where communication between the HMD 10 and the industrial device 30 is established.

The control unit 11 determines whether or not the operator has instructed to switch the mode related to the display of the first AR image A1 (S10). Specifically, the control unit 11 determines whether or not option 17A (FIG. 7) is selected via the gesture of the operator. If it is determined that there is no mode switching instruction (NO in S10), the control unit 11 waits until the mode switching instruction is received.

On the other hand, when it is determined that the mode switching instruction has been given (YES in S10), the control unit 11 determines whether or not the currently set mode (hereinafter, also referred to as “setting mode”) is the display mode. Judgment (S11). When it is determined that the setting mode is the display mode (YES in S11), the control unit 11 executes a process of changing the setting mode to the non-display mode (S12). When the setting mode is changed to the non-display mode, the first AR image A1 is hidden in the display unit 14.

On the other hand, if it is determined that the setting mode is not the display mode (NO in S11), the control unit 11 executes a process of changing the setting mode to the display mode (S13).

FIG. 9 is a flowchart showing a processing procedure that is first executed after the setting mode is changed to the display mode. In FIG. 9, the left column shows the processing procedure executed in the HMD 10, and the right column shows the processing procedure executed in the server 20. The process shown in the left column is executed by the control unit 11, for example, when the setting mode is changed to the display mode. The process shown in the right column is always executed by, for example, the control unit 21.

The control unit 11 controls the communication unit 16 so as to transmit a signal requesting information about the specified industrial equipment 30 to the server 20 (S20). Specifically, the control unit 11 controls the communication unit 16 so as to transmit a signal requesting information managed in the database D1 to the server 20 with respect to the specified industrial equipment 30.

The control unit 11 determines whether or not information has been received from the server 20 (S21). When it is determined that the information has not been received from the server 20 (NO in S21), the control unit 11 waits until the information is received from the server 20. On the other hand, when it is determined that the information has been received from the server 20 (YES in S21), the control unit 11 has the first AR based on the information shown in the database D1 and the operating state information acquired from the industrial device 30. The display unit 14 is controlled so as to start displaying the image A1 (S22). In this case, the first AR image A1 is superimposed and displayed on the position actually existing in the industrial equipment 30.

On the other hand, in the server 20, the control unit 21 determines whether or not there is a request for information regarding the industrial equipment 30 (S30). When it is determined that there is no request for information (NO in S30), the control unit 21 waits until a signal requesting information is received from the HMD 10 via the communication unit 23.

On the other hand, when it is determined that there is a request for information (YES in S30), the control unit 21 controls the communication unit 23 to transmit the requested information (information shown in the database D1) to the HMD 10 (S31). ).

FIG. 10 is a flowchart showing a processing procedure executed in the display mode. The process shown in this flowchart is, for example, always executed by the control unit 11 during the display mode.

The control unit 11 determines whether or not the operator is operating based on the outputs of various sensors 15 (for example, at least one of a speed sensor, an acceleration sensor, and a gyro sensor) (S40). Here, "in operation" means an accident when the first AR image A1 is displayed on the display unit 14, such as when the worker is walking or the worker is moving at an acceleration faster than a predetermined value. A situation that is likely to be connected. For example, when the worker moves his head at an acceleration faster than a predetermined speed, it is determined to be "in operation". The determination of S40 is performed, for example, based on the comparison result between the output value of various sensors 15 and the threshold value, and the threshold value may be set by using machine learning.

If it is determined that the image is not in operation (NO in S40), the control unit 11 displays the display unit 14 so as to continue displaying the first AR image A1 when the display unit 14 displays the first AR image A1. The display unit 14 is controlled so as to restart the display of the first AR image A1 when the first AR image A1 is not displayed on the display unit 14 (S41).

On the other hand, when it is determined that the operator is in operation (YES in S40), the control unit 11 hides the first AR image A1 when the first AR image A1 is displayed on the display unit 14. The display unit 14 is controlled so as to keep the first AR image A1 hidden when the first AR image A1 is not displayed on the display unit 14 (S42).

As described above, in the HMD 10, the display mode for displaying the first AR image A1 and the non-display mode for hiding the first AR image A1 can be switched. Therefore, according to the HMD 10, the operator can appropriately switch whether or not to display the first AR image A1 on the display unit according to his / her current work content.

Further, when the first AR image A1 is displayed on the display unit 14, the first AR image A1 covers a predetermined portion of the industrial equipment 30. As a result, the area covered by the first AR image A1 becomes difficult for the user to see. If the first AR image A1 is continuously displayed on the display unit 14 even during the operation of the operator (for example, while walking), the operator may unintentionally come into contact with a surrounding object. In the HMD 10, the first AR image A1 is not displayed on the display unit 14 when it is determined that the operator is operating. Therefore, according to the HMD 10, it is possible to suppress the occurrence of a situation in which the operator unintentionally comes into contact with a surrounding object.

<3-3. Operation related to the display of AR image showing the normal operation state>
FIG. 11 is a diagram schematically showing a second display example of the display unit 14 in a state where communication between the HMD 10 and the industrial equipment 30 is established. As shown in FIG. 11, the display unit 14 displays the second AR image A2 in addition to the information shown in FIG. 7.

The second AR image A2 shows the normal operating state of the component corresponding to the first AR image A1. The second AR image A2 is generated based on the information managed in the database D2 of the server 20. For example, when the procedure for mounting the semiconductor wafer W1 on the stage 34 by the transport mechanism 33 differs between the first AR image A1 and the second AR image A2, the operator indicates that the industrial equipment 30 is not operating normally. Can be recognized.

FIG. 12 is a flowchart showing a procedure for switching the display state of the second AR image A2 showing the normal operating state of the component corresponding to the first AR image A1. The process shown in this flowchart is, for example, repeatedly executed by the control unit 11 during the display mode.

The control unit 11 determines whether or not the display switching instruction of the second AR image A2 is given by the operator (S50). Specifically, the control unit 11 determines whether or not option 17B (FIG. 11) is selected via the gesture of the operator. If it is determined that there is no display switching instruction (NO in S50), the control unit 11 waits until the display switching instruction is received.

On the other hand, when it is determined that the display switching instruction has been given (YES in S50), the control unit 11 determines whether or not the second AR image A2 is currently displayed (S51). When it is determined that the second AR image A2 is not displayed (NO in S51), the control unit 11 executes the display process of the second AR image A2 (S52). On the other hand, when it is determined that the second AR image A2 is displayed (YES in S51), the control unit 11 controls the display unit 14 so as to hide the second AR image A2 (S53).

FIG. 13 is a flowchart showing a display processing procedure of the second AR image A2 executed in S52 of FIG. In FIG. 13, the left column shows the processing procedure executed in the HMD 10, and the right column shows the processing procedure executed in the server 20. The process shown in the left column is executed by the control unit 11, for example, when the operator receives the display instruction of the second AR image A2. The process shown in the right column is always executed by, for example, the control unit 21.

The control unit 11 identifies, for example, a portion of the industrial equipment 30 existing in front of the operator's line of sight based on an image taken by the camera 13 (S60).

The control unit 11 controls the communication unit 16 so as to transmit a signal requesting information regarding the normal operation of the specified portion to the server 20 (S61). Specifically, the control unit 11 controls the communication unit 16 so as to transmit a signal requesting information managed in the database D2 to the server 20 with respect to the specified portion.

The control unit 11 determines whether or not information has been received from the server 20 (S62). If it is determined that the information has not been received from the server 20 (NO in S62), the control unit 11 waits until the information is received from the server 20. On the other hand, when it is determined that the information has been received from the server 20 (YES in S62), the control unit 11 controls the display unit 14 to start displaying the second AR image A2 based on the received information (YES in S62). S63). In this case, the second AR image A2 is displayed on the display unit 14 in a state of being arranged side by side with the first AR image A1, for example. If the first AR image A1 and the second AR image A2 are simultaneously displayed on the display unit 14, it can be said that the first AR image A1 and the second AR image A2 are "displayed on the display unit 14 in a side-by-side manner".

On the other hand, in the server 20, the control unit 21 determines whether or not there is a request for information regarding the normal operation of a specific portion of the industrial equipment 30 (S70). When it is determined that there is no request for information (NO in S70), the control unit 21 waits until a signal requesting information is received from the HMD 10 via the communication unit 23.

On the other hand, when it is determined that there is a request for information (YES in S70), the control unit 21 controls the communication unit 23 to transmit the requested information (information managed in the database D2) to the HMD 10. (S71).

As described above, in the HMD 10, the first AR image A1 and the second AR image A2 are displayed side by side on the display unit 14. Therefore, the user can determine whether or not the current operation of the corresponding portion of the industrial equipment 30 is correct by comparing the first AR image A1 and the second AR image A2.

As with the first AR image A1, the second AR image A2 is hidden when the control unit 11 determines that the operator is in operation.

[4. feature]
As described above, in the HMD 10 according to the present embodiment, the display mode for displaying the first AR image A1 (first mode) and the non-display mode for hiding the first AR image A1 (second mode) are switched. It is possible. Therefore, according to the HMD 10, the worker (user of the HMD 10) can appropriately switch whether or not to display the first AR image A1 on the display unit 14 according to his / her current work content.

[5. Modification example]
Although the embodiments have been described above, the present invention is not limited to the above embodiments, and various modifications can be made without departing from the spirit of the present invention. Hereinafter, a modified example will be described.

<5-1>
In the above embodiment, the databases D1 and D2 are managed by the server 20. However, the databases D1 and D2 do not necessarily have to be managed by the server 20. The databases D1 and D2 may be stored in the storage unit 12 of the HMD 10, for example. In this case, the HMD 10 does not acquire the information managed in the databases D1 and D2 from the server 20.

<5-2>
Further, in the above embodiment, the HMD 10 is mentioned as an example of the display device, but the display device does not necessarily have to be a head-mounted display. The display device may be an electronic device such as a so-called tablet, smartphone or laptop personal computer.

<5-3>
Further, in the above embodiment, the operator gives an instruction to the HMD 10 via a gesture. However, the method of instruction by the operator is not limited to this. For example, the HMD 10 may be equipped with a microphone, and an operator may give an instruction to the HMD 10 via a voice. For example, the configuration may be such that the option 17A is selected by the worker reading out the characters on the option 17A (FIG. 7). Further, for example, the HMD 10 may have an eye tracking function, and the operator may give an instruction to the HMD 10 by utilizing the line-of-sight cursor. For example, the configuration may be such that when the operator places the line-of-sight cursor on the option 17A for a predetermined time or longer, the option 17A is selected.

<5-4>
Further, when a plurality of parts of the industrial equipment 30 are overlapped in the line of sight of the worker, the worker can select which part the first AR image A1 is to be displayed on the display unit 14. May be. As a result, the operator can confirm the operating state of the part for which the operation is most desired to be confirmed. Further, when a plurality of industrial devices 30 overlap each other in the line of sight of the worker, the worker decides which industrial device 30 the first AR image A1 is to be displayed on the display unit 14. It may be selectable. This makes it possible to confirm the operating state of the industrial equipment 30 whose operation is most desired to be confirmed.

<5-5>
Further, in the above embodiment, it is assumed that the various sensors 15 include the distance image sensor. However, a distance image sensor is not always necessary. For example, when the HMD 10 includes a plurality of cameras 13 and the plurality of cameras 13 function as stereo cameras, the distance between the HMD 10 and the industrial equipment 30 may be detected by the plurality of cameras 13.

<5-6>
Further, the image displayed on the display unit 14 in the HMD 10 according to the above embodiment may be shared with, for example, a supporter who is away from the user (worker) of the HMD 10. That is, the HMD 10 may be used for so-called remote support. In this case, the image shared with the supporter may not include an area that should not be shared with the supporter.

<5-7>
Further, in the HMD 10, in addition to the first AR image A1 showing the operating state inside the industrial equipment 30, an AR image showing other states inside the industrial equipment 30 may be displayed. For example, an AR image showing the pressure and temperature in the vacuum chamber 31, the gas type flowing through various pipes, and the like may be displayed on the display unit 14. When an AR image showing the temperature inside the vacuum chamber 31 is displayed, for example, temperature sensors may be attached to a plurality of locations on the inner wall of the vacuum chamber 31. The thermograph generated based on the detection result of the temperature sensor may be displayed on the display unit 14 as an AR image. Further, an AR image showing the temperature of the semiconductor wafer W1, the state of plasma generated in the vacuum chamber 31, the gas flow state in various pipes, and the like may be displayed on the display unit 14. In the HMD 10, for example, it may be possible to select which of the other states of the industrial equipment 30 to display the AR image. Further, the HMD 10 may be able to accept the selection of the range in which the first AR image A1 is desired to be displayed among the industrial equipment 30. For example, the options for these selections may be displayed on the display unit 14, and the operator may select one of the options to make the above selection.

<5-8>
Further, in the HMD 10 according to the above embodiment, when it is determined that the worker is in operation, the first AR image A1 and the second AR image A2 are hidden. However, when it is determined that the worker is in operation, the first AR image A1 and the second AR image A2 do not necessarily have to be hidden. For example, when it is determined that the worker is in operation, the display of the first AR image A1 and the second AR image A2 may be continued.

<5-9>
Further, when both the first AR image A1 and the second AR image A2 are displayed on the display unit 14, the control unit 11 automatically determines whether or not the operating state of the industrial equipment 30 is normal. May be good. For example, the control unit 11 may determine whether or not the operating state of the industrial equipment 30 is normal by comparing the first AR image A1 and the second AR image A2.

<5-10>
Further, in the above embodiment, the AR marker attached to the fixed position of the industrial equipment 30 is regarded as the reference point. However, the reference point of each industrial device 30 is not limited to the AR marker. For example, a barcode or QR code (registered trademark) may be used as a reference point. That is, the reference point is not particularly limited as long as the position of the reference point and the type of the industrial equipment 30 can be specified from the shape of the marker.

<5-11>
Further, in the above embodiment, the space is grasped by using SLAM. However, the spatial grasp does not necessarily have to be performed by SLAM. For example, the space may be grasped based on the AR marker.

<5-12>
Further, in the HMD 10 according to the above embodiment, communication is established with the industrial equipment 30, and the operating state information of the industrial equipment 30 is periodically acquired from the industrial equipment 30. However, the HMD 10 does not necessarily have to acquire the operating state information from the industrial equipment 30. In this case, the HMD 10 displays an AR image of each component in a stationary state with respect to the inside of the industrial equipment 30. By referring to the AR image of each part in the stationary state, the operator can use it for maintenance and troubleshooting of the industrial equipment 30.

1 system, 2 alignment unit, 3 load lock, 4 board storage unit, 5 vacuum transfer chamber, 6 vacuum robot, 7 gate valve, 10 HMD, 11 and 21 control unit, 11A, 21A CPU, 11B, 21B RAM, 11C, 21C ROM, 12,22 storage unit, 12A, 22A control program, 13 camera, 14 display unit, 15 various sensors, 16 communication unit, 17A, 17B choice, 20 server, 30 industrial equipment, 31 vacuum chamber, 32 peephole , 33 transfer mechanism, 33A transfer arm, 33B fork, 34 stage, 34A stage body, 34B lift pin, A1 1st AR image, A2 2nd AR image, D1, D2 database, I1 inner wall, L1 ridge, T1 part, N1 network, W1 Semiconductor wafer.

Claims (5)

A display unit configured to display a first AR (Augmented Reality) image showing the inside of a predetermined portion of the industrial equipment and a second AR image showing the correct operating state of the industrial equipment.
A control unit configured to control the display unit,
A reception unit configured to accept a user's selection of a first mode or a second mode, which is switchable by the control unit, is provided.
The control unit controls the display unit to display the first AR image when the first mode is selected, while hiding the first AR image when the second mode is selected. The display unit is controlled so as to
When the first mode is selected, at least a part of the first AR image is displayed at a position overlapping the predetermined part on the display unit .
The control unit is configured to control the display unit so that when the first AR image is displayed on the display unit, the second AR image is displayed side by side on the first AR image. .. Further comprising a detector configured to detect a value relating to the user's behavior of the mobile terminal.
The control unit is configured to determine whether or not the user is operating based on the detection result of the detection unit.
The control unit receives when the first mode is selected.
When it is determined that the user is not operating, the display unit is controlled so as to display the first AR image, while the display unit is controlled.
The mobile terminal according to claim 1, wherein when it is determined that the user is operating, the display unit is controlled so as to hide the first AR image. Further equipped with a communication unit configured to communicate with the industrial equipment,
The control unit is configured to acquire operating state information indicating the current operating state of the industrial equipment via the communication unit .
The mobile terminal according to claim 1 or 2 , wherein the control unit is configured to generate the first AR image based on the operation state information. A method of displaying a first AR image showing the inside of a predetermined portion of an industrial device and a second AR image showing the correct operating state of the industrial device on a display unit of a mobile terminal.
A step that accepts the user's selection of the first mode or the second mode,
A step of displaying the first AR image on the display unit when the first mode is selected, and
The computer is made to perform the step of hiding the first AR image when the second mode is selected.
When the first mode is selected , at least a part of the first AR image is displayed at a position overlapping the predetermined part on the display unit .
A method of displaying a second AR image showing a correct operating state of an industrial device side by side with the first AR image when the first AR image is displayed on the display unit. A program that causes a computer to execute a process of displaying a first AR image showing the inside of a predetermined part of an industrial device and a second AR image showing the correct operating state of the industrial device on a display unit of a mobile terminal.
A step that accepts the user's selection of the first mode or the second mode,
A step of displaying the first AR image on the display unit when the first mode is selected, and
The computer is made to perform the step of hiding the first AR image when the second mode is selected.
When the first mode is selected , at least a part of the first AR image is displayed at a position overlapping the predetermined part on the display unit .
A program for displaying a second AR image showing a correct operating state of an industrial device side by side with the first AR image when the first AR image is displayed on the display unit.

Images