WO2024155171A1

WO2024155171A1 - Head mounted device for transmitting manipulation input and method for operating same

Info

Publication number: WO2024155171A1
Application number: PCT/KR2024/001040
Authority: WO
Inventors: 강지영; 손동일; 이채경; 최윤정; 권원규
Original assignee: 삼성전자 주식회사
Priority date: 2023-01-20
Filing date: 2024-01-22
Publication date: 2024-07-25

Abstract

A head mounted device according to one embodiment of the present disclosure may comprise: a communication interface; a camera module; a memory for storing instructions; and a processor. The instructions, when executed by the processor (410), may cause the head mounted device (101) to: identify one or more external electronic devices that display a screen in a designated first manner through the camera module; transmit, to a server, a signal requesting that the identified one or more external electronic devices display a marker in a designated second method that is at least partially different from the designated first method, through the communication interface; obtain information about each of the one or more external electronic devices corresponding to the marker from the server through the communication interface; identify information about at least some of the one or more external electronic devices corresponding to the marker on the basis of an image or video obtained through the camera module; and transmit a manipulation input obtained from at least one input device to the at least some of the one or more external electronic devices through the communication interface on the basis of the identified information about the at least some of the one or more external electronic devices.

Description

Head mounted device for transmitting manipulation input and method of operating the same

The present disclosure relates to a head mounted device that transmits a manipulation input obtained from an input device to an external electronic device and a method of operating the same.

With the advancement of electronic device technology, various types of electronic devices such as mobile communication terminals, PDAs (personal digital assistants), electronic notebooks, smart phones, tablet PCs (personal computers), and wearable devices are widely used. . For example, electronic devices include virtual reality (VR), which allows users to have a realistic experience in a computer-generated virtual world, or augmented reality (VR), which adds virtual information (or objects) to the real world. Augmented reality (AR) and mixed reality (MR), which combine virtual reality and augmented reality, can be provided.

As electronic devices become more diverse and complex, users may use multiple electronic devices simultaneously, and for this purpose, an input device corresponding to each electronic device may be individually required.

The above information may be provided as background art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing can be applied as prior art to the present disclosure.

A head mounted device according to an embodiment of the present disclosure may include a communication interface, a camera module, memory for storing instructions, and a processor. The instructions, when executed by the processor, may cause the head mounted device to identify one or more external electronic devices that display a screen in a designated first manner through the camera module. The instructions, when executed by the processor, cause the head mounted device to, through the communication interface, display a marker on the identified one or more external electronic devices in a designated second manner that is at least partially different from the designated first manner. This can cause a signal requesting to be sent to the server. When executed by the processor, the instructions may cause the head mounted device to respectively obtain information about the one or more external electronic devices corresponding to the marker from the server through the communication interface. When executed by the processor, the instructions cause the head mounted device to identify information on at least some of the one or more external electronic devices corresponding to the marker based on the image or video acquired through the camera module. It can cause it to happen. When executed by the processor, the instructions cause the head mounted device to transmit a manipulation input obtained from at least one input device to the communication interface based on the information on at least some of the identified one or more external electronic devices. It can cause transmission to at least some of the one or more external electronic devices through .

A method of operating a head mounted device according to an embodiment of the present disclosure may include identifying one or more external electronic devices that display a screen in a designated first manner through a camera module. A method of operating a head mounted device according to an embodiment includes sending, through a communication interface, a signal requesting that the identified one or more external electronic devices display a marker in a designated second method that is at least partially different from the designated first method. It may include the operation of transmitting to . A method of operating a head mounted device according to an embodiment may include obtaining information on the one or more external electronic devices corresponding to the marker from the server through the communication interface. A method of operating a head mounted device according to an embodiment may include identifying information on at least some of the one or more external electronic devices corresponding to the marker based on an image or video acquired through the camera module. You can. A method of operating a head mounted device according to an embodiment includes, based on the information on at least some of the identified one or more external electronic devices, receiving a manipulation input obtained from at least one input device through the communication interface. It may include transmitting to at least some of the external electronic devices.

In a storage medium storing computer-readable instructions according to an embodiment of the present disclosure, the instructions, when executed by a processor of the head-mounted device, cause the head-mounted device to, through a camera module, may cause the device to identify one or more external electronic devices that display the screen in such a way. The instructions, when executed by a processor of the head mounted device, cause the head mounted device to, through a communication interface, mark the identified one or more external electronic devices in a designated second method that is at least partially different from the designated first method. This can cause a signal to be sent to the server requesting that it be displayed. The instructions, when executed by a processor of the head mounted device, may cause the head mounted device to respectively obtain information about the one or more external electronic devices corresponding to the marker from the server through the communication interface. there is. The instructions, when executed by the processor of the head mounted device, cause the head mounted device to display at least some of the one or more external electronic devices corresponding to the marker based on the image or video acquired through the camera module. When executed by a processor of the head mounted device, the instructions may cause the head mounted device to identify information on at least some of the identified one or more external electronic devices. It may cause a manipulation input obtained from one input device to be transmitted to at least some of the one or more external electronic devices through the communication interface.

1 is a block diagram of an electronic device in a network environment, according to embodiments of the present disclosure.

FIG. 2 is a perspective view illustrating the internal configuration of a wearable electronic device 200 according to an embodiment of the present disclosure.

FIGS. 3A and 3B are diagrams showing the front and back sides of the wearable electronic device 300 according to an embodiment.

Figure 4 is a block diagram of a head-mounted device (HMD) according to an embodiment of the present disclosure.

FIGS. 5A, 5B, 5C, and 5D illustrate a process in which a head mounted device sets a connection state with an external electronic device according to an embodiment of the present disclosure.

FIG. 5E is a screen frame in which a marker of an external electronic device is not displayed according to an embodiment of the present disclosure.

FIG. 5F is a screen frame in which a marker of an external electronic device is displayed according to an embodiment of the present disclosure.

FIG. 6 illustrates a virtual pointer to one or more external electronic devices according to an embodiment of the present disclosure.

FIGS. 7A, 7B, 7C, and 7D illustrate the operation of the head mounted device according to the user's gaze direction according to an embodiment of the present disclosure.

Figure 8 shows the movement of a pointer according to the user's gaze direction, according to an embodiment of the present disclosure.

Figure 9 illustrates movement of a pointer according to a user's manipulation input, according to an embodiment of the present disclosure.

FIG. 10 illustrates a manipulation input through a keyboard for one or more external electronic devices according to an embodiment of the present disclosure.

FIG. 11 illustrates a state in which a manipulation input is transmitted to at least some of one or more external electronic devices based on the gaze direction according to an embodiment of the present disclosure.

FIG. 12 illustrates control of one or more external electronic devices according to a manipulation input of at least one input device according to an embodiment of the present disclosure.

FIG. 13 illustrates the operation of one or more external electronic devices according to a manipulation input of at least one input device according to the first embodiment of the present disclosure.

FIG. 14 illustrates the operation of one or more external electronic devices according to a manipulation input of at least one input device according to a second embodiment of the present disclosure.

FIGS. 15A, 15B, and 15C illustrate operations of one or more external electronic devices according to a manipulation input of at least one input device according to a third embodiment of the present disclosure.

FIG. 16 illustrates the operation of one or more external electronic devices according to the manipulation input of at least one input device according to the fourth embodiment of the present disclosure.

FIGS. 17A, 17B, and 17C illustrate the operation of one or more external electronic devices according to the manipulation input of at least one input device according to the fifth embodiment of the present disclosure.

FIG. 18 illustrates the operation of one or more external electronic devices according to the manipulation input of at least one input device according to the sixth embodiment of the present disclosure.

19A to 19B are flowcharts of a method of operating a head mounted device according to an embodiment of the present disclosure.

1 is a block diagram of an electronic device 101 in a network environment 100, according to embodiments of the present disclosure.

Referring to FIG. 1, in the network environment 100, the electronic device 101 communicates with the electronic device 102 through a first network 198 (e.g., a short-range wireless communication network) or a second network 199. It is possible to communicate with at least one of the electronic device 104 or the server 108 through (e.g., a long-distance wireless communication network). According to one embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108. According to one embodiment, the electronic device 101 includes a processor 120, a memory 130, an input module 150, an audio output module 155, a display module 160, an audio module 170, and a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or may include an antenna module 197. In some embodiments, at least one of these components (eg, the connection terminal 178) may be omitted or one or more other components may be added to the electronic device 101. In some embodiments, some of these components (e.g., sensor module 176, camera module 180, or antenna module 197) are integrated into one component (e.g., display module 160). It can be.

The processor 120, for example, executes software (e.g., program 140) to operate at least one other component (e.g., hardware or software component) of the electronic device 101 connected to the processor 120. It can be controlled and various data processing or calculations can be performed. According to one embodiment, as at least part of data processing or computation, the processor 120 stores commands or data received from another component (e.g., sensor module 176 or communication module 190) in volatile memory 132. The commands or data stored in the volatile memory 132 can be processed, and the resulting data can be stored in the non-volatile memory 134. According to one embodiment, the processor 120 includes the main processor 121 (e.g., a central processing unit or an application processor) or an auxiliary processor 123 that can operate independently or together (e.g., a graphics processing unit, a neural network processing unit ( It may include a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device 101 includes a main processor 121 and a secondary processor 123, the secondary processor 123 may be set to use lower power than the main processor 121 or be specialized for a designated function. You can. The auxiliary processor 123 may be implemented separately from the main processor 121 or as part of it.

The auxiliary processor 123 may, for example, act on behalf of the main processor 121 while the main processor 121 is in an inactive (e.g., sleep) state, or while the main processor 121 is in an active (e.g., application execution) state. ), together with the main processor 121, at least one of the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or the communication module 190) At least some of the functions or states related to can be controlled. According to one embodiment, co-processor 123 (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module 180 or communication module 190). there is. According to one embodiment, the auxiliary processor 123 (eg, neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. Artificial intelligence models can be created through machine learning. For example, such learning may be performed in the electronic device 101 itself on which the artificial intelligence model is performed, or may be performed through a separate server (e.g., server 108). Learning algorithms may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but It is not limited. An artificial intelligence model may include multiple artificial neural network layers. Artificial neural networks include deep neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN), restricted boltzmann machine (RBM), belief deep network (DBN), bidirectional recurrent deep neural network (BRDNN), It may be one of deep Q-networks or a combination of two or more of the above, but is not limited to the examples described above. In addition to hardware structures, artificial intelligence models may additionally or alternatively include software structures.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176) of the electronic device 101. Data may include, for example, input data or output data for software (e.g., program 140) and instructions related thereto. Memory 130 may include volatile memory 132 or non-volatile memory 134.

The program 140 may be stored as software in the memory 130 and may include, for example, an operating system 142, middleware 144, or application 146.

The input module 150 may receive commands or data to be used in a component of the electronic device 101 (e.g., the processor 120) from outside the electronic device 101 (e.g., a user). The input module 150 may include, for example, a microphone, mouse, keyboard, keys (eg, buttons), or digital pen (eg, stylus pen).

The sound output module 155 may output sound signals to the outside of the electronic device 101. The sound output module 155 may include, for example, a speaker or a receiver. Speakers can be used for general purposes such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display module 160 can visually provide information to the outside of the electronic device 101 (eg, a user). The display module 160 may include, for example, a display, a hologram device, or a projector, and a control circuit for controlling the device. According to one embodiment, the display module 160 may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of force generated by the touch.

The audio module 170 can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module 170 acquires sound through the input module 150, the sound output module 155, or an external electronic device (e.g., directly or wirelessly connected to the electronic device 101). Sound may be output through the electronic device 102 (e.g., speaker or headphone).

The sensor module 176 detects the operating state (e.g., power or temperature) of the electronic device 101 or the external environmental state (e.g., user state) and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 176 includes, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 177 may support one or more designated protocols that can be used to connect the electronic device 101 directly or wirelessly with an external electronic device (eg, the electronic device 102). According to one embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102). According to one embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 can convert electrical signals into mechanical stimulation (e.g., vibration or movement) or electrical stimulation that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 can capture still images and moving images. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 can manage power supplied to the electronic device 101. According to one embodiment, the power management module 188 may be implemented as at least a part of, for example, a power management integrated circuit (PMIC).

Battery 189 may supply power to at least one component of electronic device 101. According to one embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

Communication module 190 is configured to provide a direct (e.g., wired) communication channel or wireless communication channel between electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108). It can support establishment and communication through established communication channels. Communication module 190 operates independently of processor 120 (e.g., an application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., : LAN (local area network) communication module, or power line communication module) may be included. Among these communication modules, the corresponding communication module is a first network 198 (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (e.g., legacy It may communicate with an external electronic device 104 through a telecommunication network such as a cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., LAN or WAN). These various types of communication modules may be integrated into one component (e.g., a single chip) or may be implemented as a plurality of separate components (e.g., multiple chips). The wireless communication module 192 uses subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199. The electronic device 101 can be confirmed or authenticated.

The wireless communication module 192 may support 5G networks after 4G networks and next-generation communication technologies, for example, NR access technology (new radio access technology). NR access technology provides high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and access to multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low latency). -latency communications)) can be supported. The wireless communication module 192 may support a high frequency band (eg, mmWave band), for example, to achieve a high data rate. The wireless communication module 192 uses various technologies to secure performance in high frequency bands, for example, beamforming, massive array multiple-input and multiple-output (MIMO), and full-dimensional multiplexing. It can support technologies such as input/output (FD-MIMO: full dimensional MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module 192 may support various requirements specified in the electronic device 101, an external electronic device (e.g., electronic device 104), or a network system (e.g., second network 199). According to one embodiment, the wireless communication module 192 supports Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mmTC, or U-plane latency (e.g., 164 dB or less) for realizing URLLC. Example: Downlink (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) can be supported.

The antenna module 197 may transmit or receive signals or power to or from the outside (eg, an external electronic device). According to one embodiment, the antenna module 197 may include an antenna including a radiator made of a conductor or a conductive pattern formed on a substrate (eg, PCB). According to one embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is connected to the plurality of antennas by, for example, the communication module 190. can be selected. Signals or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (eg, radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module 197.

According to one embodiment, the antenna module 197 may form a mmWave antenna module. According to one embodiment, a mmWave antenna module includes a printed circuit board, an RFIC disposed on or adjacent to a first side (e.g., bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., mmWave band), And a plurality of antennas (e.g., array antennas) disposed on or adjacent to the second side (e.g., top or side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal ( (e.g. commands or data) can be exchanged with each other.

According to one embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199. Each of the external

electronic devices

102 or 104 may be of the same or different type as the electronic device 101. According to one embodiment, all or part of the operations performed in the electronic device 101 may be executed in one or more of the external

electronic devices

102, 104, or 108. For example, when the electronic device 101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 may perform the function or service instead of executing the function or service on its own. Alternatively, or additionally, one or more external electronic devices may be requested to perform at least part of the function or service. One or more external electronic devices that have received the request may execute at least part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may process the result as is or additionally and provide it as at least part of a response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology can be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an Internet of Things (IoT) device. Server 108 may be an intelligent server using machine learning and/or neural networks. According to one embodiment, the external electronic device 104 or server 108 may be included in the second network 199. The electronic device 101 may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

Referring to FIG. 2 , a wearable electronic device 200 according to an embodiment of the present disclosure may include at least one of a light output module 211, a display member 201, and a camera module 250.

According to an embodiment of the present disclosure, the light output module 211 may include a light source capable of outputting an image, and a lens that guides the image to the display member 201. According to an embodiment of the present disclosure, the light output module 211 may be a liquid crystal display (LCD), a digital mirror device (DMD), or a liquid crystal on silicon (LCoS) device. ), a light emitting diode (LED) on silicon (LEDoS), an organic light emitting diode (OLED), or a micro LED (micro light emitting diode, micro LED). .

According to one embodiment of the present disclosure, the display member 201 may include an optical waveguide (eg, wave guide). According to an embodiment of the present disclosure, the output image of the optical output module 211 incident on one end of the optical waveguide may be propagated within the optical waveguide and provided to the user. According to an embodiment of the present disclosure, the optical waveguide may include at least one of at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) or a reflective element (e.g., a reflective mirror). . For example, the optical waveguide may guide the image output from the optical output module 211 to the user's eyes using at least one diffractive element or reflective element.

According to an embodiment of the present disclosure, the camera module 250 can capture still images and/or moving images. According to one embodiment, the camera module 250 may be disposed within a lens frame and around the display member 201.

According to an embodiment of the present disclosure, the first camera module 251 may photograph and/or recognize the trace of the user's eyes (eg, pupil, iris) or gaze. According to an embodiment of the present disclosure, the first camera module 251 periodically sends information (e.g., trajectory information) related to the trajectory of the user's eyes or gaze to a processor (e.g., the processor 120 of FIG. 1). Alternatively, it can be transmitted aperiodically.

According to an embodiment of the present disclosure, the second camera module 253 can capture external images.

According to an embodiment of the present disclosure, the third camera module 255 may be used for hand detection and tracking, and recognition of user gestures (eg, hand movements). The third camera module 255 according to an embodiment of the present disclosure can be used for 3 degrees of freedom (3DoF), 6DoF head tracking, location (space, environment) recognition, and/or movement recognition. there is. The second camera module 253 may be used for hand detection and tracking and user's gesture recognition according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, at least one of the first to third camera modules 251 to 255 may be replaced with a sensor module (eg, LiDAR sensor). For example, the sensor module may include at least one of a vertical cavity surface emitting laser (VCSEL), an infrared sensor, and/or a photodiode.

3A and 3B, in one embodiment,

camera modules

311, 312, 313, and 314 are installed on the first side 310 of the housing to obtain information related to the surrounding environment of the wearable electronic device 300. , 315, 316) and/or a depth sensor 317 may be disposed.

In one embodiment, the

camera modules

311 and 312 may acquire images related to the surrounding environment of the wearable electronic device 300.

In one embodiment, the

camera modules

313, 314, 315, and 316 may acquire images while the wearable electronic device 300 is worn by a user.

Camera modules

313, 314, 315, and 316 may be used for hand detection, tracking, and recognition of user gestures (eg, hand movements). The

camera modules

313, 314, 315, and 316 may be used for 3DoF, 6DoF head tracking, location (space, environment) recognition, and/or movement recognition. In one embodiment,

camera modules

311 and 312 may be used for hand detection and tracking, and user gestures.

In one embodiment, the depth sensor 317 may be configured to transmit a signal and receive a signal reflected from an object, and may be used to determine the distance to an object, such as time of flight (TOF). You can. Instead of or additionally to the depth sensor 317,

camera modules

313, 314, 315, and 316 may check the distance to the object.

According to one embodiment,

camera modules

325 and 326 and/or a display 321 (and/or lens) for face recognition may be disposed on the second surface 320 of the housing.

In one embodiment, the face

recognition camera modules

325 and 326 adjacent to the display 321 may be used to recognize the user's face, or may recognize and/or track both eyes of the user.

In one embodiment, the display 321 (and/or lens) may be disposed on the second side 320 of the wearable electronic device 300. In one embodiment, the wearable electronic device 300 may not include

camera modules

315 and 316 among the plurality of

camera modules

313, 314, 315, and 316. Although not shown in FIGS. 3A and 3B , the wearable electronic device 300 may further include at least one of the components shown in FIG. 2 .

As described above, the wearable electronic device 300 according to one embodiment may have a form factor to be worn on the user's head. The wearable electronic device 300 may further include a strap and/or a wearing member to be fixed on a user's body part. The wearable electronic device 300 may provide a user experience based on augmented reality, virtual reality, and/or mixed reality while worn on the user's head.

FIG. 4 is a block diagram 400 of a head-mounted device (HMD) 101 according to an embodiment of the present disclosure.

Referring to FIG. 4, the head mounted device 101 (e.g., the wearable electronic device 200 of FIG. 2 or the wearable electronic device 300 of FIG. 3A) according to an embodiment includes a processor 410 (e.g., the wearable electronic device 200 of FIG. 1). Processor 120), communication interface 420 (e.g., communication module 190 of FIG. 1), camera module 430 (e.g., camera module 180 of FIG. 1, camera module 250 of FIG. 2, or FIG.

Camera modules

311, 312, 313, 314, 315, and 316 of 3a), and/or display display 440, such as the display module 160 of FIG. 1, the light output module 211 of FIG. 2, or It may include the display 321 of 3a.

The processor 410 according to one embodiment may execute an application or operation stored in a memory (not shown, for example, memory 130 in FIG. 1). In one embodiment, the processor 410 may control at least one other component (eg, hardware or software component) included in the head mounted device 101, or may perform various data processing or operations.

The communication interface 420 according to one embodiment may communicate with one or more external electronic devices (O) or at least one input device (I) using a designated communication method.

In one embodiment, the head mounted device 101 may set a connection state with one or more external electronic devices (O) or at least one input device (I) using a designated connection method through the communication interface 420. Here, the operation of setting the connection state may include the operation of the head mounted device 101 performing a wireless connection with the external electronic device (O) or the input device (I) through the communication interface 420. For example, the designated communication method may be a wired communication method, or may be designated as at least one wireless communication method among Bluetooth (BT), WiFi direct, mobile hotspot, WiFi connection using an access point (AP), and/or WiFi tethering. You can.

In one embodiment, the head mounted device 101 may transmit or receive data or signals with one or more external electronic devices (O) or at least one input device (I) through a designated communication method through the communication interface 420. there is.

In one embodiment, the head mounted device 101 may receive data or signals corresponding to a manipulation input from at least one input device (I) through the communication interface 420. In one embodiment, the head mounted device 101 may transmit data or signals corresponding to a manipulation input received from at least one input device (I) to at least one electronic device through the communication interface 420. . In one embodiment, the head mounted device 101 provides a manipulation input to at least one electronic device identified based on the user's gaze direction recognized through a camera module 430 (e.g., a first camera 431) to be described later. Data or signals corresponding to can be transmitted.

The camera module 430 according to one embodiment includes a first camera 431 that recognizes the user's gaze direction, for example, the first camera module 251 in FIG. 2 or the

camera modules

325 and 326 for face recognition in FIG. 3B. )) and a second camera 432 (e.g., the second camera module 253 in FIG. 2 or the

camera modules

311, 312, 313, 314, 315 in FIG. 3A) with a shooting range including the user's gaze direction. 316)) may be included.

In one embodiment, the first camera 431 is included in the head mounted device 101 and can photograph the user's eyeballs. In one embodiment, the first camera 431 may capture and/or recognize the iris and/or pupil included in the user's eye. In one embodiment, the first camera 431 may be equipped with an LED that emits light (eg, infrared (IR)) in a frequency band designated for the user's eyes. In one embodiment, the head mounted device 101 may track the user's gaze direction based on the photographed and/or recognized pupil.

In one embodiment, the second camera 432 (e.g., the second camera module 253 in FIG. 2) is included in the head mounted device 101 and can capture an external (e.g., front) image or video. there is. In one embodiment, the second camera 432 is arranged so that its field of view (FOV) includes the gaze direction of the user wearing the head mounted device 101, and corresponds to an image or video recognizable by the user. An image or video can be obtained.

In one embodiment, the second camera 432 may acquire an image or video including the external electronic device O and identify one or more external electronic devices O based on the acquired image or video. In one embodiment, the head mounted device 101 (e.g., processor 410) is configured to store an image or video acquired through the second camera 432 based on the user's gaze direction acquired through the first camera 431. At least some of the one or more external electronic devices (O) included in can be identified, and a manipulation input obtained from the at least one input device (I) can be transmitted to at least some of the identified one or more external electronic devices (O).

The display 440 according to one embodiment may provide a visual notification to the user by displaying a recognizable screen. In one embodiment, the display 440 may be arranged to display a specific screen in front of the user's eyes. For example, the display display 440 may be a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), or an organic light emitting diode (LCoS). It may include OLED (organic light emitting diode) or micro LED (micro light emitting diode). In one embodiment, when the display 440 is made of one of a liquid crystal display device, a digital mirror display device, or a silicon liquid crystal display device, it may further include a light source that radiates light to the screen output area of the display display 440. . In one embodiment, when the display 440 is made of one of organic light emitting diodes or micro LEDs, a screen of good quality can be provided even if it does not include a separate light source. In one embodiment, the display 440 can display a screen on a transparent or translucent display area (e.g., the display member 201 in FIG. 2) and transmit light to the display area through a waveguide.

The head mounted device 101 according to one embodiment may display an object in the form of augmented reality (AR) on the display 440. In one embodiment, the head mounted device 101 may display an image or video acquired through a camera module 430 (eg, the second camera 432) on the display 440.

The head mounted device 101 according to an embodiment displays the screen of the external electronic device O displayed in the designated first method on the display display 440 in the image or video captured by the second camera 432. However, the marker displayed in the designated second method may not be displayed on the display display 440.

In addition, the head mounted device 101 according to an embodiment may further include other components included in the electronic device 101 of FIG. 1.

FIGS. 5A to 5D illustrate a process in which the head mounted device 101 sets a connection state with one or more external electronic devices O1 and O2 according to an embodiment of the present disclosure.

Referring to FIG. 5A, the head mounted device 101 according to an embodiment mounts one or more external electronic devices (O1, O2) through a camera module (e.g., the camera module 430 or the second camera 432 of FIG. 4). ). For example, the head mounted device 101 may identify one or more external electronic devices O1 and O2 located within the field of view (FOV) of the camera module. O1, O2) may be an output device including an external display (not shown) that displays a visual screen.

The head mounted device 101 according to an embodiment may identify one or more external electronic devices O1 and O2 located around it through a communication interface (e.g., the communication interface 420 of FIG. 4). For example, the head mounted device 101 may identify external electronic devices O1 and O2 that can be connected to the head mounted device 101 based on a designated communication method.

In one embodiment, the head mounted device 101 may identify one or more external electronic devices O1 and O2 when it is in a designated control mode (eg, VST control mode) or when a user input occurs.

In one embodiment, the external electronic devices O1 and O2 may display the screen in a designated first method. Here, the designated first method may be a method of displaying the screen at a designated first number of frames per unit time (eg, 30 fps, 60 fps, 90 fps, 120 fps).

In one embodiment, the head mounted device 101 may check whether one or more identified external electronic devices O1 and O2 are already registered devices (e.g., devices with a connection state set). For example, external electronic devices O1 and O2 already registered in the head mounted device 101 may display markers M1 and M2 in a designated second method. For example, the markers M1 and M2 are identification codes that can identify at least one external electronic device O1 and O2, and may be, for example, a quick response code (QR code).

In one embodiment, when the head mounted device 101 identifies one or more unregistered external electronic devices (O1, O2), the head mounted device 101 determines whether to proceed with the registration process for the identified one or more external electronic devices (O1, O2). Notifications can be provided to users. For example, a notification may be a visual notification displayed on a display display.

Referring to FIG. 5B, when the head mounted device 101 according to an embodiment determines that the identified one or more external electronic devices O1 and O2 are unregistered devices, the identified one or more external electronic devices O1 and O2 are connected to each other through a communication interface. The external electronic devices O1 and O2 may transmit a signal to the server S requesting that the markers M1 and M2 be displayed in a designated second manner.

In one embodiment, one or more external electronic devices O1 and O2 may be connected to the server S in advance. For example, one or more external electronic devices O1 and O2 may be previously connected to the server S in response to the user's account (eg, Samsung account). In one embodiment, one or more external electronic devices O1 and O2 may receive a signal from the server S requesting to display the markers M1 and M2 in a designated second manner. In one embodiment, one or more external electronic devices (O1, O2) display the markers (M1, M2) in a designated second manner based on a signal requesting to display the markers (M1, M2) in a designated second manner. can do.

In one embodiment, the head mounted device 101 may obtain information on one or more external electronic devices O1 and O2 corresponding to the markers M1 and M2 from the server S through a communication interface. In one embodiment, the head mounted device 101 may store information on one or more acquired external electronic devices O1 and O2. In one embodiment, the head mounted device 101 is configured to receive one or more external electronic devices (O1, O2) from the server (S) when one or more external electronic devices (O1, O2) are already registered or when information on one or more external electronic devices (O1, O2) is stored. The above operation of acquiring information on the external electronic devices O1 and O2 can be omitted.

In one embodiment, the head mounted device 101 acquires an image or video of the markers (M1, M2) displayed on one or more external electronic devices (O1, O2) through a camera module (e.g., a second camera). You can.

In one embodiment, the head mounted device 101 may receive mapping data of the coordinate systems of the external electronic devices O1 and O2 from the server S. In one embodiment, the head mounted device 101 may set the position of the pointer according to the user's gaze direction based on mapping data of the coordinate system received from the server (S).

In one embodiment, the head mounted device 101 is configured to use one or more images or videos of one or more external electronic devices (O1, O2) acquired through mapping data of a coordinate system received from the server (S) and a camera module. The coordinate system of external electronic devices (O1, O2) can be mapped. For example, the head mounted device 101 may be configured to control the size of the image or video of one or more external electronic devices (O1, O2) and/or the location of the image or video of one or more external electronic devices (O1, O2), e.g. The location of the feature point) may be identified, and the coordinate system of one or more external electronic devices O1 and O2 may be mapped based on the size and/or location of the identified external electronic devices O1 and O2.

Referring to FIG. 5C, the head mounted device 101 according to an embodiment may identify information on one or more external electronic devices O1 corresponding to the marker M1 based on the acquired image or video. The head mounted device 101 according to an embodiment recognizes the marker M1 based on the acquired image or video, and one or more external electronic devices ( Information on O1) can be identified.

The head mounted device 101 according to one embodiment may provide a notification related to confirmation of one or more external electronic devices (O1, O2) in response to the acquired information on one or more external electronic devices (O1, O2). there is.

Referring to FIG. 5D, the head mounted device 101 according to one embodiment is connected to one or more external electronic devices (O1) identified based on information on one or more external electronic devices (O1) corresponding to the marker (M1). You can complete the registration process. In one embodiment, one or more external electronic devices O1 registered with the head mounted device 101 may display a marker M1 on the screen.

In one embodiment, when the head mounted device 101 determines that the identified one or more external electronic devices O1 are pre-registered devices, the head mounted device 101 is connected to the identified one or more external electronic devices O1 through a communication interface. can be set. In one embodiment, the head mounted device 101 may set a connection state with one or more external electronic devices O1 identified based on recognition of a marker through a camera module. In one embodiment, the head mounted device 101 may transmit and receive data or signals with one or more external electronic devices O1 based on the set connection state.

FIG. 5E is a screen frame in which the marker M of the external electronic device O according to an embodiment of the present disclosure is not displayed. FIG. 5F is a screen frame in which a marker M of an external electronic device O according to an embodiment of the present disclosure is displayed.

Referring to FIGS. 5E and 5F , the external electronic device O according to one embodiment may display a screen in a designated first method. Here, the designated first method may be a method in which the external electronic device O displays the screen at a designated first number of frames per unit time (eg, 60 fps). For example, the screen frame corresponding to the first number of frames specified per unit time may include all screens displayed on the external electronic device O.

The external electronic device O according to one embodiment may display the marker M in a designated second method. Here, the designated first method may be a method in which the external electronic device O displays the screen with a designated second frame number (eg, 30 fps) that is smaller than the designated first frame number per unit time. In one embodiment, a marker (M) may be displayed on a screen frame corresponding to the second number of frames, which is a portion of the screen frames corresponding to the specified number of first frames. For example, the one or more external electronic devices (O) may display a specified number of second frames in a specified second manner that is a specified ratio (e.g., 1/2, 1/3) of the specified number of first frames in a specified first manner. The screen can be displayed.

The external electronic device O according to one embodiment may display a screen using a screen frame in which the marker M is displayed and a screen frame in which the marker M is not displayed.

The head mounted device 101 according to one embodiment uses a second camera (e.g., the second camera 432 in FIG. 4) to display a screen frame in which the marker M is displayed and a screen frame in which the marker M is not displayed. All screen frames can be obtained. In one embodiment, the head mounted device 101 displays only screen frames in which the marker M is not displayed on the display display (display display 440 in FIG. 4), or displays only screen frames in which the marker M is displayed. The marker (M) can be deleted or replaced and displayed on the display display 440. Accordingly, the marker M can be recognized by the second camera 432, but may not be displayed on the display 440.

FIG. 6 illustrates a virtual pointer P to one or more external electronic devices O1 and O2 according to an embodiment of the present disclosure.

Referring to FIG. 6, a head mounted device (e.g., the head mounted device 101 of FIG. 4) according to an embodiment includes at least one input device (e.g., at least one input device of FIG. 4) based on a designated communication method. (I)) can be connected. For example, at least one input device (I) may include a mouse (not shown) that obtains a manipulation input corresponding to movement or selection on a two-dimensional plane.

The head mounted device 101 according to one embodiment may set a connection state based on a designated communication method with two external electronic devices O1 and O2 located within the field of view (FOV). In one embodiment, the head mounted device 101 includes a first external electronic device O1 that displays a screen corresponding to a specific application as a whole, and a second external electronic device O1 that displays screens corresponding to a plurality of applications A1 and A2, respectively. 2 You can set the connection status based on the external electronic device (O2) and the specified communication method.

The head mounted device 101 according to an embodiment recognizes the user's gaze direction and sets a virtual pointer (P) to one of one or more external electronic devices (O1, O2) (O2) based on the recognized gaze direction. ) can be located. The head mounted device 101 according to one embodiment may activate the external electronic device O2 where the virtual pointer P is located or the application A2 where the virtual pointer P is located.

In one embodiment, the virtual pointer P may be a pointer of at least one input device (eg, mouse) connected to the head mounted device 101 based on a specified communication method.

In one embodiment, the head mounted device 101 may move the virtual pointer P based on the movement of the user's gaze direction. In one embodiment, the head mounted device 101 may move the pointer P when the user's gaze direction moves beyond a specified range or a specified distance.

The head mounted device 101 according to one embodiment may transmit a mouse over manipulation input corresponding to the pointer P to the external electronic device O2 where the pointer P of the mouse is located.

The head mounted device 101 according to one embodiment may receive a manipulation input corresponding to a click, drag, or scroll from a mouse (not shown) connected through a designated communication method, and may display the received manipulation input as a pointer ( It can be transmitted to an external electronic device (O2) where P) is located.

FIGS. 7A to 7D illustrate the operation of a head mounted device (eg, the head mounted device 101 of FIG. 4 ) according to the user's gaze direction according to an embodiment of the present disclosure.

Referring to FIG. 7A, the head mounted device 101 according to an embodiment measures the user's gaze direction (E) through a camera module (e.g., the camera module 430 or the first camera 431 of FIG. 4). It can be obtained. In one embodiment, the head mounted device 101 may recognize a specific point (E) on one or more external electronic devices (O1, O2) based on the obtained gaze direction. For example, a head mounted device may recognize a specific point (E) according to the gaze direction, but may not display it as a visual cue.

In one embodiment, the head mounted device 101 is installed in the user's gaze direction based on mapping data of one or more external electronic devices O1 and O2 received from a server (e.g., server S in FIG. 5B). The coordinates of a specific point (E) can be identified. In one embodiment, the head mounted device 101 is based on images or images of one or more external electronic devices O1 and O2 acquired through the camera module 430 (e.g., the second camera 431 in FIG. 4). Thus, the coordinate system of one or more electronic devices O1 and O2 can be mapped. For example, the head mounted device 101 maps the coordinate system of one or more external electronic devices (O1, O2) based on feature points (e.g., corners or borders) of the one or more external electronic devices (O1, O2), and maps The coordinates for a specific point (E) of the head mounted device 101 can be mapped based on one coordinate system.

Referring to FIG. 7B, the head-mounted device 101 according to one embodiment is configured so that the user's gaze direction is at a specific point (E, or within a specified range from a specific point) on one or more external electronic devices (O1, O2). If it remains for more than a specified time (e.g. N seconds), the virtual pointer (P) can be moved to a specific point (E) or displayed at a specific point (E).

The head mounted device 101 according to one embodiment may move the pointer so that the virtual pointer P overlaps with at least one external electronic device O1 and O2 identified.

Referring to FIG. 7C, the head mounted device 101 according to one embodiment is configured to use an external electronic device (O1, O2) or an application (not shown) in response to a pointer (P) that moves based on the user's gaze direction. can be activated. In one embodiment, the head mounted device 101 may display a visual cue on the activated external electronic device O1 or an application (not shown). For example, the visual cue may correspond to a border on the screen of an activated external electronic device or application, but is not limited to this.

Referring to FIG. 7D, the head mounted device 101 according to one embodiment can move the pointer P based on a physical movement input of a mouse (not shown). In one embodiment, a mouse (not shown) corresponding to at least one input device may be connected to the head mounted device 101, and the pointer (P) may be moved based on the physical manipulation input of the mouse (not shown). You can. For example, the head mounted device 101 may control the pointer P by prioritizing physical manipulation input through a mouse (not shown) over movement in the gaze direction.

In one embodiment, the head mounted device 101 may activate an external electronic device O2 or an application (not shown) where the pointer P is located. The head mounted device 101 according to one embodiment may switch the activated external electronic device O2 or application (not shown) based on the movement of the pointer P.

When receiving a manipulation input from at least one input device (not shown), the head mounted device 101 according to one embodiment transmits the manipulation input to an activated external electronic device (O2) or an application (not shown). You can.

Figure 8 shows the movement of the pointer P according to the user's gaze direction, according to an embodiment of the present disclosure.

Referring to FIG. 8, the field of view (FOV) of the head mounted device 101 according to one embodiment may include one or more external electronic devices O1 and O2.

As shown in (a) of FIG. 8, the head mounted device 101 according to one embodiment provides a pointer (P) to the user based on the manipulation input of an input device (e.g., a keyboard) other than the mouse. You can hide the pointer (P) so that it is not displayed.

The head mounted device 101 according to one embodiment may display the pointer P when the manipulation input of at least one input device (not shown) is interrupted (e.g., a specified time has elapsed since the last manipulation input). there is. In one embodiment, the head mounted device 101 displays a pointer (P) at a specific point (E) according to the user's gaze direction through one or more external electronic devices (e.g., display display 440 of FIG. 4). It can be displayed on the table.

The head mounted device 101 according to one embodiment may recognize the user's gaze direction through a camera module (eg, the camera module 430 or the first camera 431 in FIG. 4). The head mounted device 101 according to one embodiment can move the pointer (P) to a specific point (E) according to the gaze direction, as shown in (b) of FIG. 8.

In one embodiment, the head mounted device 101 moves a specific point (E) according to the user's gaze direction to another external electronic device (O1), or to the external electronic device (O1), as shown in (c) of FIG. 8. When moving to another application screen (not shown) of the electronic device O2, the pointer P can be moved based on the gaze direction.

Based on the movement of the recognized gaze direction, the head-mounted device 101 according to one embodiment switches the external electronic device (O2, e.g., activated external electronic device) identified based on the gaze direction to another external electronic device (O1). ) can be converted to .

Figure 9 shows movement of the pointer P according to a user's manipulation input, according to an embodiment of the present disclosure.

Referring to FIG. 9 , the field of view (FOV) of a head mounted device (eg, the head mounted device 101 of FIG. 4 ) according to an embodiment may include one or more external electronic devices O1 and O2.

As shown in (a) of FIG. 9, the head mounted device 101 according to one embodiment is installed at a specific position (E) according to the user's gaze direction so as to overlap one or more external electronic devices (O1, O2). You can display the pointer (P) or move the pointer (P).

The head mounted device 101 according to one embodiment, as shown in (b) of FIG. 9, is a user's device within the screen of an external electronic device (O1) or an application (not shown) where the pointer (P) is located. When the gaze direction moves, the displayed pointer (P) may not be moved even though the specific position (E) moves according to the gaze direction.

The head mounted device 101 according to one embodiment does not move the displayed pointer (P) despite the movement of the specific position (E) according to the gaze direction when the user's gaze direction moves within a specified distance range. It may not be possible.

As shown in (c) of FIG. 9, the head mounted device 101 according to one embodiment includes a manipulation input through at least one input device (e.g., at least one input device (I) of FIG. 4). In this case, even if the specific position (E) according to the gaze direction moves, the displayed pointer (P) may not be moved. For example, the head mounted device 101 generates a physical manipulation input (e.g., click, drag, scroll) through a mouse (not shown), or a signal corresponding to an input key through a keyboard (not shown). When occurs, the pointer P may not be moved even if the gaze direction moves.

When obtaining a manipulation input through at least one input device (I), the head mounted device 101 according to one embodiment may, despite the gaze direction moving, at least one of the external electronic devices identified based on the gaze direction. Some (O1, e.g. external electronic devices activated) may remain. For example, the head mounted device 101 moves the gaze direction when a physical manipulation input is generated through a mouse (not shown) or a signal corresponding to an input key is generated through a keyboard (not shown). Even so, at least some of the external electronic devices identified based on the gaze direction (O1, e.g., activated external electronic device) may be maintained.

FIG. 10 illustrates a manipulation input using a keyboard (not shown) for one or more external electronic devices O1 and O2 according to an embodiment of the present disclosure.

Referring to FIG. 10, a head mounted device (e.g., the head mounted device 101 of FIG. 4) according to an embodiment includes at least one input device (e.g., at least one input device of FIG. 4) based on a designated communication method. (I)) can be connected. For example, at least one input device (I) may include a keyboard (not shown) that obtains a manipulation input corresponding to an input key.

The head mounted device 101 according to one embodiment may transmit a manipulation input through a keyboard (not shown) to at least some (O2) of the one or more identified external electronic devices (O1, O2).

The head mounted device 101 according to one embodiment recognizes the user's gaze direction through a camera module (e.g., the camera module 430 or the first camera 431 in FIG. 4) and creates a virtual camera according to the recognized gaze direction. A pointer (P) can be displayed. For example, when the virtual pointer P is located in the text box included in the application A2 of the external electronic device O2, as shown, the head mounted device 101 is connected to the external electronic device O2. Alternatively, the key input pad of the application A2 may be activated, or a signal corresponding to the acquired input key may be displayed in text. In one embodiment, a signal corresponding to an input key obtained from the external electronic device O2 is received from the head mounted device 101, or is transmitted to the head mounted device 101 based on an input from a keyboard (not shown). It may have been conveyed by

FIG. 11 illustrates a state in which a manipulation input is transmitted to at least some (O2) of one or more external electronic devices (O1, O2, and O3) based on the gaze direction according to an embodiment of the present disclosure.

Referring to FIG. 11, a head mount device (e.g., the head mount device 101 of FIG. 4) according to an embodiment includes a camera module (e.g., the camera module 430 or the second camera 432 of FIG. 4). It can be connected to one or more external electronic devices (O1, O2, O3) located within a field of view (FOV) using a designated communication method.

The head mounted device 101 according to an embodiment is based on the user's gaze direction recognized through a camera module (e.g., the camera module 430 or the first camera 431 in FIG. 4), and uses one or more external electronic devices. One (O2) of the devices (O1, O2, and O3) can be identified, and one (O2) of the identified external electronic devices (O1, O2, and O3) (O2) can be activated.

The head mounted device 101 according to an embodiment identifies one application (A2) included in one or more external electronic devices (O1, O2, O3) based on the user's gaze direction, and The application (A2) can be activated. The head mounted device according to one embodiment may display or move a virtual pointer (P) at a specific point (E) according to the user's gaze direction.

The head mounted device 101 according to one embodiment provides a virtual pointer (P) when the user's gaze direction corresponds to the screen of the second application (A2) displayed on the second external electronic device (O2). 2 Can be displayed on the screen of the application (A2). The head mounted device 101 according to an embodiment activates the second application A2 when the user's gaze direction corresponds to the screen of the second application A2 displayed on the second external electronic device O2. You can do it.

The head mounted device 101 according to one embodiment transmits the manipulation input to the activated second external electronic device O2 when a manipulation input is generated from at least one input device (not shown) connected through a designated communication method. Alternatively, a manipulation input may be transmitted to the activated second application (A2) through the second external electronic device (O2). For example, when acquiring a signal corresponding to an input key from a keyboard (not shown), the head mounted device 101 transmits the input key to the activated second external electronic device (O2) or the activated second application (A2). You can enter text corresponding to (e.g. typing).

The head mounted device 101 according to an embodiment may, when a specific point (E) according to the user's gaze direction moves to the third external electronic device (O3), move to the third external electronic device (O3) or the third external electronic device (O3). An application (not shown) running on the electronic device (O3) can be activated. When a manipulation input is generated from at least one input device (not shown) connected through a designated communication method, the head mounted device 101 according to one embodiment transmits the manipulation input to the activated third external device O3 or , Alternatively, a manipulation input may be transmitted to an activated application (not shown) running through a third external electronic device. For example, when obtaining a signal corresponding to an input key from a keyboard (not shown), the head mounted device 101 may input text corresponding to the input key into the third external electronic device O3.

FIG. 12 illustrates control of one or more external electronic devices O1 and O2 according to a manipulation input from at least one input device (not shown) according to an embodiment of the present disclosure.

Referring to FIG. 12, a head mounted device (e.g., the head mounted device 400 of FIG. 4) according to an embodiment is configured to connect at least one input device (not shown) within a field of view (FOV) through a designated communication method. can be connected to the mouse. The head mounted device 101 according to an embodiment may transmit a manipulation input obtained from a mouse (not shown) to at least one of one or more external electronic devices O1 and O2.

The head mounted device 101 according to one embodiment, as shown in (a) of FIG. 12, displays a virtual pointer based on a specific point (E) according to the user's gaze direction to the second external electronic device (O2). ) or it can be located in an application running on the second external electronic device (O2).

The head mounted device 101 according to an embodiment may receive a manipulation input for the second external electronic device O2 or an application running on the second external electronic device O2 from a mouse (not shown). For example, the manipulation input received from a mouse (not shown) may be a drag input or a drop input for the header of an application running on the second external electronic device O2. The head mounted device 101 according to one embodiment may transmit the received manipulation input to the second external electronic device O2. According to one embodiment, the second external electronic device O2 displays the screen of an application executed on the second external electronic device O2 based on the manipulation input of the mouse (not shown) received from the head mounted device 101. can be moved.

The head mounted device 101 according to one embodiment may input or drag a movement from a mouse (not shown) to the first external electronic device (O1) for the header of an application running on the second external electronic device (O2). It may be a drag & drop input. The head mounted device 101 according to an embodiment may transmit a manipulation input received from a mouse (not shown) to the first external electronic device O1 and/or the second external electronic device O2.

As shown in (b) of FIG. 12, the first external electronic device O1 according to an embodiment selects the first application to be executed on the second external electronic device O2 based on the received manipulation input. It can be moved to an external electronic device and executed, or it can be displayed on the first external electronic device O1. The second external electronic device O2 according to an embodiment terminates the application running on the second external electronic device O2, or terminates the application running on the second external electronic device O2 based on the received manipulation input. 2 The external electronic device (O2) can stop displaying the application screen.

The head mounted device 101 according to an embodiment provides a movement input from a mouse (not shown) to an external space (e.g., virtual space) for the header of an application running on the second external electronic device O2. Alternatively, it may be a drag & drop input. The head mounted device 101 according to one embodiment may transmit a manipulation input received from a mouse (not shown) to the second external electronic device O2.

The second external electronic device O2 according to an embodiment displays a screen of an application running on the second external electronic device O2 based on the received manipulation input, as shown in (c) of FIG. 12. It can be moved to be displayed in a virtual space rather than displayed on the second external electronic device (O2), or can be executed as a virtual application (A.O.).

In one embodiment, the head mounted device 101 copies applications, data, and content between different external electronic devices based on a manipulation input obtained from at least one input device (not shown), or You can move.

FIG. 13 illustrates the operation of one or more external electronic devices (O1, O2, O3) according to the manipulation input of at least one input device (M, K) according to the first embodiment of the present disclosure. FIG. 14 illustrates the operation of one or more external electronic devices (O1, O2, O3) according to the manipulation input of at least one input device (M, K) according to the second embodiment of the present disclosure. FIGS. 15A to 15C illustrate the operation of one or more external electronic devices O1, O2, and O3 according to the manipulation input of at least one input device (M, K) according to the third embodiment of the present disclosure. FIG. 16 illustrates the operation of one or more external electronic devices (O1, O2, O3) according to the manipulation input of at least one input device (M, K) according to the fourth embodiment of the present disclosure.

13 to 16, one or more external electronic devices O1, O2, and O3 are located within the field of view (FOV) of the head mounted device (e.g., the head mounted device 101 of FIG. 4) according to an embodiment. can be located. For example, a first external electronic device O1, a second external electronic device O2, and a third external electronic device O3 may be included within the field of view (FOV) of the head mounted device 101. In one embodiment, the first application A1 and the second application A2 are executed on the second external electronic device O2, and screens corresponding to each application A1 and A2 may be displayed. .

Additionally, virtual applications (A.O.1, A.O.2) may be executed within the field of view (FOV) of the head mounted device 101. In one embodiment, the first virtual application (A.O.1) and the second virtual application (A.O.2) may be executed in a virtual space located above the second external electronic device (O2). In one embodiment, the first virtual application (A.O.1) and the second virtual application (A.O.2) may be executed on the head mounted device 101 or on the second external electronic device (O2).

Referring to FIG. 13, the head mounted device 101 according to an embodiment may activate the second application A2 displayed on the second external electronic device O2 based on the user's gaze direction. The head mounted device 101 according to one embodiment may position a virtual pointer (P) at a specific point (E) according to the user's gaze direction so that it overlaps the screen of the second application (A2).

The head mounted device 101 according to an embodiment can input (e.g. typing) a signal corresponding to an input key received from the keyboard K into the second external electronic device O2 or the second application A2. there is.

The head mounted device 101 according to one embodiment may activate the first virtual application (A.O.1) displayed in the virtual space based on the movement of the user's gaze direction. In one embodiment, the head mounted device 101 responds to the movement of the user's gaze direction when the manipulation input from at least one input device (e.g., keyboard K) is interrupted for more than a specified time (e.g., N seconds). Based on this, the virtual pointer (P) can be moved. In one embodiment, when the head mounted device 101 does not receive a manipulation input from at least one input device (e.g., keyboard K) for more than a specified time (e.g., N seconds), the user's gaze direction is moved. Based on this, the activating external electronic device (O2) or application (A2, A.O.1) can be switched.

The head mounted device 101 according to one embodiment is configured to respond to the movement of the user's gaze direction when the manipulation input from at least one input device (e.g., keyboard K) is interrupted for more than a specified time (e.g., N seconds). Based on this, a signal corresponding to the input key received from the keyboard (K) can be input (eg, typing) into the first virtual application (A.O.1).

Referring to FIG. 14, the head mounted device 101 according to an embodiment may activate the second application A2 displayed on the second external electronic device O2 based on the user's gaze direction. The head mounted device 101 according to one embodiment can position the virtual pointer (P) to overlap the screen of the second application (A2) based on the movement of the specific point (E) according to the user's gaze direction. there is.

The head mounted device 101 according to one embodiment is configured to control the user's input when the manipulation input from at least one input device (e.g., keyboard K) is continuous or is interrupted for less than a specified time (e.g., N seconds). Even if the direction of gaze moves, the external electronic device (O2) or application (A2) that is activated can be maintained without switching. The head mounted device 101 according to one embodiment is configured to control the user's input when the manipulation input from at least one input device (e.g., keyboard K) is continuous or is interrupted for less than a specified time (e.g., N seconds). The virtual pointer (P) can be moved based on the direction of gaze moving.

The head mounted device 101 according to one embodiment is configured to control the user's input when the manipulation input from at least one input device (e.g., keyboard K) is continuous or is interrupted for less than a specified time (e.g., N seconds). Despite the gaze direction moving, a signal corresponding to the input key received from the keyboard K can be input (eg, typing) into the second application A2.

Referring to FIGS. 15A to 15C , the head mounted device 101 according to an embodiment is based on the user's gaze direction and the manipulation input of at least one input device (M, K), one or more external electronic devices (O1) , O2, O3) or applications (A1, A2, A.O.1, A.O.2) can be activated respectively.

The head mounted device 101 according to one embodiment may activate the first application A1 running on the second external electronic device O2 based on the user's gaze direction, as shown in FIG. 15A. there is. In one embodiment, the head mounted device 101 is positioned at a specific point (E) according to the direction of the user's gaze on the screen of the second external electronic device (O2) or the first application (A1) running on the second external electronic device. A virtual pointer (P) can be positioned. In one embodiment, when the head mounted device 101 receives a signal corresponding to an input key from the keyboard K connected through a designated communication method, the head mounted device 101 executes the second external electronic device O2 or the second external electronic device. A signal corresponding to the input key may be input (eg, typing) into the first application (A1).

The head mounted device 101 according to one embodiment may move the pointer (P) based on a movement input received from the mouse (M) connected through a designated communication method. For example, based on receiving a movement input to the third external electronic device O3 through the mouse M, the head mounted device 101 moves the third external electronic device O3, as shown in FIG. 15B. You can move the pointer (P) to the application running in ).

The head mounted device 101 according to one embodiment switches the activated external electronic device (O2, O3) or the activated application (A2) based on the movement of the pointer (P) according to the manipulation input of the mouse (M). can do. For example, the head mounted device 101 may activate the third external electronic device O3 where the pointer P is located or an application running on the third external electronic device O3. The head mounted device 101 according to one embodiment may transmit a signal related to the scroll input received from the mouse M to the third external electronic device O3.

In one embodiment, the head mounted device 101 moves the user's gaze direction from corresponding to the third external electronic device O3 to corresponding to the second external electronic device O2, independent of the mouse M. When receiving a manipulation input from the keyboard K, the pointer P can be moved back to the second external electronic device O2. In one embodiment, the head mounted device 101 moves the user's gaze direction from the third external electronic device O3 to the second external electronic device O2, as shown in FIG. 15C, and the keyboard K ), the pointer P can be moved back to the second external electronic device O2. In one embodiment, the head mounted device 101 may switch the activated external electronic device or application to the second external electronic device (O2) or the first application (A1) running on the second external electronic device (O2). there is.

In one embodiment, when the head mounted device 101 receives a signal corresponding to an input key from the keyboard K, the second external electronic device O2 or the first application A1 running on the second external electronic device ), you can input a signal (e.g. typing) corresponding to the input key.

In one embodiment, when the user's gaze moves back to the third external electronic device O3 and a manipulation input from the mouse M occurs, the head mounted device 101 controls the activated external electronic device or application. 3 You can switch to an application running on an external electronic device (O3) or a third external electronic device (O3). In one embodiment, the head mounted device 101 may transmit the manipulation input of the mouse M to an application running on the external electronic device O3 or a third external electronic device O3.

Referring to FIG. 16, while the head mounted device 101 according to one embodiment does not recognize the gaze direction, the input signal received from at least one input device (M, K) is later recognized as an external signal according to the gaze direction. It can be delivered to an electronic device (O2) or an application (A2).

In one embodiment, the head mounted device 101 moves to the first virtual application (A.O.1) where a specific point (E) according to the user's gaze direction is displayed in a virtual space, and then moves back to the second external electronic device ( When moving to the second application (A2) of O2), the user's gaze direction may not be recognized for a certain period of time. In one embodiment, when the head mounted device 101 receives a manipulation input from at least one input device (M, K) while not recognizing the user's gaze direction, at least one The manipulation input received from the input devices (M, K) can be typed into the previously activated first virtual application (A.O.1).

In one embodiment, when the manipulation input from the at least one input device (M, K) is interrupted, the head mounted device 101 receives the recognition of the gaze direction from the at least one input device (M, K) during the delayed time. One input can be transferred from the first virtual application (A.O.1) to the second application (A2) according to the later recognized gaze direction.

17A to 17C illustrate the operation of one or more external electronic devices O1 and O2 according to the manipulation input of at least one input device I according to the fifth embodiment of the present disclosure. FIG. 18 illustrates the operation of one or more external electronic devices O1 and O2 according to a manipulation input of at least one input device I according to the sixth embodiment of the present disclosure.

17A to 17C and 18, one or more external electronic devices O1 and O2 are located within the field of view (FOV) of the head mounted device (e.g., the head mounted device 101 of FIG. 4) according to an embodiment. ) can be located. For example, a first external electronic device O1 and a second external electronic device O2 may be included within the field of view (FOV) of the head mounted device 101. In one embodiment, the first application A1 and the second application A2 are executed on the second external electronic device O2, and screens corresponding to each application may be displayed.

Referring to FIG. 17, the head mounted device 101 according to an embodiment can move a virtual pointer (P) based on a specific point (E) according to the user's gaze direction, and moves the virtual pointer (P) based on the user's gaze direction. Thus, the external electronic device (O1, O2, O3) or the application (A1, A2, A.O.1, A.O.2) running on the external electronic device (O1, O2, O3) can be activated.

The head mounted device 101 according to one embodiment may position the pointer P on the second application A2 running on the first external electronic device O2 based on the user's gaze direction. The head mounted device 101 according to one embodiment may activate the second application A2 running on the first external electronic device O1 based on the user's gaze direction, as shown in FIG. 17A. .

The head mounted device 101 according to one embodiment moves the pointer P to the first application A1 based on the user's gaze direction moving to the first application A1, as shown in FIG. 17B. You can move. When the head mounted device 101 according to an embodiment receives a signal corresponding to an input key from the keyboard (I) before switching the activated application to the first application (A1) based on the movement of the gaze direction. , the signal corresponding to the received input key may be incorrectly transmitted to the previously activated second application (A2).

As shown in FIG. 17C, the head mounted device 101 according to an embodiment is delayed in switching between external electronic devices or applications to be activated, and thus changes the input key corresponding to the input key incorrectly transmitted to the second application A2. The signal can be transmitted to the first application (A1). For example, the head mounted device 101 may cut text input (eg, typing) into the second application and paste it into the first application.

Referring to FIG. 18, the head mounted device 101 according to one embodiment connects the activated external electronic device O1 to another external electronic device O2 based on the movement of a specific point E according to the user's gaze direction. ), and accordingly, the manipulation input received from at least one input device can be transmitted to the newly activated external electronic device O2.

In one embodiment, one or more external electronic devices (O1, O2) may have different operating systems (OS), and accordingly, input keys (e.g., shortcut keys) corresponding to specific manipulation inputs of the input device (I) , shortcut) may be different from each other.

The head mounted device 101 according to an embodiment may, when the activated external electronic device is switched between external electronic devices O1 and O2 having different operating systems, receive information from at least one input device (I). The signal corresponding to the input key may be converted and transmitted to at least one of the external electronic devices (O2).

For example, when the head mounted device 101 receives a specific input key (e.g., ctrl + c) from the keyboard (I) while the first external electronic device (O1) is activated, the head mounted device 101 uses a specific input key (e.g., ctrl + c). A signal corresponding to ctrl + c) can be transmitted to the first external electronic device O1. Thereafter, when the head mounted device 101 receives a specific input key (e.g., command + v) from the keyboard (I) while the activated external electronic device is switched to the second external electronic device (O2), the head mounted device 101 The input key converted to correspond to the second external electronic device O2 (e.g., ctrl + v) may be transmitted to the second external electronic device O2.

When the activated external electronic device is switched between external electronic devices O1 and O2 having different operating systems, the head mounted device 101 according to one embodiment is configured to switch the active external electronic device O2 corresponding to the switched external electronic device O2. A guide object related to an input key may be displayed on a display (e.g., display 440 of FIG. 4) corresponding to at least one input device (I).

19A to 19B are

flowcharts

1900a and 1900b of a method of operating the head mounted device 101 according to an embodiment of the present disclosure.

19A to 19B, in operation 1910, the head mounted device 101 (e.g., processor 410) according to an embodiment detects one or more external electronic devices through the second camera 432. O1, O2, O3) can be identified. In one embodiment, the head mounted device 101 may identify one or more external electronic devices O1, O2, and O3 located within the field of view (FOV) of the second camera 432.

In operation 1920, the head mounted device 101 according to an embodiment uses the communication interface 420 to allow one or more external electronic devices O1, O2, and O3 to display the markers M1 and M2 in a designated second manner. A signal requesting display can be transmitted to the server (S). In one embodiment, the server (S) may transmit a signal requesting to display the markers (M1, M2) in a designated second manner to one or more external electronic devices (O1, O2, O3) based on the received signal. there is. In one embodiment, one or more external electronic devices O1, O2, and O3 may display the markers M1 and M2 in a designated second method based on the received signal.

In operation 1930, the head mounted device 101 according to an embodiment may determine whether one or more identified external electronic devices O1, O2, and O3 are registered devices. For example, the head mounted device 101 has markers (M1, M2) corresponding to the identified one or more external electronic devices (O1, O2, O3) and markers (M1, M2) corresponding to the identified one or more external electronic devices (O1, O2, O3). It can be determined whether the corresponding information has been stored in advance.

In one embodiment, if the identified one or more external electronic devices (O1, O2, O3) are not registered devices (1930-No), the head mounted device 101, in operation 1935, registers the device through the communication interface 420. , Information on at least one external electronic device O1, O2, and O3 corresponding to the markers M1 and M2, respectively, can be obtained from the server S.

In one embodiment, if the identified one or more external electronic devices (O1, O2, O3) are registered devices (1930 - Yes), in operation 1940, the user's gaze direction is recognized through the first camera 431. You can.

The head mounted device 101 according to one embodiment may determine whether the user's gaze direction has been recognized in operation 1945. The head mounted device 101 according to an embodiment may repeatedly execute operation 1940 when the user's gaze direction is not recognized (1945-Yes).

When the user's gaze direction is recognized (1945 - Yes), the head mounted device 101 according to one embodiment may acquire an image or video captured through the second camera 432 in operation 1950.

In operation 1960, the head mounted device 101 according to an embodiment mounts a device in the gaze direction among one or more external electronic devices O1, O2, and O3 based on an image or video acquired through the second camera 432. At least some of the corresponding information can be identified. For example, the head mounted device 101 identifies at least a portion of one or more external electronic devices (O1, O2, O3) corresponding to the gaze direction, and displays a marker (M1) displayed on the identified external electronic device. By recognizing ,M2), at least some information corresponding to the identified external electronic device can be identified.

The head mounted device 101 according to an embodiment may activate at least a portion of one or more external electronic devices O1, O2, and O3 corresponding to the gaze direction. The head mounted device 101 according to an embodiment may activate at least one application executed on at least a portion of the one or more external electronic devices O1, O2, and O3 corresponding to the gaze direction.

The head mounted device 101 according to one embodiment, in operation 1970, based on the image or video acquired through the second camera 432, displays the designated markers M1 and M2 in the designated second manner. The screen of the external electronic device displayed in the first method may be displayed on the display 440.

The head mounted device 101 according to one embodiment may display at least a portion of the image or video acquired through the second camera 432 on the display display 440. The head mounted device 101 according to an embodiment displays the screen of an external electronic device displayed in a designated first method on the display display 440, and the markers M1 and M2 displayed in a designated second method are displayed on the display display 440. It can be excluded from (440).

The head mounted device 101 according to one embodiment may detect whether the user's gaze direction is moving in operation 1980. For example, the head mounted device 101 may detect that the user's gaze direction moves when the user's gaze direction moves beyond a specified range or when the user moves to another application or other external electronic device. .

When the head mounted device 101 according to an embodiment detects that the gaze direction has moved (operation 1980 - example), in operation 1985, one or more external electronic devices (O1, At least some of O2, O3) can be converted to other devices. The head mounted device 101 according to one embodiment may switch the identified application to another application based on movement of the gaze direction.

In operation 1990, the head mounted device 101 according to one embodiment may determine whether a manipulation input is obtained from at least one input device (I).

When the head mounted device 101 according to one embodiment obtains a manipulation input from at least one input device (I) (operation 1990 - Yes), in operation 1993, one or more external electronic devices (O1, O2, O3) ) Based on at least some of the information, a manipulation input may be transmitted to at least some of the one or more external electronic devices O1, O2, and O3 through the communication interface 420. The head mounted device 101 according to an embodiment may transmit a manipulation input obtained from at least one input device (I) to an external electronic device or application activated based on the user's gaze direction.

If a manipulation input is not obtained from at least one input device (I) (operation 1990-No), the head mounted device 101 according to one embodiment, in operation 1997, uses one or more identified devices based on the gaze direction. A virtual pointer P may be displayed on at least some of the external electronic devices O1, O2, and O3.

The head mounted device according to an embodiment of the present disclosure may include a communication interface 420, a camera module 430, the communication interface 420, a memory 130 for storing instructions, and a processor 410. . When executed by the processor 410, the instructions cause the head mounted device 101 to display a screen in a designated first manner through the camera module 430. O2, O3) can be identified. When executed by the processor 410, the instructions cause the head mounted device 101 to use the identified one or more external electronic devices O1, O2, and O3 through the communication interface 420. It may cause the server S to transmit a signal requesting to display the markers M1 and M2 in a designated second manner that is at least partially different from the designated first manner. The instructions, when executed by the processor 410, cause the head mounted device 101 to receive the information corresponding to the markers M1 and M2 from the server S through the communication interface 420. This may cause information on one or more external electronic devices (O1, O2, and O3) to be obtained respectively. When executed by the processor 410, the instructions cause the head mounted device 101 to correspond to the markers M1 and M2 based on the image or video acquired through the camera module 430. This can cause information on at least some of the one or more external electronic devices O1, O2, and O3 to be identified. When executed by the processor 410, the instructions cause the head mounted device 101 to, based on the information on at least some of the identified one or more external electronic devices O1, O2, and O3, at least A manipulation input obtained from one input device (I) may be caused to be transmitted to at least some of the one or more external electronic devices (O1, O2, and O3) through the communication interface 420.

In the head mounted device according to one embodiment, the camera module 430 includes a first camera 431 that recognizes the user's gaze direction and a second camera 432 having a shooting range including the gaze direction. You can. When executed by the processor 410, the instructions may cause the head mounted device 101 to recognize the user's gaze direction through the first camera 431. When executed by the processor 410, the instructions cause the head mounted device 101 to perform the one corresponding to the gaze direction based on the image or video captured through the second camera 432. This may cause identification of at least some of the external electronic devices O1, O2, and O3.

In the head mounted device according to one embodiment, the instructions, when executed by the processor 410, cause the head mounted device 101 to recognize the movement of the user's gaze direction through the camera module 430. It can cause it to happen. When executed by the processor 410, the instructions cause the head mounted device 101 to use the identified one or more external electronic devices O1, O2, and O3 based on the recognized movement of the gaze direction. This may cause at least some of the above to be switched to another device.

In the head mounted device according to one embodiment, the instructions, when executed by the processor 410, cause the head mounted device 101 to use the identified one or more external electronic devices O1, O2, and O3. When the manipulation input is generated through the at least one input device (I) as at least part of the operation of switching to another device, the identified one or more external electronic devices (O1, O2 ,O3) can cause at least some of the above to be maintained.

The head mounted device according to one embodiment may further include a display 440 that displays a screen corresponding to an image or video acquired through the camera module 430. When executed by the processor 410, the instructions cause the head mounted device 101 to display a marker displayed in the designated second method based on an image or video acquired through the camera module 430. The screen of the one or more external electronic devices (O1, O2, O3) displayed in the designated first manner excluding (M1, M2) may be caused to be displayed on the display 440.

In the head mounted device according to one embodiment, the instructions, when executed by the processor 410, cause the head mounted device 101 to use the one or more external electronic devices corresponding to the markers M1 and M2. At least as part of the operation of identifying the information of the at least some of the (O1, O2, O3), based on the image or video of the at least some of the identified one or more external electronic devices (O1, O2, O3), the one The coordinate system of at least some of the external electronic devices O1, O2, and O3 may be mapped.

In the head mounted device according to one embodiment, the at least one input device (I) may include a mouse (M). When executed by the processor 410, the instructions may cause the head mounted device 101 to recognize the user's gaze direction through the camera module 430. When executed by the processor 410, the instructions cause the head mounted device 101 to use at least some of the identified one or more external electronic devices O1, O2, and O3 based on the gaze direction. It can cause a pointer (P) to be displayed.

In the head mounted device according to one embodiment, the instructions, when executed by the processor 410, cause the head mounted device 101 to select one of the identified one or more external electronic devices O1, O2, and O3. As at least part of the operation of displaying the pointer P on the at least part, the pointer P may be caused to move based on a movement input through the mouse M or a movement in the gaze direction.

In the head mounted device according to one embodiment, the instructions, when executed by the processor 410, cause the head mounted device 101 to transmit the manipulation input to one or more external electronic devices (O1, O2) that have been identified. , O3), at least as part of the operation of transmitting to at least some of the identified one or more external electronic devices (O1, O2, O3) may cause transmission to at least some of the above.

In the head mounted device according to one embodiment, the at least one input device (I) may include a keyboard (K). The instructions, when executed by the processor 410, cause the head mounted device 101 to transmit the manipulation input to at least some of the identified one or more external electronic devices O1, O2, and O3. At least as part of the operation, a signal corresponding to an input key obtained from the keyboard K may be caused to be transmitted to at least some of the identified one or more external electronic devices O1, O2, and O3.

A method of operating a head mounted device according to an embodiment of the present disclosure includes the operation of identifying one or more external electronic devices (O1, O2, O3) that display a screen in a designated first manner through the camera module 430 ( 1910) may be included. A method of operating a head mounted device according to an embodiment includes, through the communication interface 420, the identified one or more external electronic devices (O1, O2, O3) using a designated second method that is at least partially different from the designated first method. It may include an operation 1920 of transmitting a signal requesting to display the markers M1 and M2 to the server S. A method of operating a head mounted device according to an embodiment includes the one or more external electronic devices (O1, O2, O3) corresponding to the markers (M1, M2) from the server (S) through the communication interface 420. ) may include an operation 1935 of acquiring information. A method of operating a head mounted device according to an embodiment includes, based on an image or video acquired through the camera module 430, the one or more external electronic devices (O1, O2) corresponding to the markers (M1, M2). ,O3) may include an operation 1960 of identifying at least some information. A method of operating a head mounted device according to an embodiment includes an operation obtained from at least one input device (I) based on the information on at least some of the identified one or more external electronic devices (O1, O2, O3). It may include an operation 1993 of transmitting an input to at least some of the one or more external electronic devices O1, O2, and O3 through the communication interface 420.

In a method of operating a head mounted device according to an embodiment, the camera module 430 includes a first camera 431 that recognizes the user's gaze direction and a second camera 432 that has a shooting range including the gaze direction. ) may be included. A method of operating a head mounted device according to an embodiment may include an operation 1940 of recognizing the user's gaze direction through the first camera 431. A method of operating a head mounted device according to an embodiment includes, based on an image or video captured through the second camera 432, the one or more external electronic devices (O1, O2, O3) corresponding to the gaze direction. It may include an operation 1960 of identifying at least some of the above.

A method of operating a head mounted device according to an embodiment may include an operation 1980 of recognizing a movement of the user's gaze direction through the camera module 430. A method of operating a head mounted device according to an embodiment includes switching at least some of the identified one or more external electronic devices (O1, O2, O3) to another device based on the recognized movement of the gaze direction. (1985) may be included.

In a method of operating a head mounted device according to an embodiment, the operation (1985) of switching the one or more external electronic devices (O1, O2, O3) to another device includes the operation (1985) of the at least one input device (I). When a manipulation input is generated, at least some of the identified one or more external electronic devices O1, O2, and O3 may be maintained despite the gaze direction moving.

A method of operating a head mounted device according to an embodiment is based on an image or video acquired through the camera module 430, excluding the markers M1 and M2 displayed in the designated second method. An operation 1970 of displaying the screen of the one or more external electronic devices O1, O2, and O3 displayed in this manner on the display 440 may be further included.

In a method of operating a head mounted device according to an embodiment, an operation (1960) of identifying information on at least some of the one or more external electronic devices (O1, O2, O3) corresponding to the markers (M1, M2) , Based on the image or video of the at least some of the identified one or more external electronic devices (O1, O2, O3), the coordinate system of at least some of the one or more external electronic devices (O1, O2, O3) is mapped. You can.

In a method of operating a head mounted device according to an embodiment, the at least one input device (I) may include a mouse (M). A method of operating a head mounted device according to an embodiment may include an operation 1940 of recognizing the user's gaze direction through the camera module 430. A method of operating a head mounted device according to an embodiment includes the operation of displaying a pointer (P) on at least some of the identified one or more external electronic devices (O1, O2, O3) based on the gaze direction (1997). ) may include.

In a method of operating a head mounted device according to an embodiment, the operation (1997) of displaying the pointer (P) on at least some of the identified one or more external electronic devices (O1, O2, O3) includes the mouse ( The pointer (P) can be moved based on a movement input through M) or movement in the gaze direction.

In the method of operating a head mounted device according to an embodiment, the operation (1993) of transmitting the manipulation input to at least some of the identified one or more external electronic devices (O1, O2, O3) includes the mouse (M) A movement input or a click input corresponding to the pointer P obtained from may be transmitted to at least some of the identified one or more external electronic devices O1, O2, and O3.

In a method of operating a head mounted device according to an embodiment, the at least one input device (I) may include a keyboard (K). In the method of operating a head mounted device according to an embodiment, the operation (1993) of transmitting the manipulation input to at least some of the identified one or more external electronic devices (O1, O2, O3) includes the keyboard (K) A signal corresponding to the input key obtained from may be transmitted to at least some of the identified one or more external electronic devices O1, O2, and O3.

In the storage medium storing computer-readable instructions according to an embodiment of the present disclosure, the instructions, when executed by the processor 410 of the head mounting device 101, cause the head mounting device 101 to , through the camera module 430, can cause identification of one or more external electronic devices (O1, O2, O3) that display the screen in a designated first manner. When executed by the processor 410 of the head mounted device 101, the instructions allow the head mounted device 101 to connect the identified one or more external electronic devices (O1, O2) through the communication interface 420. , O3) may cause the server S to transmit a signal requesting to display the markers M1 and M2 in a designated second manner that is at least partially different from the designated first manner. The instructions, when executed by the processor 410 of the head mounted device 101, cause the head mounted device 101 to receive the markers M1, This may cause information on the one or more external electronic devices (O1, O2, O3) corresponding to M2) to be obtained respectively. When executed by the processor 410 of the head mounted device 101, the instructions cause the head mounted device 101 to display the marker ( This may cause identification of at least some information among the one or more external electronic devices (O1, O2, O3) corresponding to M1, M2). When executed by the processor 410 of the head mounted device 101, the instructions cause the head mounted device 101 to use at least some of the identified one or more external electronic devices O1, O2, and O3. Based on the information, a manipulation input obtained from at least one input device (I) may be caused to be transmitted to at least some of the one or more external electronic devices (O1, O2, O3) through the communication interface (420). there is.

Electronic devices according to embodiments disclosed in this document may be of various types. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, electronic devices, or home appliances. Electronic devices according to embodiments of this document are not limited to the above-described devices.

The embodiments of this document and the terms used herein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various changes, equivalents, or replacements of the embodiments. In connection with the description of the drawings, similar reference numbers may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the relevant context clearly indicates otherwise. As used herein, “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “A Each of phrases such as “at least one of , B, or C” may include any one of the items listed together in the corresponding phrase, or any possible combination thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish one component from another, and to refer to those components in other respects (e.g., importance or order) is not limited. One (e.g., first) component is said to be “coupled” or “connected” to another (e.g., second) component, with or without the terms “functionally” or “communicatively.” Where mentioned, it means that any of the components can be connected to the other components directly (e.g. wired), wirelessly, or through a third component.

The term "module" used in embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as logic, logic block, component, or circuit, for example. It can be used as A module may be an integrated part or a minimum unit of the parts or a part thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

One embodiment of the present document is one or more instructions stored in a storage medium (e.g., built-in memory 136 or external memory 138) that can be read by a machine (e.g., electronic device 101). It may be implemented as software (e.g., program 140) including these. For example, a processor (e.g., processor 120) of a device (e.g., electronic device 101) may call at least one command among one or more commands stored from a storage medium and execute it. This allows the device to be operated to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves). This term refers to cases where data is stored semi-permanently in the storage medium. There is no distinction between temporary storage cases.

According to one embodiment, the method according to the embodiments disclosed in this document may be provided and included in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or through an application store (e.g. Play StoreTM) or on two user devices (e.g. It can be distributed (e.g. downloaded or uploaded) directly between smart phones) or online. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

According to one embodiment, each component (e.g., module or program) of the above-described components may include a single or plural entity, and some of the plurality of entities may be separately placed in other components. there is. According to one embodiment, one or more of the above-described corresponding components or operations may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components (eg, modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each component of the plurality of components identically or similarly to those performed by the corresponding component of the plurality of components prior to the integration. . According to embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, or omitted. Alternatively, one or more other operations may be added.

Claims

In the head mounted device 101,

communication interface 420;

Camera module 430;

a memory 130 that stores instructions; and

Includes a processor 410,

When executed by the processor 410, the instructions cause the head mounted device 101 to display a screen in a designated first manner through the camera module 430. Identify O2,O3),

Through the communication interface 420, the identified one or more external electronic devices (O1, O2, O3) request to display the markers (M1, M2) in a designated second method that is at least partially different from the designated first method. Transmit the signal to the server (S),

Obtaining information on the one or more external electronic devices (O1, O2, O3) corresponding to the markers (M1, M2) from the server (S) through the communication interface 420, respectively,

Based on the image or video acquired through the camera module 430, identify information on at least some of the one or more external electronic devices (O1, O2, O3) corresponding to the markers (M1, M2),

Based on the information on at least some of the identified one or more external electronic devices (O1, O2, O3), a manipulation input obtained from at least one input device (I) is transmitted to the one or more of the identified external electronic devices (O1, O2, O3) through the communication interface 420. causing transmission to at least some of the external electronic devices (O1, O2, O3),

Head mounted device (101).
According to claim 1,

The camera module 430 includes a first camera 431 that recognizes the user's gaze direction and a second camera 432 that has a shooting range including the gaze direction,

When executed by the processor 410, the instructions cause the head mounted device 101 to recognize the user's gaze direction through the first camera 431,

Based on the image or video captured through the second camera 432, causing identification of at least some of the one or more external electronic devices (O1, O2, O3) corresponding to the gaze direction,

Head mounted device (101).
According to claim 2,

When executed by the processor 410, the instructions cause the head mounted device 101 to recognize the movement of the user's gaze direction through the camera module 430,

Based on the recognized movement of the gaze direction, causing at least some of the identified one or more external electronic devices (O1, O2, O3) to be switched to another device,

Head mounted device (101).
According to claim 3,

The instructions, when executed by the processor 410, cause the head mounted device 101 to, at least as part of an operation to switch the identified one or more external electronic devices O1, O2, and O3 to another device, When the manipulation input is generated through the at least one input device (I), the at least some of the identified one or more external electronic devices (O1, O2, O3) are maintained despite the gaze direction moving. causing,

Head mounted device (101).
According to any one of claims 1 to 4,

It further includes a display display 440 that displays a screen corresponding to the image or video acquired through the camera module 430,

When executed by the processor 410, the instructions cause the head mounted device 101 to display a marker displayed in the designated second method based on an image or video acquired through the camera module 430. causing the screen of the one or more external electronic devices (O1, O2, O3) displayed in the specified first manner excluding (M1, M2) to be displayed on the display display 440,

Head mounted device (101).
According to any one of claims 1 to 5,

When executed by the processor 410, the instructions cause the head mounted device 101 to use the at least one of the one or more external electronic devices O1, O2, and O3 corresponding to the markers M1 and M2. At least as part of the operation of identifying some information, based on the image or video of the at least some of the identified one or more external electronic devices (O1, O2, O3), ) causing to map at least some of the coordinate systems,

Head mounted device (101).
According to any one of claims 1 to 6,

The at least one input device (I) includes a mouse (M),

When executed by the processor 410, the instructions cause the head mounted device 101 to recognize the user's gaze direction through the camera module 430,

Based on the gaze direction, causing a pointer (P) to be displayed on at least some of the identified one or more external electronic devices (O1, O2, O3),

Head mounted device (101).
According to claim 7,

When executed by the processor 410, the instructions cause the head mounted device 101 to place the pointer P on at least some of the identified one or more external electronic devices O1, O2, and O3. At least as part of the display operation, causing the pointer (P) to move based on a movement input through the mouse (M) or a movement in the gaze direction,

Head mounted device (101).
According to any one of claims 7 to 8,

The instructions, when executed by the processor 410, cause the head mounted device 101 to transmit the manipulation input to at least some of the identified one or more external electronic devices O1, O2, and O3. At least as part of the operation, causing a movement input or a click input corresponding to the pointer (P) obtained from the mouse (M) to be transmitted to the at least some of the identified one or more external electronic devices (O1, O2, O3) doing,

Head mounted device (101).
According to any one of claims 1 to 9,

The at least one input device (I) includes a keyboard (K),

The instructions, when executed by the processor 410, cause the head mounted device 101 to transmit the manipulation input to at least some of the identified one or more external electronic devices O1, O2, and O3. At least as part of the operation, causing a signal corresponding to an input key obtained from the keyboard (K) to be transmitted to the at least some of the identified one or more external electronic devices (O1, O2, O3),

Head mounted device (101).
In the method of operating the head mounted device 101,

An operation 1910 of identifying one or more external electronic devices O1, O2, and O3 that display a screen in a designated first manner through the camera module 430;

A signal requesting, through the communication interface 420, that the identified one or more external electronic devices (O1, O2, O3) display the markers (M1, M2) in a designated second method that is at least partially different from the designated first method. An operation of transmitting to the server (S) (1920);

An operation (1935) of acquiring information on the one or more external electronic devices (O1, O2, O3) corresponding to the markers (M1, M2) from the server (S) through the communication interface (420);

An operation of identifying information on at least some of the one or more external electronic devices (O1, O2, O3) corresponding to the markers (M1, M2) based on the image or video acquired through the camera module 430 ( 1960); and

Based on the information on at least some of the identified one or more external electronic devices (O1, O2, O3), a manipulation input obtained from at least one input device (I) is transmitted to the one or more of the identified external electronic devices (O1, O2, O3) through the communication interface 420. Including an operation (1993) of transmitting to at least some of the external electronic devices (O1, O2, O3),

How it works.
According to claim 11,

The camera module 430 includes a first camera 431 that recognizes the user's gaze direction and a second camera 432 that has a shooting range including the gaze direction,

An operation 1940 of recognizing the user's gaze direction through the first camera 431; and

An operation 1960 of identifying at least some of the one or more external electronic devices O1, O2, and O3 corresponding to the gaze direction based on the image or video captured through the second camera 432. containing,

How it works.
According to claim 12,

An operation (1980) of recognizing the movement of the user's gaze direction through the camera module 430; and

Based on the recognized movement of the gaze direction, the operation (1985) of switching at least some of the identified one or more external electronic devices (O1, O2, O3) to another device,

How it works.
According to claim 13,

The operation (1985) of switching the one or more external electronic devices (O1, O2, O3) to another device is performed even when the gaze direction moves when the manipulation input is generated through the at least one input device (I). Nonetheless, maintaining at least some of the identified one or more external electronic devices (O1, O2, O3),

How it works.
In the storage medium storing computer-readable instructions, the instructions, when executed by the processor 410 of the head mounted device 101, cause the head mounted device 101 to:

Identifying one or more external electronic devices (O1, O2, O3) that display a screen in a designated first manner through the camera module 430,

A signal requesting, through the communication interface 420, that the identified one or more external electronic devices (O1, O2, O3) display the markers (M1, M2) in a designated second method that is at least partially different from the designated first method. Send to the server (S),

Obtaining information on the one or more external electronic devices (O1, O2, O3) corresponding to the markers (M1, M2) from the server (S) through the communication interface 420, respectively,

Based on the image or video acquired through the camera module 430, identify information on at least some of the one or more external electronic devices (O1, O2, O3) corresponding to the markers (M1, M2),

Based on the information on at least some of the identified one or more external electronic devices (O1, O2, O3), a manipulation input obtained from at least one input device (I) is transmitted to the one or more of the identified external electronic devices (O1, O2, O3) through the communication interface 420. causing transmission to at least some of the external electronic devices (O1, O2, O3),

storage media.