RU2802281C2 - Application launch method and electronic device - Google Patents

Abstract

FIELD: computer technology.

SUBSTANCE: method for launching an application applicable to an electronic device having at least one camera and a touch screen, the method comprising displaying, by the electronic device, a first picture of the first application, the first application supporting user control of the first application through a touch operation on the touch screen, capturing by the electronic device of the user's first image using at least one camera, and recognizing the first motion reading related to the position of the user's body in the first image based on the movement coordinates of the bones converted from the first user image, determining by the electronic device the first information of the touch operation corresponding to the first reading motion, based on a predetermined correspondence between the motion sensing and the touch operation, wherein the first touch operation information contains the type of the first touch operation and the position at which the first touch operation is performed on the touch screen, and providing the electronic device with a response from the first application.

EFFECT: improving the accuracy of reading the motion command.

10 cl, 24 dwg, 3 tbl

Description

FIELD OF TECHNOLOGY

[0001] This application relates to the field of electronic technology and, in particular, to a method of launching an application and an electronic device.

BACKGROUND OF THE INVENTION

[0002] Games have become an important entertainment function for people. The motion sensing game is an electronic game that is performed (operated) by the player by changing the motion sensings (perceptions) and trains the body while entertaining the player, thereby being popular with the players.

[0003] However, most existing motion sensing games require a body motion sensing peripheral in addition to the game console. The peripheral device captures an image of the player, and then recognizes the player's motion reading based on the image, and the game console performs an operation corresponding to the motion reading. It can be seen that the player needs to additionally purchase a dedicated peripheral in order to experience the motion sensing gameplay. In addition, a game maker needs to specifically develop a motion sensing game or specifically develop a version supporting motion sensing gameplay for an existing game. In addition, since the software and hardware capabilities of different game consoles are different, the motion sensing supported is different, resulting in a relatively high development cost for a motion sensing game experience.

SUMMARY OF THE INVENTION

[0004] This application provides a method for launching an application and an electronic device such that an application supporting touch control can also be controlled by a motion sensing (motion-sensing action, motion-sensing action) of a user, thereby helping to improve user experience and reduce development costs.

[0005] To achieve the above goal, the following technical solutions are used in the embodiments of the present application.

[0006] According to a first aspect, a method for launching an application applicable to an electronic device having at least one camera and a touch screen is provided. The method includes: displaying a first picture of a first application by the electronic device, wherein the first application supports user control of the first application through a touch operation on the touch screen; capturing by the electronic device a first image of the user using at least one camera and recognizing the first reading of the user's movement in the first image; determining, by the electronic device, first reading operation information corresponding to the first motion reading, where the first touch operation information includes the type of the first touch operation and the position at which the first touch operation is performed on the touch screen; and performing by the electronic device an operation corresponding to the first touch operation in response to determining the first information of the touch operation corresponding to the first motion reading.

[0007] The first application may be any application installed on the electronic device, and the application may support a touch operation for control. For example, the first application may be a game application.

[0008] The operation corresponding to the first touch operation may be to display another picture, or may be to output a specific voice, or may be to output another specific signal, or the like. The operation corresponding to the first touch operation is not limited to the embodiments of this application.

[0009] It can be seen that in this application, the motion sensing is mapped to the touch operation natively supported by the first application, so that the function for which the first application supports motion sensing gameplay can also be implemented without modifying the first application's own code, thereby improving the experience of using the first application. That is, in the method presented in this application, the motion sensing gameplay support function does not need to be developed separately for each first application, which is advantageous for reducing development costs.

[0010] In an exemplary implementation, the electronic device performing an operation corresponding to the first touch operation includes: displaying by the electronic device a second picture of the first application, where the second picture is the same as the picture displayed by the electronic device in response to the user performing the first touch operation on the first picture.

[0011] In an exemplary implementation, determining by the electronic device the first touch operation information corresponding to the first motion reading includes: calling the application infrastructure layer interface by the electronic device and determining the first touch operation information corresponding to the first motion reading.

[0012] It can be seen that in this application, through a system service, motion sensing can be mapped to a touch operation natively supported by the first application to realize a function for which the first application supports motion sensing gameplay. That is, each first application supporting the touch operation can implement a motion-sensing gameplay support function by calling a system service.

[0013] In an exemplary implementation, before the electronic device captures the first image of the user by calling at least one camera, the method further includes: receiving the electronic device an operation in which the user activates the first function; and activating, by the electronic device, a first function in response to an operation in which the user activates the first function, where the first function is a function whose first application supports the user to control the first application by motion sensing. It can be seen that the user can select whether to activate the motion sensing gameplay function.

[0014] In a possible implementation, after the electronic device activates the first function, the method further includes: adding a layer to the first application picture by the electronic device, where the layer includes an opaque area and a transparent area, and the opaque area displays the first function setting interface. Therefore, this application provides a method for displaying a setting interface of the first function in order to facilitate the setting of the first function by the user.

[0015] In an exemplary implementation, the method further includes: receiving by the electronic device an operation in which a user sets a correspondence between a motion sense and a touch operation using the first function setting interface, where the touch operation includes a type of touch operation and a position in which touch operation on the touch screen. It can be seen that in the method provided in this application, the user can adjust the correspondence between motion sensing and touch operation to achieve personalization and variety of motion sensing gaming experience.

[0016] In an exemplary implementation, the type of tap operation includes at least one of a tap operation, a slide operation, a double tap operation, a tap and hold operation, a finger knuckle tap operation, a multipoint tap operation, and a predefined gesture.

[0017] In an exemplary implementation, the method further includes: storing, by the electronic device, a user setting to match between the motion sensing and the touch operation; and adding a second layer by the electronic device to the first application picture, where the second layer includes an opaque area and a transparent area, and the opaque area displays a preview window of at least one camera. Therefore, the user can view the motion sensing of the user with the preview window of the camera, thereby improving the accuracy of using the motion sensing and improving the user's experience of use.

[0018] In a possible implementation, at least one camera includes a depth camera.

[0019] In a possible implementation, the method further includes: activating the screen mirroring function by the electronic device and instructing another electronic device that is connected to said electronic device and has a screen to display a picture of the electronic device. It can be seen that the electronic device can project the picture displayed on the screen onto another electronic device with a large screen, which is useful for improving the user's visual experience.

[0020] According to a second aspect, an electronic device is provided, including: at least one processor, a touch screen, and at least one camera, the touch screen and at least one camera being connected to the processor. The touch screen is configured to display the first picture of the first application, wherein the first application supports user control of the first application through a touch operation on the touch screen; at least one camera is configured to capture a first image of the user; and at least one processor is configured to recognize the first reading of the user's movement in the first image; determining first touch operation information corresponding to the first motion reading, where the first touch operation information includes the type of the first touch operation and the position at which the first touch operation is performed on the touch screen; and performing an operation corresponding to the first touch operation in response to determining the first touch operation information corresponding to the first motion reading.

[0021] The first application may be, for example, a game application.

[0022] In an exemplary implementation, performing an operation corresponding to the first touch operation specifically includes: operating the touch screen to display a second picture of the first application, where the second picture is the same as the picture displayed by the electronic device in response to which the user performs the first touch operation on the first picture.

[0023] In an exemplary implementation, determining the first touch operation information corresponding to the first motion reading specifically includes: calling an application infrastructure layer interface of the electronic device and determining the first touch operation information corresponding to the first motion reading.

[0024] In an exemplary implementation, at least one processor is further configured to: receive an operation in which the user activates the first function; and activating the first function in response to an operation in which the user activates the first function, where the first function is a function in which the first application supports the user to control the first application by motion sensing.

[0025] In an exemplary implementation, at least one processor is further configured to add a layer to the first application picture, where the layer includes an opaque region and a transparent region, and the opaque region displays the first function setting interface.

[0026] In an exemplary implementation, the at least one processor is further configured to receive an operation in which a user sets a correspondence between a motion sensing and a touch operation using a first function setting interface, where the touch operation includes a type of touch operation and a position, in which the touch operation is performed on the touch screen.

[0027] In an exemplary implementation, the tap operation type includes at least one of a tap operation, a slide operation, a double tap operation, a tap and hold operation, a finger knuckle tap operation, a multipoint tap operation, and a predefined gesture.

[0028] In an exemplary implementation, at least one processor is further configured to store a user setting for matching between a motion sensing and a touch operation; and adding a second layer to the first application picture, where the second layer includes an opaque area and a transparent area, and the opaque area displays a preview window of at least one camera.

[0029] In a possible implementation, at least one camera includes a depth camera.

[0030] In an exemplary implementation, at least one processor is further configured to activate the screen mirroring function and cause another electronic device that is connected to the electronic device and has a screen to display a picture of the electronic device.

[0031] According to a third aspect, a computer storage medium is provided including computer instructions, where the computer instructions, when run on a terminal, cause the terminal to execute the method according to the first aspect and any of its possible implementations.

[0032] According to a fourth aspect, a computer program product is provided, wherein the computer program product, when run on the computer, causes the computer to execute the method according to the first aspect and any of its possible implementations.

[0033] According to a fifth aspect, there is provided a chip system including at least one processor, wherein when the at least one processor executes instructions, the at least one processor executes the method of the first aspect and any of its possible implementations.

BRIEF DESCRIPTION OF THE DRAWINGS

[0034] FIG. 1 is a schematic structural diagram 1 of an electronic device according to an embodiment of this application.

[0035] FIG. 2 is a schematic structural diagram 2 of an electronic device according to an embodiment of this application.

[0036] FIG. 3A is a schematic representation of some user interfaces of an electronic device in accordance with an embodiment of this application.

[0037] FIG. 3B is a schematic representation of some more user interfaces of an electronic device in accordance with an embodiment of this application.

[0038] FIG. 3C is a schematic representation of some more user interfaces of an electronic device in accordance with an embodiment of this application.

[0039] FIG. 4A is a schematic representation of some more user interfaces of an electronic device in accordance with an embodiment of this application.

[0040] FIG. 4B is a schematic representation of some more user interfaces of an electronic device in accordance with an embodiment of this application.

[0041] FIG. 4C is a schematic representation of some more user interfaces of an electronic device in accordance with an embodiment of this application.

[0042] FIG. 5A is a schematic representation of some more user interfaces of an electronic device in accordance with an embodiment of this application.

[0043] FIG. 5B is a schematic representation of some more user interfaces of an electronic device in accordance with an embodiment of this application.

[0044] FIG. 5C is a schematic representation of some more user interfaces of an electronic device in accordance with an embodiment of this application.

[0045] FIG. 5D is a schematic representation of some more user interfaces of an electronic device in accordance with an embodiment of this application.

[0046] FIG. 5E is a schematic representation of some more user interfaces of an electronic device in accordance with an embodiment of this application.

[0047] FIG. 5F is a schematic representation of some more user interfaces of an electronic device in accordance with an embodiment of this application.

[0048] FIG. 5G is a schematic representation of some more user interfaces of an electronic device according to an embodiment of this application.

[0049] FIG. 5H is a schematic representation of some more user interfaces of an electronic device in accordance with an embodiment of this application.

[0050] FIG. 5I is a schematic representation of some more user interfaces of an electronic device according to an embodiment of this application.

[0051] FIG. 6A is a schematic representation of a method for recognizing bone points in a portrait according to an embodiment of this application.

[0052] FIG. 6B is a schematic representation of another method for recognizing bone points in a portrait according to an embodiment of this application.

[0053] FIG. 6C is a schematic representation of another method for recognizing bone points in a portrait according to an embodiment of this application.

[0054] FIG. 6D is a schematic diagram of another method for recognizing bone points in a portrait according to an embodiment of this application.

[0055] FIG. 6E is a diagram of another method for recognizing bone points in a portrait according to an embodiment of this application.

[0056] FIG. 6F is a schematic diagram of another method for recognizing bone points in a portrait according to an embodiment of this application.

[0057] FIG. 7 is a schematic block diagram of a chip system according to an embodiment of this application.

Description of Embodiments

[0058] In the descriptions of the embodiments of this application, unless otherwise indicated, "/" represents OR, meaning, for example, that A/B can be A or B; and "and/or" in this specification is simply an association relation to describe the associated objects and means that three relations can exist. For example, A and/or B could represent the following three cases: there is only A, there are both A and B, and there is only B.

[0059] The terms "first" and "second" mentioned below are used for purposes of description only and should not be construed as indicating or implying the relative importance or implying the number of said technical characteristics. Therefore, a function limited to "first" or "second" may explicitly or implicitly include one or more functions. In the description of the embodiments of this application, unless otherwise indicated, "many" means two or more.

[0060] In embodiments of this application, the word "exemplary" or "for example" is used to represent an example, illustration, or description. Any embodiment or design scheme described as "exemplary" or "exemplary" in the embodiments of this application should not be explained as being more preferable or having more advantages than another embodiment or design scheme. In particular, the use of the word "exemplary" or "for example", etc. is intended to represent a related concept in a specific way.

[0061] For example, the electronic device in this application can be a mobile phone, a tablet computer, a personal computer (PC), a personal digital assistant (PDA), a smart watch, a netbook, a wearable electronic device, or an augmented device. reality (augmented reality, AR), virtual reality device (virtual reality, VR), on-board device, smart car, smart speaker, robot, etc. The specific shape of the electronic device is not specifically limited in this application.

[0062] FIG. 1 is a block diagram of the electronic device 100.

[0063] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charge control module 140, a power control module 141, a battery 142, an antenna 1 , antenna 2, mobile module 150, wireless module 160, audio module 170, speaker 170A, handset 170V, microphone 170C, headset jack 170D, sensor module 180, key 190, motor 191, indicator 192, camera 193, screen 194 display, a subscriber identity module (SIM) card interface 195, and the like. The sensor module 180 may include a pressure sensor 180A, a gyro sensor 180B, an atmospheric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, an optical proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, and a touch sensor 180K, ambient light sensor 180L, bone conduction sensor 180M, and the like.

[0064] It can be understood that the exemplary structure in this embodiment of the present invention imposes no particular limitation on the electronic device 100. In some other embodiments of this application, the electronic device 100 may include more or fewer components than shown in the figure, or some components may be combined, or some components may be separated, or components may be arranged differently. The components in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0065] Processor 110 may include one or more processing units. For example, processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor ( digital signal processor (DSP), baseband processor, and/or neural-network processing unit (NPU). The various processing units may be independent components or may be integrated into one or more processors.

[0066] The controller may generate an operation control signal in accordance with the instruction opcode and the sequence signal to complete the instruction fetch and execution control.

[0067] Memory may be further configured in processor 110 to store instructions and data. In some embodiments, the memory in processor 110 is a cache. The memory may store an instruction or data that has just been used or cycled by the processor 110. If the processor 110 needs to use the instruction or data again, the processor may directly recall the instruction or data from memory to avoid re-access and reduce the processor 110 timeout, thereby increasing system efficiency.

[0068] In some embodiments of this application, the processor 110 may analyze the image captured by the camera 193 to determine the motion reading (motion-reading action-sensing motion) of the user and determining whether the reading (perception) of the user's motion is a predefined reading (perception ) movement. Then, a target game touch operation corresponding to the predetermined motion sensing is found, and an instruction of the corresponding touch operation is sent to the target game application, so that the target game application performs an appropriate response to achieve the effect that the user controls the target game using the motion sensing.

[0069] In some embodiments, processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated Circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver/transmitter interface (universal asynchronous receiver/transmitter, UART), mobile industry processor interface (MIPI), general-purpose input/output (GPIO), subscriber identity module (SIM) ), a universal serial bus (USB) interface, and/or the like.

[0070] The I2C interface is a two-way serial clock bus and includes a serial data line (serial data line, SDA) and a serial clock line (serail clock line, SCL). In some embodiments, processor 110 may include multiple I2C bus groups. The processor 110 may be connected to a touch sensor 180K, a charger, a flash, a camera 193, and the like. using various I2C bus interfaces. For example, processor 110 may be coupled to touch sensor 180K via an I2C interface such that processor 110 communicates with touch sensor 180K via an I2C bus interface to implement the touch function of electronic device 100.

[0071] The I2S interface may be used for audio communication. In some embodiments, processor 110 may include multiple I2S bus groups. The processor 110 may be connected to the audio module 170 using the I2S bus to communicate between the processor 110 and the audio module 170. In some embodiments, the audio module 170 may transmit an audio signal to the wireless module 160 using the I2S interface to implement call answering functionality using a Bluetooth headset.

[0072] The PCM interface can also be used for audio communications, as well as sampling, quantization, and analog coding. In some embodiments, audio module 170 may be connected to wireless communication module 160 using a PCM bus interface. In some embodiments, the audio module 170 may alternatively transmit an audio signal to the wireless communication module 160 using the PCM interface to implement call answering functionality using a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0073] The UART interface is a universal serial data bus and is used for asynchronous communication. The bus may be a talkback bus. The bus converts the data to be transferred between serial communication and parallel communication. In some embodiments, the implementation of the UART interface is typically configured to connect the processor 110 to the wireless module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless module 160 using the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 may transmit an audio signal to the wireless module 160 using a UART interface to implement music playback functionality using a Bluetooth headset.

[0074] The MIPI interface can be configured to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI ) and so on. In some embodiments, the processor 110 communicates with the camera 193 using the CSI interface to implement the photographing function of the electronic device 100. The processor 110 communicates with the display screen 194 using the DSI interface to implement the display function of the electronic device 100.

[0075] The GPIO interface can be configured using software. The GPIO interface can be configured to send a control signal or can be configured to send a data signal. In some embodiments, the GPIO interface may be configured to connect processor 110 to camera 193, display screen 194, wireless module 160, audio module 170, sensor module 180, and the like. The GPIO interface can also be configured as I2C interface, I2S interface, UART interface, MIPI interface, etc.

[0076] The USB interface 130 is an interface that conforms to the specification of the USB standard, and may be a mini USB interface, a micro USB interface, a USB Type C interface, or the like, among others. The USB interface 130 may be configured to connect to a charger to charge the electronic device 100, or may be used to transfer data between the electronic device 100 and a peripheral device, or may be configured to connect to a headset to play audio through the headset. Alternatively, the interface may be configured to connect to another electronic device, such as an AR device.

[0077] It can be understood that the interface connection relationship between modules shown in this embodiment of the present invention is merely an example for description and is not a limitation on the design of the electronic device 100. In some other embodiments of this application, the electronic device 100 may alternatively use a connection method interface other than the connection method in the previous embodiment, or use a combination of a plurality of interface connection methods.

[0078] The charge control module 140 is configured to receive charging input from the charger. The charger may be a wireless charger or may be a wired charger. In some wired charging embodiments, charge control module 140 may receive charging input from a wired charger using USB interface 130. In some wireless charging embodiments, charge control module 140 may receive wireless charging input using the wireless charging coil of electronic device 100. Charge control module 140 may supply power to the electronic device using power control module 141 while battery 142 is being charged.

[0079] The power control module 141 is configured to be connected to the battery 142, the charge control module 140, and the processor 110. The power control module 141 receives input from the battery 142 and/or the charge control module 140 to supply power to the processor 110, the internal memory 121, a display screen 194, a camera 193, a wireless communication module 160, and the like. The power management module 141 can be further configured to monitor parameters such as battery capacity, battery cycles, and battery health (leakage and impedance). In some other embodiments, power management module 141 may alternatively be located on processor 110. In some other embodiments, power management module 141 and charge management module 140 may be further configured on the same device.

[0080] The wireless communication function of the electronic device 100 may be implemented using antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, baseband processor, and the like.

[0081] Antenna 1 and antenna 2 are configured to transmit and receive an electromagnetic wave signal. Each antenna of electronic device 100 may be configured to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 may be multiplexed as a wireless LAN diversity antenna. In some other embodiments, the antenna may be used in conjunction with a tuning switch.

[0082] The mobile communication module 150 may provide a wireless communication solution such as 2G/3G/4G/5G applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA), etc. The mobile communication unit 150 may receive the electromagnetic wave through the antenna 1, perform processing such as filtering and amplifying the received electromagnetic wave, and transmit the processed electromagnetic wave to the modem processor for demodulation. The mobile communication module 150 may further amplify the signal modulated by the modem processor and convert the signal into an electromagnetic wave for radiation through the antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be located in the processor 110. In some embodiments, implementation at least some of the functional modules of the mobile communication module 150 and at least some of the processor 110 modules may be located in the same component.

[0083] The modem processor may include a modulator and a demodulator. The modulator is configured to modulate a low frequency baseband signal to be sent into a medium high frequency signal. The demodulator is configured to demodulate the received electromagnetic wave signal into a low frequency baseband signal. The demodulator then passes the demodulated low frequency baseband signal to the baseband processor for processing. The low frequency baseband signal is processed by the baseband processor and then transmitted to the application processor. The application processor outputs an audio signal through an audio device (which is not limited to speaker 170A, handset 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In some other embodiments, the modem processor may be independent of the processor 110, and the modem processor and the mobile communication module 150 or other functional module may be located in the same component.

[0084] The wireless communication unit 160 can provide a wireless communication solution including a wireless local area network (WLAN) (such as a wireless fidelity network (Wi-Fi)), Bluetooth (Bluetooth, BT ), and global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (infrared, IR), etc. for application to electronic device 100. Wireless communication module 160 may be one or more devices incorporating at least one communication processing module. The wireless communication module 160 receives the electromagnetic wave with the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signal, and sends the processed signal to the processor 110. The wireless communication module 160 may alternatively receive the signal to be sent from the processor 110, perform frequency modulation and amplification for the signal to be sent, and convert the signal into an electromagnetic wave for radiation with antenna 2.

[0085] In some embodiments, antenna 1 and mobile communication module 150 of electronic device 100 are connected, and antenna 2 and wireless communication module 160 are connected such that electronic device 100 can communicate with a network and another device using wireless technology. Wireless technology may include global system for mobile communications (GSM), global system for mobile communications (GPRS), code division multiple access (CDMA) , wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long-term evolution (LTE), BT, GNSS, WLAN, NFC, FM, IR technology and/or the like. GNSS may include a global positioning system (global positioning system, GPS), a global navigation satellite system (global navigation satellite system, GLONASS), a beidou navigation satellite system (BDS), a quasi-zenith satellite system (quasi-zenith satellite system, QZSS) and/or satellite based augmentation systems (SBAS).

[0086] The electronic device 100 implements a display function using a graphics processing unit (GPU), a display screen 194, an application processor, and the like. The GPU is an image processing microprocessor and is connected to the display screen 194 and the application processor. The GPU is configured to perform mathematical and geometric calculations and render graphics. The processor 110 may include one or more GPUs and execute software instructions to generate or modify display information.

[0087] The display screen 194 is configured to display an image, video, or the like. The display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode ( flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diodes (QLED), etc. In some embodiments, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0088] The electronic device 100 can implement a photographing function using an ISP, a camera 193, a video codec, a graphics processing unit (GPU), a display screen 194, an application processor, and the like.

[0089] The ISP is configured to process the data returned by the camera 193. For example, the shutter is activated during photography. Light is transmitted to the photosensitive element of the camera through the lens, and the optical signal is converted into an electrical signal. The photosensitive element of the camera transmits an electrical signal to the ISP for processing, hence the electrical signal is converted into an image visible to the naked eye. The ISP can also optimize the algorithm for noise points, brightness, and skin tone. The ISP can also optimize settings such as exposure and scene color temperature. In some embodiments, the ISP may be located in chamber 193.

[0090] Camera 193 is configured to capture a still image or video. An optical image of the object is generated through the lens and projected onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert the electrical signal into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in standard RGB and YUV formats. In some embodiments, electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0091] In embodiments of this application, when the target game is launched on a mobile phone, the camera 193 may be called to capture an image of the user, such that the processor 110 analyzes the captured image to determine a read of the user's motion.

[0092] The digital signal processor is configured to process a digital signal, and in addition to the digital image signal, the digital signal processor may further process another digital signal. For example, when the electronic device 100 performs frequency selection, the digital signal processor is configured to perform a Fourier transform or the like. with regard to frequency energy.

[0093] The video codec is configured to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, the electronic device 100 can play back or record video in a variety of coding formats, such as moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, and MPEG 4.

[0094] The NPU is a neural-network (NN) computing processor that quickly processes input information using a biological neural network structure such as a transmission mode between human brain nerve cells, and can further continuously perform self-learning. The NPU may be used to implement an application such as intelligent recognition of the electronic device 100, such as image recognition, face recognition, voice recognition, and text understanding.

[0095] In some embodiments of this application, the NPU may analyze the image captured by the camera 193 and may determine a portrait in the image using, for example, portrait recognition technology. Subsequently, the positions of joints such as elbows, wrists and knees in the portrait are determined by bone recognition technology (the process of estimating the positions of human body joint points in an image or video using computer vision technology), and the positioned points of the joints can finally be to form an image of the bones of the human body, which can reflect information about the current posture (ie, motion readings) of the human body, and the like.

[0096] The external memory interface 120 can be configured to connect to an external memory card, such as a micro SD card, to expand the storage capabilities of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement a data storage function such as saving a file such as music or video on an external memory card.

[0097] The internal memory 121 may be configured to store the program code executable on the computer, and the executable program code includes an instruction. The internal memory 121 may include a program storage area and a data storage area. The program storage area may store an operating system, an application required by at least one function (eg, a voice playback function or a picture playback function), and the like. The data storage area may store data (for example, audio data and an address book) and the like created by using the electronic device 100. In addition, the internal memory 121 may include a high-speed random access memory or may include a non-volatile memory such as as at least one memory on a magnetic disk, flash memory or universal flash storage (universal flash storage, UFS). The processor 110 executes instructions stored in the internal memory 121 and/or instructions stored in the memory located in the processor to perform various functional applications and data processing of the electronic device 100.

[0098] The electronic device 100 can realize an audio function such as playing and recording music using an audio module 170, a speaker 170A, a handset 170B, a microphone 170C, a headset jack 170D, an application processor, and the like.

[0099] The audio module 170 is configured to convert digital audio information to an analog audio output signal, and is further configured to convert an analog audio input to a digital audio signal. The audio module 170 may be further configured to encode and decode the audio signal. In some embodiments, audio module 170 may be located in processor 110, or some functional modules of audio module 170 may be located in processor 110.

[00100] Speaker 170A, also referred to as a "speaker", is configured to convert an electrical audio signal into an audio signal. Music can be listened to or a call answered hands-free using the speaker 170A in the electronic device 100.

[00101] A 170V handset, also referred to as a "receiver", is configured to convert an electrical audio signal into an audio signal. When the electronic device 100 is being used to answer a call or receive voice information, the handset 170B may be brought close to the human ear to receive the voice information.

[00102] Microphone 170C, also referred to as a "microphone" or "microphone", is configured to convert an audio signal into an electrical signal. When making a call or sending voice information, the user may speak with their mouth close to the microphone 170C to input an audio signal into the microphone 170C. At least one microphone 170C may be located in the electronic device 100. In some other embodiments, two microphones 170C may be located in the electronic device 100 to capture the audio signal and implement the noise canceling function. In some other embodiments, electronic device 100 may alternatively have three, four, or more microphones 170C for audio capture, noise reduction, sound source recognition, directional recording, and the like.

[00103] The headset jack 170D is configured to connect to a wired headset. Headset connector 170D may be a USB interface 130 or may be a 3.5 mm open mobile terminal platform (OMTP) interface or a standard cellular telecommunications industry association of the USA (CTIA) interface .

[00104] The pressure sensor 180A is configured to read the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A may be located on the display screen 194. There are many types of pressure sensors 180A, such as a resistive pressure sensor, an inductive pressure sensor, and a capacitive pressure sensor. The capacitive pressure sensor may include at least two parallel plates with conductive materials. When a force is applied to the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the force of pressure based on the capacitance change. When a touch operation is performed on the display screen 194, the electronic device 100 detects the strength of the touch operation using the pressure sensor 180A. The electronic device 100 may further calculate the touch position based on the detection signal of the pressure sensor 180A. In some embodiments, touch operations that are performed at the same touch position but have a different touch operation strength may correspond to different operation instructions. For example, when a touch operation whose touch operation strength is smaller than the first pressure threshold is performed on the short message application icon, a short message checking instruction is executed. When a touch operation is performed on the short message application icon, the touch operation strength of which is greater than or equal to the first pressure threshold value, an instruction to create a new short message is executed.

[00105] Gyro sensor 180B may be configured to determine the movement position of electronic device 100. In some embodiments, the angular velocity of electronic device 100 about three axes (namely x, y, and z axes) may be determined using gyroscope sensor 180B. The gyroscope sensor 180B can be used to stabilize the image during photography. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle at which the electronic device 100 is shaking, and calculates, based on the angle, the distance that the lens module must compensate, and allows the lens to reverse the shaking of the electronic device 100, thereby realizing image stabilization. . The gyroscope sensor 180B can also be used in navigation and in a motion sensing game scenario.

[00106] The barometric pressure sensor 180C is configured to measure barometric pressure. In some embodiments, the electronic device 100 calculates the altitude using the barometric pressure value measured by the barometric pressure sensor 180C to assist in positioning and navigation.

[00107] The magnetic sensor 180D includes a Hall effect sensor. The electronic device 100 can detect the open or closed state of the leather flip case using the magnetic sensor 180D. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 may detect an open state or a closed state of the flip cover based on the magnetic sensor 180D. In addition, based on the detected open or close state of the leather case or the detected open or close state of the flip cover, a function such as automatic release of the flip cover is set.

[00108] The acceleration sensor 180E can detect the acceleration value of the electronic device 100 in all directions (typically three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. The acceleration sensor can be further configured to sense the position of an electronic device and be used to switch between landscape and portrait orientation, a pedometer, or another application.

[00109] The distance sensor 180F is configured to measure distance. The electronic device 100 can measure distance using infrared or laser light. In some embodiments, in a photography scenario, the electronic device 100 may measure distance using the distance sensor 180F to achieve fast focusing.

[00110] The optical proximity sensor 180G may include, for example, a light emitting diode (LED) and an optical detector such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 may emit infrared light using a light emitting diode. The electronic device 100 detects reflected infrared light from a nearby object using a photodiode. If enough reflected light is detected, the electronic device 100 can determine that an object is near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object in the vicinity of the electronic device 100. The electronic device 100 can detect, with the optical proximity sensor 180G, that the user holds the electronic device 100 close to the ear during a call, so that the screen is automatically turned off to save power. Alternatively, the 180G optical proximity sensor can be used in leather case mode or pocket mode to automatically unlock or lock the screen.

[00111] The ambient light sensor 180L is configured to sense the brightness of the ambient light. The electronic device 100 may adaptively adjust the brightness of the display screen 194 according to the sensed brightness of the ambient light. The ambient light sensor 180L can be optionally configured to automatically adjust the white balance during still photography. The ambient light sensor 180L may further cooperate with the optical proximity sensor 180G to detect whether the electronic device 100 is in the pocket to prevent accidental touching.

[00112] The fingerprint sensor 180H is configured to capture fingerprints. The electronic device 100 can realize fingerprint unlock, application lock access, fingerprint photography, fingerprint call answering, and the like. using the captured fingerprint feature.

[00113] Temperature sensor 180J is configured to detect temperature. In some embodiments, the electronic device 100 executes the temperature processing policy using the temperature detected by the temperature sensor 180J. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of the processor in the vicinity of the temperature sensor 180J to reduce power consumption and implement overheating protection. In some other embodiments, when the temperature is below another threshold, the electronic device 100 heats the battery 142 to prevent the electronic device 100 from crashing due to low temperature. In some other embodiments, when the temperature is below another threshold, the electronic device 100 increases the output voltage of the battery 142 to prevent abnormal shutdown caused by the low temperature.

[00114] The touch sensor 180K is also referred to as a "touch device". The touch sensor 180K may be located on the display screen 194 . The touch sensor 180K and the display screen 194 form a touch screen. The touch sensor 180K is configured to detect a touch operation performed on or near the touch sensor. The touch sensor may communicate the detected touch operation to the application processor to determine the type of touch event. The touch sensor 180K may provide visual output related to the touch operation using the display screen 194 . In some other embodiments, the touch sensor 180K may alternatively be located on the surface of the electronic device 100 and located in a position different from that of the display screen 194.

[00115] The bone conduction sensor 180M can receive a vibration signal. In some embodiments, the bone conduction sensor 180M may receive a vibratory signal from a human vocal cord bone. Alternatively, the bone conduction sensor 180M can be in contact with a human pulse and receive a blood pressure pulse signal. In some embodiments, the bone conduction sensor 180M may alternatively be located in a headset to form a bone conduction headset. The audio module 170 can acquire the voice signal by parsing based on the vibration signal that comes from the vibratory bone of the vocal cord portion and is received by the bone conduction sensor 180M to realize the voice function. The application processor can analyze the heart rate information based on the blood pressure pulse signal received by the bone conduction sensor 180M to realize the heart rate detection function.

[00116] The key 190 includes a power key, a volume key, and the like. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate a key input associated with user configuration and control of the functions of the electronic device 100.

[00117] Motor 191 may generate a vibration prompt. Motor 191 may be configured to provide a vibration prompt for an incoming call and may be further configured to provide touch vibration feedback. For example, touch operations performed in different applications (such as taking photos and playing audio) may correspond to different vibration feedback effects. For touch operations performed in different areas of the display screen 194, the motor 191 may also correspond to different vibration feedback effects. Different application scenarios (such as time prompting, getting information, alarm clock, game) can also correspond to different vibration feedback effects. The touch vibration feedback effect can be further customized.

[00118] The indicator 192 may be an indicator light, may be configured to indicate charging status and battery replacement, and may also be configured to indicate a message, missed call, notification, or the like.

[00119] The SIM card interface 195 is configured to connect to a SIM card. The SIM card may be inserted into the SIM card interface 195 or detached from the SIM card interface 195 to contact or be detached from the electronic device 100. The electronic device 100 may support one or N SIM card interfaces, where N is a positive integer. , large 1. The SIM card interface 195 can support a nano SIM card, a micro SIM card, a SIM card, and the like. A plurality of cards can be inserted into the same SIM card interface 195 . The plurality of cards may be of the same type or of different types. The SIM card interface 195 may be further compatible with various types of SIM cards. The SIM card interface 195 may also be compatible with an external memory card. The electronic device 100 communicates with the network using a SIM card to perform functions such as calling and data transfer. In some embodiments, the electronic device 100 uses an eSIM, that is, an embedded SIM card. The eSIM card may be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[00120] The software system of the electronic device 100 may use a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. In this embodiment of the present invention, the Android system with layered architecture is used as an example to illustrate the software structure of the electronic device 100.

[00121] FIG. 2 is a block diagram of the software structure of an electronic device 100 according to an embodiment of the present invention.

[00122] In a layered architecture, the software is divided into multiple layers, and each layer has a clear role and task. The levels interact with each other through a software interface. In some embodiments, the Android system is divided into four layers, which are respectively the application layer, the application infrastructure layer, the Android runtime and the system library, and the kernel layer from top to bottom.

[00123] The application layer may include a number of application packages.

[00124] As shown in FIG. 2, application packages may include applications such as camera, gallery, calendar, phone, map, navigation, WLAN, Bluetooth, music, video, and short messages.

[00125] In some embodiments of this application, the application packages further include game applications such as parkour games, music-type games, and dance-type games. It should be noted that the game application described here may not support motion sensing gameplay.

[00126] Application packages may further include a first application, such as a game assistant or an application assistant. The user can select, using the interface provided by the first application, a game application such as Subway Parkour to be played with motion sensing gameplay to activate a motion sensing gaming function or the like.

[00127] The application infrastructure layer provides an application programming interface (API) and a programming infrastructure for applications at the application layer. The application infrastructure layer includes some predefined functions.

[00128] As shown in FIG. 2, the application infrastructure layer may include a window manager, a content provider, a viewer, a phone manager, a resource manager, a notification manager, and the like.

[00129] A window manager is configured to manage a windowed application. The window manager can get the size of the display screen, determine if there is a status bar, lock the screen, capture the screen, and so on.

[00130] The content provider is configured to store and retrieve data so that the application can get data to it. The data may include video, image, audio, calls made and answered, browsing history and bookmarks, address book, and the like.

[00131] The viewing system includes a visual control, such as a control for displaying text or a control for displaying an image. The viewer system may be configured to build the application. The display interface can be formed by one or more views. For example, the display interface including the short message notification icon may include a view for displaying text and a view for displaying an image.

[00132] The phone manager is configured to provide a communication function of the electronic device 100, such as call state management (including ringing, clearing, etc.).

[00133] The resource manager provides various resources to the application, such as a localized character string, an icon, an image, a layout file, and a video file.

[00134] The notification manager allows an application to display notification information in the status bar. The notification information may be a message used to indicate the type of notification, and may automatically disappear after a short stay without user interaction. For example, the notification manager is configured to notify download completion, message prompt, and the like. Alternatively, the notification manager may be a notification that appears on the top system status bar as a graphic or scrollbar text, such as a notification about an application running in the background, or may be a notification that appears on the screen as a dialog box. For example, the status bar prompts text information, prompts an audible signal, vibrates an electronic device, or flashes an indicator light.

[00135] In some embodiments of this application, the application infrastructure layer further includes an augmented reality engine, a pose recognition layer, and a game matching layer. The augmented reality engine layer is configured to analyze the user's image captured by the camera, including performing bone point recognition on the user's image, converting the user's image into bone motion coordinates united by points and lines, and the like.

[00136] The posture recognition level is configured to recognize converted bone motion coordinates and determine motion readings such as body tilt left/right/forward/backward tilt, jumping, crouching, left hand up/down/left/forward/backward, left hand raise arms up/down/left//forward/backward, left leg lift forward/left, and right leg lift forward/right away from the user.

[00137] The game matching layer is configured to perform matching of the predefined motion sensing and the touch operation of the game application. When the user plays the game and after detecting the user's motion sensing, if the motion sensing is a predetermined motion sensing, a touch operation in the game application corresponding to the predetermined motion sensing is determined, and an instruction of the corresponding touch operation is sent to the gaming application, so that the gaming application can execute a response corresponding to touch operations, that is, to achieve a game application control effect by using motion sensing.

[00138] The Android runtime includes a kernel library and a virtual machine. The Android runtime is responsible for scheduling and administering the Android system.

[00139] The core library includes two parts: one part is a performance function to be called by the Java language, and the other part is the Android core library.

[00140] The application layer and the application infrastructure layer run on a virtual machine. The virtual machine executes the application layer and application infrastructure layer Java files as binary files. The virtual machine is used to perform functions such as object lifecycle administration, stack administration, thread administration, security and exception administration, and garbage collection.

[00141] The system library may include a plurality of functional modules, such as a surface manager (surface manager), a multimedia library (Media Libraries), a 3D graphics processing library (for example, OpenGL ES) and a 2D graphics engine (for example, SGL).

[00142] The surface manager is configured to administer the display subsystem and combine the 2D and 3D layers of multiple applications.

[00143] The media library supports playback and recording in a variety of common audio and video formats, and also supports still image files and the like. The media library can support many audio and video encoding formats such as MPEG4, H.264, MP3, AAC, AMR, JPG and PNG.

[00144] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, layer processing, and the like.

[00145] The 2D graphics engine is a drawing engine for 2D drawings.

[00146] The kernel level is the level between hardware and software. The kernel level includes at least a display driver, a camera driver, an audio driver, and a sensor driver.

[00147] In some embodiments of this application, the first application may, using a camera driver, call a camera in the hardware to capture an image of the user during the game, so that the augmented reality engine and the pose recognition layer of the application infrastructure layer analyze the captured image of the user and determine the motion reading, performed by the user, etc. When the augmented reality engine analyzes the captured image of the user, and the pose recognition layer performs the pose recognition, the AI processing chip in the hardware can also be called. For example, the NPU performs data processing such as bone point recognition or pose recognition.

[00148] All the technical solutions included in the following embodiments can be implemented in the electronic device 100 having the above hardware architecture and software architecture. The following describes in detail the technical solution provided in the embodiments of this application, in which the electronic device 100 is a mobile phone as an example.

[00149] At present, a user can conveniently install various game applications such as parkour games (eg, Subway Parkour) and music-type games (eg, Dance Mission) on a mobile phone. In order to further improve the level of game enjoyment, an embodiment of this application provides a method for allowing a general game (which refers to a game that does not natively support motion-sensing gameplay) on a mobile phone to support motion-sensing gameplay, so that a user can control game using motion sensing, to achieve an integrated entertainment and fitness effect, thus meeting the diverse needs of the user.

[00150] In some embodiments of this application, during the launch of the target game, a first application, such as a game assistant, is launched, and the first application calls the mobile phone's camera using a kernel-level camera driver to capture an image of the user. The mobile phone then analyzes the captured image of the user and recognizes the motion reading performed by the user. If the motion sensing performed by the user is a predetermined motion sensing, a target game touch operation instruction corresponding to the predetermined motion sensing is sent to the target game so that the target game can execute an appropriate response to allow the target game not natively supporting motion sensing gameplay to also respond to a motion sensing performed by the user. Thus, the native code of the target game may not be modified, that is, the target game may support motion sensing gameplay.

[000151] In other words, a function in which the target game supports the motion sensing game can be implemented using another application (eg, the first application) or a mobile phone system function without changing the target game. It can be seen that the game launch method provided in this embodiment is applicable to games supporting the touch operation, that is, the motion sensing game process support version does not need to be developed separately for each first application, which is beneficial for reducing development costs.

[00152] It can be understood that, in addition to the game application, the method provided in this embodiment of this application is also applicable to another application installed on a mobile phone that can support a touch operation for control. The following describes in detail the technical method provided in the embodiments of this application, using the game application as an example.

[00153] Referring to FIG. 3A - fig. 3C, FIG. 4A - fig. 4C and FIG. 5A - fig. 5I, descriptions are made below with reference to schematic illustrations of graphical user interfaces.

[00154] First, the mobile phone activates the motion sensing gameplay function of the target game.

[00155] In some embodiments, the user may activate the motion sensing gameplay function of the target game with the first application (eg, game assistant or application assistant).

[00156] For example, the interface 300 shown in FIG. 3A is a schematic representation of a desktop of a mobile phone. The interface 300 may include a status bar, one or more application icons such as a game assistant icon 301 and a target game icon 302 (eg, Subway Parkour), and a dock (toolbar). The status bar may include the operator's name, time, battery level status, network connection status, and so on. The toolbar may include many application shortcut icons.

[00157] When the user wants to play a game using the motion sensing gameplay, the user can enter the game assistant interface by tapping the game assistant icon 301 and activate the motion sensing gameplay function of the target game. For example, as shown in FIG. 3B, the target game, such as Subway Parkour, in which the motion sensing gameplay is to be activated is selected in the game assistant interface 303 . Then, the motion sensing game function control 305 is tapped to activate the Subway Parkour motion sensing game function. In this case, the interface 307 shown in FIG. 3C, Game Boost, Do Not Disturb, and Motion Sensing Gameplay are enabled in Subway Parkour. For the game acceleration and do not disturb functions, reference may be made to the prior art, and details will not be repeated here. Next, the motion sensing game process function and the specific implementation of the motion sensing game process function will be described in detail.

[00158] If there is no target game in the game assistant, the target game can alternatively be added to the game assistant using the add control 306 in the game assistant interface 303, after which the target game motion sensing gameplay is activated.

[00159] In some other embodiments, a user may enable a motion sensing gameplay feature through a system setting. The motion sensing gameplay function may be an independent function in a system setting, or may be a sub-function of a game assistant or the like. This is not limited in this embodiment of the present application.

[00160] For example, the setup interface 400 shown in FIG. 4A is an example of a mobile phone system setting interface. The setting interface 400 includes a motion sensing gameplay function 401 . In response, the user taps the motion sensing gameplay function 401, the mobile phone displays the interface 402 shown in FIG. 4b. As shown in FIG. 4C, in response to the user tapping the add control 403, a selection window 404 pops up on the mobile phone, and the user can add, using the selection window 404, a game that requires activation of the motion sensing gameplay function.

[00161] In still some embodiments, during the execution of the target game, the user may alternatively activate the motion sensing gameplay function of the target game by entering a voice command or performing a predetermined gesture such as sliding three fingers down, or in another manner. In other words, the manner in which the mobile phone activates the motion sensing game process is not specifically limited in this application.

[00162] After activating the motion sensing gameplay of the target game, it is necessary to further predetermine some motion sensing, and adjust the correspondence between the predefined motion sensing and touch operations in the target game.

[00163] Touch operations refer to operations such as tapping, double tapping, tapping and holding, sliding, and drawing a specific pattern that the user is allowed to perform in the target game through the touch screen.

[00164] In particular, the user may open the target game using the target game icon on the desktop of the mobile phone, or may open the target game using some other shortcuts.

[00165] For example, in response to the user tapping the target game icon 501 (eg, Subway Parkour) on the desktop of the mobile phone shown in FIG. 5A, the mobile phone starts the target game and displays the target game home interface shown in FIG. 5b. In response to the user tapping the "tap to start game" control 502 on the target game source interface, the target game is launched on the mobile phone, that is, the interface shown in FIG. 5C.

[00166] In some embodiments, as shown in FIG. 5D, after the mobile phone starts the game, the game assistant dialog 503 may pop up automatically or may pop up in response to an operation (e.g., a predetermined gesture is performed or a voice instruction is entered to operate a specific control) of the user on the mobile phone. The user can set the corresponding functions of the game assistant using the dialog box 503. For example, game acceleration, do not disturb mode, secure key touch, and motion-sensing gameplay are set. In some other embodiments, alternatively, after the mobile phone enters the target game but does not start the game, the game assistant dialog 503 may automatically pop up or may pop up in response to a user operation on the mobile phone.

[00167] The dialog box 503 may be displayed in a game interface in a partially transparent or opaque manner. That is, the dialog 503 may block part of the game interface. Alternatively, the dialog 503 may be displayed in a manner that does not block the game interface. For example, a dialog box is displayed on the left, right, top, or bottom of the game interface. The dialog box 503 and the game interface are displayed on the screen without blocking each other. It should be noted that the display time, display method, display position, etc. dialog box 503 is not specifically limited in this embodiment of this application.

[00168] In a specific implementation, after detecting that the user is starting the target game application (or detecting that the mobile phone is displaying a specific interface of the target game application), the mobile phone may add a game assistant layer to the game interface of the target game application using the dispatcher. application infrastructure layer windows or other system service. The game assistant layer is located at the game interface layer of the target game application and includes a game assistant dialog 503, other areas of the layer being transparent in addition to the dialog 503. Thus, the user can see the game interface of the target game application and the dialog box 503 at the same time, and the user can control the control in the dialog box 503. It can be seen that in this implementation, the mobile phone adds a setting interface for predefined motion sensing to the target game application without modification. the original target game application.

[00169] In another specific implementation, upon detecting that the user is running the target game application, or detecting that the mobile phone is displaying a particular interface of the target game application), the mobile phone may alternatively adjust the size and position of the game interface of the target game application with an application infrastructure level window manager or other system service. For example, the game interface of the target game application is reduced to leave a specific size display area on the left side of the mobile phone. In addition, a setting interface is displayed in the display area to prompt the user to set the predefined motion reading. It can be seen that in this implementation, the mobile phone adds a setting interface for predefined motion sensing to the target game application without modifying the original target game application.

[00170] Then, in response to the operation of tapping by the user on the setting control 504 corresponding to the function of the motion sensing gameplay shown in FIG. 5D, the mobile phone displays the interface 505 shown in FIG. 5E, and the specific setting item 506 for the motion sensing gameplay function is displayed on the interface 505. For example, the user can select a predefined motion sensing using the following menu and set the corresponding touch operation for the selected predefined motion sensing.

[00171] In addition, when selecting a predefined motion sensing, the user may alternatively select an image including the predefined motion sensing. Therefore, the mobile phone recognizes the image selected by the user and sets the recognized motion reading as a predefined motion reading, that is, sets some predefined motion readings. The image including the predetermined motion reading may be a picture in an album of the mobile phone, or may be a picture downloaded by the mobile phone from the Internet, or may be a picture including a portrait captured by a camera currently being called by the mobile phone. A method for a mobile phone to recognize a predetermined motion sensing in an image is described in detail below, and the details are not repeated here.

[00172] Table 1 is an example of a correspondence between a set of predefined motion reads and touch operations in the target game.

Table 1

Predefined motion readings Touch operations in the target game Body tilt to the left Swipe left on the screen Body tilt to the right Swipe right on the screen squat Sliding down the screen jumping up Sliding up the screen Continuously jump twice Double tap on the screen ... ...

[00173] Note that, in this application, when the mobile phone detects that the user's motion sensing is a predefined motion sensing, the mobile phone must send a touch operation instruction corresponding to the predefined motion sensing to the target game, so that the target game executes the response corresponding to the operation. touch. In addition to including types of touch operations (e.g., tapping, double tapping, sliding and tapping and holding), the touch operation instruction should further include the specific position (or coordinates) at which the touch operation is performed on the screen, that is, where touch operation is performed on the screen.

[00174] In some examples, the mobile phone may configure one or more default screen positions for touch operations corresponding to predefined motion readings. That is, the user may not specifically adjust the screen position for the touch operation corresponding to the predetermined motion sensing. For example, in the game Subway Parkour, the user can slide left/right/up/down anywhere on the screen, and the game character walks left/walks right/jumps up/crouches down. Thus, if the motion reading and the touch operation, such as tilting the body to the left corresponding to sliding left, are set, subsequently, after checking the reading of the motion reading of the user's body tilting to the left, the mobile phone can send the slide left instruction, that is, send the default coordinates (for example, coordinates the center of the screen or the coordinates of any position at the bottom of the screen) and the action of sliding left into the game Subway Parkour. After receiving the instruction, the Subway Parkour game considers that the slide left operation is received at the default coordinate position, and executes the response corresponding to the operation, that is, the game character moves to the left.

[00175] In some other examples, given that a touch operation is to be performed in a specific area of the screen, the target game may respond when the mobile phone sets up a correspondence between predefined motion readings and touch operations to the user that it can set the positions at which touch operations are performed on the screen.

[00176] The description has been made using an example in which the target game is Honor of Kings. In the Honor of Kings game, when the user sets the attack key in the lower right corner, and when the user sets the predefined motion reading corresponding to the attack key, in addition to setting the touch operation corresponding to the predefined tapping motion reading, the user further needs to set the position where a touch operation is performed so that it is the same as the position of the original attack key in the target game. That is, the target position is set in the area where the original attack key is located.

[00177] For example, as shown in FIG. 5F, the motion sensing gameplay settings may further include a position control 507. The user can adjust, using the position control 507, the position at which each touch operation is performed, which is configured to match the predefined motion sensing. In a specific implementation, in response to the user tapping the position control 507 corresponding to the predefined motion sensing, the mobile phone prompts the user to select the position (or coordinates) on the screen at which the currently configured tap operation is being performed. The user can tap the target position (that is, the position at which the touch operation is to be performed) on the screen, and the mobile phone records the coordinates of the tapped position and sets the coordinates as the coordinates at which the touch operation is performed. Of course, the user can make adjustments by inputting coordinates or selecting coordinates. It is not specifically limited in this embodiment of this application.

[00178] Table 2 is another example of the correspondence between a set of predefined motion reads and touch operations in a target game (eg, Subway Parkour).

table 2

Predefined motion readings Touch operations in the target game Position at which the corresponding touch operation is performed (unit: pixel) Body tilt to the left Swipe left on the screen (200, 100) Body tilt to the right Swipe right on the screen (200, 300) squat Sliding down the screen (100, 200) jumping up Sliding up the screen (300, 200) Continuously jump twice Double tap on the screen (200, 400) ... ...

[00179] In another example, as shown in FIG. 5G, when setting the predefined motion sensing and touch operation, the mobile phone may also display some keys, such as key 1 and key 2, in the game interface. During setup, keys can be associated with predefined motion readings. For example, body tilt left (predefined motion sensing) is associated with slide left operation (touch operation) and key 1. This means that body tilt left motion reading is equivalent to the user performing slide left touch operation at key position 1. That is, upon detecting the left body tilt motion reading, the mobile phone sends the key position 1 and the slide left instruction to the target game.

[00180] Of course, the user can change the positions of the keys by moving the keys. Alternatively, the user can change the number of keys by removing or adding a control. It can be understood that the keys are displayed when the predetermined motion senses are set, so that the user clearly remembers the positions at which touch operations corresponding to the predetermined motion senses are performed. After completing the setup, the keys are not displayed in the game interface.

[00181] Table 3 is another example of the correspondence between a set of predefined motion reads and touch operations in a target game (eg, Subway Parkour).

Table 3

key 1 Predefined motion readings Touch operations in the target game key 2 Body tilt to the left Swipe left on the screen key 3 Body tilt to the right Swipe right on the screen key 4 squat Sliding down the screen key 5 jumping up Sliding up the screen key 6 Continuously jump twice Double tap on the screen ... ...

[00182] Key position 1 on the mobile phone screen is (200, 100), key position 2 is (200, 300), key position 3 is (100, 200), key position 4 is (300, 200), and key position 5 is (200, 400). The unit of measure is pixel.

[00183] It can be understood that, in general, various gaming applications also have some common touch operations. Therefore, the mobile phone can customize the general touch operations according to the default predefined motion sensing. That is, the same touch operation in different gaming applications corresponds to the same predefined motion reading.

[00184] Alternatively, given that game applications of the same type have more similar touch operations, therefore, the correspondence between predefined motion senses and touch operations can be configured by default for game applications of the same type. That is, the same touch operation in the same type of game applications corresponds to the same predefined motion reading, which helps to reduce the burden on the user to remember, so that the user maintains the same operation habit, thereby increasing the versatility of the predefined motion reading. In addition, the cost of setting up a predetermined motion sensing by the user can also be reduced.

[00185] After setting the predefined motion sensing, the specific motion sensing gameplay function setting item can be closed using the close control 508 in FIG. 5G. As shown in FIG. 5H, the game interface is still displayed.

[00186] The above description has been made using an example in which, when the motion sensing game process function is first activated, it is necessary to set up the motion sensing game process. Subsequently, when the mobile phone activates the motion sensing game again, setting the motion sensing game may no longer be required, and the user can directly play the game using the motion sensing. Of course, alternatively, the mobile phone may support changing the motion sensing game setting by the user.

[00187] The following describes a process in which a user can play a game using a motion sensing game process (that is, control a target game using motion sensing).

[00188] In some embodiments, to provide a better visual experience, the user may also be prompted to activate the screen mirroring function, and the game interface is projected onto another electronic device with a large screen (for example, a TV) that is connected to a mobile phone. Alternatively, upon detecting that the user activates the motion sensing gaming function, the mobile phone automatically activates the screen mirroring function to project the game interface onto another screen.

[00189] Upon activation of the motion-sensing gameplay function, the mobile phone (eg, the first application) may create a camera preview window. For example, a mobile phone may add a first application layer to the game interface of the target game application. The layer may include a preview window created by the first application, and other areas of the layer are transparent. Thus, the user can simultaneously observe the game interface of the target game application and the newly created camera preview window. In addition, the user can check, using the preview window, the predetermined motion reading performed by the user to determine whether the predetermined motion reading by the user is correct. The camera called by the mobile phone may be one or more front cameras, or it may be one or more rear cameras. When the mobile phone is equipped with a depth camera, the depth camera can also be called to obtain depth data of the user's image to further determine postures such as forward/backward body tilt, forward/backward arm raise, and forward/backward leg raise of the user. The depth camera includes any one or more of a structured light camera, a binocular camera, a time of flight (TOF) camera, and the like.

[00190] In a specific example, a game interface displayed on a mobile phone is projected onto a large screen electronic device such as a TV, which is connected to the mobile phone, the mobile phone calls the rear camera to capture the user's image, then determines the user's motion reading based on the captured user image and determines whether the motion sensing is a predefined motion sensing.

[00191] FIG. 5I is the game interface 509 of the target game. The game interface 509 includes a camera preview window 510 . The user can adjust, using the camera preview window 510, the position at which the mobile phone is located and determine the position at which the user performs the predetermined motion reading. In some examples, the mobile phone may start calling the camera after starting the target game, or may start calling the camera after starting the game. The mobile phone can display the camera preview window all the time, so that the user checks, using the camera preview window, the motion reading performed by the user to determine whether the motion reading performed is correct. Of course, after the user becomes familiar with the motion sensing gameplay, the preview window may alternatively be manually closed or the preview window is automatically closed after the mobile phone displays the preview window for a period of time. Thus, the user can view the game interface in full screen mode. This is not limited in this embodiment of the present application.

[00192] Subsequently, when the user plays the target game, the mobile phone camera captures a real-time image of the user, analyzes the captured image of the user, recognizes the user's motion reading, and determines whether the motion reading is a predetermined motion reading. If the motion sensing is a predetermined motion sensing, a touch operation corresponding to the predetermined motion sensing is searched, a corresponding touch operation instruction is sent to the target game, and the target game executes the corresponding response. Therefore, an effect is achieved when the user controls the target game using the predetermined motion sensing.

[00193] The following illustrates a specific implementation of a user's motion sensing recognition.

[00194] First, the mobile phone performs bone point recognition on the captured user image (which is typically an RGB image or may include a depth image captured by a depth camera) and converts the portrait on the user image into bone movement coordinates combined using the bone points (and/or connection lines between bone points) to determine the motion reading corresponding to the user's image. It can be seen that the coordinates of the movement of the bones include the coordinates of the points of the bones in the portrait.

[00195] For example, FIG. 6A is an image 600 captured by a mobile phone. The mobile phone can detect a portrait such as portrait 601 in an image using portrait recognition technology, for example. Subsequently, the positions of joints such as elbow joints, wrist joints, and knee joints in the 601 portrait are determined using bone recognition technology (the process of estimating the position of the joint points of the bones of the human body in an image or video using computer vision technology), and the positioned points of the joints are finally , may form an image of the bones of the human body, which may reflect information such as the current posture (ie, motion reading) of the human body, or the like.

[00196] When the pose (i.e., motion reading) of a portrait is analyzed based on the movement coordinates of the bones, the portrait pose may be determined based on the coordinate positions of the points of the bones, or may be determined based on the relative correspondence of the positions between the points of the bones, or may be determined based on the connection line between any two points of the bones. This is not limited in this embodiment of the present application.

[00197] For example, in FIG. 6A shows some recognized bone points in black triangles, and the bone points may include: frontal bone 602, cervical vertebrae, shoulder blade, elbow joint, wrist joint, pelvic bone, knee joint, ankle joint, and the like. It can be understood that when more bone points are recognized, the recognized posture of the user is more accurate. Of course, the mobile phone can determine, based on the complexity of the predetermined motion reading, which bone points need to be recognized in order to reduce the recognition of unnecessary bone points, thereby helping to reduce the amount of computation and improve the processing capability of the mobile phone. This is not limited in this embodiment of the present application.

[00198] In some examples, the user's posture may be determined directly based on the coordinates of bone points, or the relative position relationship between bone points, or connection lines between bone points. In some other examples, the reference portrait may alternatively be predetermined, ie the reference coordinates or bone point positions are specified in the reference portrait. Subsequently, when the user's pose is determined, the user's pose can be determined based on the relative position relationship between the bone points in the portrait to be determined and the bone points in the reference portrait. The reference coordinates of the bone points in the reference portrait may be the average values of the bone points determined from different portraits in the massive pictures. Alternatively, the reference coordinates of the bone points in the reference portrait may be the coordinate values of the bone points determined based on the image of a particular person (eg, a mobile phone user). It is not specifically limited in this embodiment of this application.

[00199] For example, from the coordinates of the bone points in the portrait 601, it can be seen that the connection line between the frontal bone 602 and the cervical vertebrae is parallel (or approximately parallel) to the vertical direction, and the coordinates of other bone points are also in a specific positional relationship. Therefore, it can be determined that the user in the portrait 601 is standing upright and the arms are hanging naturally.

[00200] In another example, portrait 603 shown in FIG. 6B, from the coordinates of the points of the bones in the portrait 603, it can be seen that the connection line between the frontal bone 602 and the cervical vertebra forms a specific angle with the vertical direction (for example, turning counterclockwise by α degrees, with α greater than 10 degrees and less than 90 degrees ). Therefore, it can be determined that the user in the portrait 603 is in a left-leaning posture. In some other examples, the coordinates of the points of the bones in the portrait 601 may alternatively be set as a reference, and the portrait whose position is to be determined is compared with the reference. For example, the connection line between the frontal bone 602 and the cervical vertebra in the portrait 603 makes a specific angle (eg, a counterclockwise rotation of β degrees, with β greater than 10 degrees and less than 90 degrees) with the connection line between the frontal bone 602 and the cervical vertebra in the portrait. 601. Therefore, it can be determined that the user in the portrait 603 is in a left-leaning posture.

[00201] In another example, the description is still made using the example in which the coordinates of the points of the bones in the portrait 601 are set as reference. In portrait 604 shown in FIG. 6C, the wrist joint 605 of the left hand in portrait 604 is located closer to the upper left corner than the coordinates of the wrist joint of the left hand in portrait 601, and the coordinates of the remaining points of the bones have not changed much. Therefore, it can be determined that the user is in the position of raising the left forearm.

[00202] In another example, the description is still made using the example in which the coordinates of the points of the bones in the portrait 601 are set as reference. In portrait 606 shown in FIG. 6D, the left knee joint 607 and the left foot wrist joint 608 in portrait 606 are closer to the upper left than the left knee joint and left foot wrist joint in portrait 601, and the coordinates of the remaining bone points are almost unchanged. Therefore, it can be determined that the user is in the left leg lifting position.

[00203] It should be noted that the above descriptions are made using an RGB image captured by a mobile phone as an example. It can be appreciated that the mobile phone may alternatively invoke the depth camera to obtain portrait depth data in the image. Thus, more positions can be further determined based on the depth data.

[00204] For example, FIG. 6E is a schematic profile view when the user's upper body is tilted forward. As can be seen from FIG. 6E, if the depth data of the bone points (for example, the frontal bone 602, scapula, elbow joint, and wrist joint) of the upper body of the portrait in the image is smaller than the depth data of the bone points of the lower body, it can be recognized that the upper body is closer to the mobile camera. phone. Therefore, it can be determined that the portrait in the image is tilted forward. Similarly, if the depth data of the bone points of the upper body of the portrait is greater than the depth data of the bone points of the lower body, it can be known that the upper body is further away from the mobile phone camera. Therefore, it can be determined that the portrait in the image is tilted backwards.

[00205] Alternatively, a reference image is predefined. If it is determined that the depth data of the bone points of the upper body of the portrait in the image is smaller than the depth data of the bone points of the upper body of the reference portrait, it can be determined that the portrait in the image is tilted forward. If it is determined that the depth data of the bone points of the upper body of the portrait in the image is larger than the depth data of the bone points of the upper body of the reference portrait, it can be determined that the portrait in the image is tilted back.

[00206] In another example, in FIG. 6F is a schematic profile view when the user's left hand is extended forward. As can be seen from FIG. 6F, if the depth data of bone points (eg, left elbow joint and left wrist joint) of the left hand in the image is smaller than the depth data of other bone points, it can be recognized that the left hand is closer to the mobile phone camera. Therefore, it can be determined that the left arm of the portrait in the image is stretched forward. Similarly, if the depth data of the bone points of the left hand of the portrait is greater than the depth data of other bone points, it can be known that the left hand is farther from the cell phone camera. Therefore, it can be determined that the left arm of the portrait in the image is stretched back.

[00207] It is understood that another method of recognizing a portrait pose (ie, reading movement) in an image or video can alternatively be used. This is not limited in this embodiment of the present application.

[00208] An embodiment of this application further provides the chip system shown in FIG. 7. The chip system includes at least one processor 1101 and at least one interface chip 1102. The processor 1101 may be connected to the interface chip 1102 via a communication link. For example, interface chip 1102 may be configured to receive a signal from another device (eg, the memory of electronic device 100). In another example, interface chip 1102 may be configured to send a signal to another device (eg, processor 1101). For example, interface chip 1102 may read instructions stored in memory and send instructions to processor 1101. Instructions executed by processor 1101 may cause an electronic device to execute steps performed by electronic device 100 (e.g., a mobile phone) in the above embodiments. Of course, the chip system may further include another discrete device. It is not specifically limited in this embodiment of this application.

[00209] It can be understood that in order to implement the above functions, the above terminal includes an appropriate hardware structure and/or software modules to perform the functions. One skilled in the art will appreciate that, in combination with the modules and algorithm steps of the examples described in the embodiments disclosed herein, the embodiments of the present application may be implemented in hardware or a combination of hardware and computer software. Whether the function is performed by hardware or by computer software that powers the hardware depends on particular applications and design constraints. A person skilled in the art may use different ways to implement the described functions for each particular application, but the implementation should not be considered outside the scope of the embodiments of the present invention.

[00210] In this embodiment, the functional modules of the terminal can be obtained by separation in accordance with the above example methods. For example, functional modules may be obtained by separation corresponding to different functions, or two or more functions may be integrated into one processing module. An integrated module may be implemented in hardware, or may be implemented in a software function module. It should be noted that in this embodiment of the present invention, the division of modules is merely an example, and is merely a division of logical functions. During the actual implementation, there may be another way to split.

[00211] The above descriptions of the embodiments enable one skilled in the art to understand that, for purposes of convenience and concise description, a division of the above functional modules is used as an example for illustration. In the actual application, the above functions can be distributed and implemented by various modules according to the requirement, that is, the internal structure of the device is divided into various functional modules to implement all or part of the functions described above. For the particular workflow of the system, device, and unit described above, reference may be made to the corresponding process in the above method embodiments, and the details are not re-described here.

[00212] The functional blocks in embodiments of this application may be integrated into a single processing unit, or each of the units may physically exist on its own, or two or more units may be combined into one unit. The integrated block may be implemented in hardware or may be implemented as a software functional block.

[00213] When an integrated block is implemented in the form of a software functional block and sold or used as a standalone product, the integrated block may be stored on a computer-readable storage medium. Based on such an understanding, the technical solutions of the embodiments of this application in essence or part contributing to the corresponding technology, or all or some of the technical solutions can be implemented as a software product. The computer program product is stored on a storage medium and includes several instructions for directing a computing device (which may be a personal computer, a server, a network device, etc.) or a processor to perform all or some of the steps of the methods described in the embodiments of this application. . The aforementioned storage medium includes: any medium on which program code can be stored, such as flash memory, removable hard disk, read only memory, random access memory, magnetic disk or optical disk.

[00214] One of skill in the art may also appreciate that when an integrated module is implemented as a functional software module and sold or used as a standalone product, the integrated module may be transmitted in one or more signals distributed using a communications network. The signal may be an electrical signal, an optical signal, a sound wave signal, a radio signal, or another form of signal. The end user can download the software to a local device (such as a personal computer, mobile phone, or tablet computer) using a signal.

[00215] The above descriptions are merely specific embodiments of this application, but are not intended to limit the scope of protection of this application. Any modification or replacement within the technical scope disclosed in this application shall fall within the protection scope of this application. Thus, the scope of the present application should be used as the scope of the claims.

Claims (23)

1. A method for launching an application applicable to an electronic device having at least one camera and a touch screen, the method comprising: displaying, by the electronic device, a first picture of a first application, the first application supporting the user to control the first application through a touch operation on the touch screen; capturing by the electronic device a first image of the user using at least one camera and recognizing a first motion reading related to the position of the user's body in the first image based on the motion coordinates of the bones converted from the first image of the user; determining by the electronic device the first touch operation information corresponding to the first motion reading based on the predetermined correspondence between the motion reading and the touch operation, which includes the type of the touch operation and the position at which the touch operation is performed on the touch screen, wherein the first touch operation information contains the type of the first touch operation and the position at which the first touch operation is performed on the touch screen; And providing the electronic device with a response from the first application corresponding to the first touch operation in response to determining the first touch operation information corresponding to the first motion reading without touch operation corresponding to the first touch operation information performed by the user. 2. The method of claim 1, wherein the method further comprises: displaying by the electronic device a second picture of the first application, the second picture being the same as the picture displayed by the electronic device in response to the user performing a first touch operation on the first picture. 3. The method according to claim 1 or 2, characterized in that the determination by the electronic device of the first information of the touch operation corresponding to the first motion reading comprises: calling, by the application infrastructure layer interface electronic device, and determining the first touch operation information corresponding to the first motion reading. 4. The method according to any one of paragraphs. 1-3, characterized in that before the electronic device captures the first image of the user by calling at least one camera, the method further comprises: receiving by the electronic device an operation in which the user activates the first function; And activation by the electronic device of the first function in response to an operation in which the user activates the first function, the first function being a function whose first application supports the user to control the first application by motion sensing. 5. The method according to claim 4, characterized in that after activation of the first function by the electronic device, the method further comprises: adding a layer by the electronic device to the picture of the first application, wherein the layer contains an opaque area and a transparent area, and the opaque area displays the setting interface of the first function. 6. The method of claim 5, further comprising: receiving by the electronic device an operation in which the user sets the correspondence between the motion sensing and the touch operation using the first function setting interface, wherein the touch operation contains the type of the touch operation and the position at which the touch operation is performed on the touch screen. 7. The method according to p. 6, further comprising: storing, by the electronic device, user settings for matching between the motion reading and the touch operation; And adding by the electronic device a second layer to the picture of the first application, wherein the second layer contains an opaque area and a transparent area, and the opaque area displays a preview window of at least one camera. 8. An electronic device comprising: at least one processor, a touch screen, and at least one camera, wherein the touch screen and at least one camera are connected to the processor, and the electronic device is activated to perform the method of launching an application according to any one of paragraphs. 1-7. 9. A computer storage medium containing computer instructions, wherein the computer instructions, when executed on an electronic device, cause the electronic device to execute a method for launching an application according to any one of paragraphs. 1-7. 10. A chip system comprising at least one processor, wherein when the at least one processor executes instructions, the at least one processor executes the method of launching an application according to any one of claims. 1-7.

Images