WO2023030067A1

WO2023030067A1 - Remote control method, remote control device and controlled device

Info

Publication number: WO2023030067A1
Application number: PCT/CN2022/113924
Authority: WO
Inventors: 牛洋; 任哲坡; 汤琼; 丁洪霞
Original assignee: 华为技术有限公司
Priority date: 2021-08-30
Filing date: 2022-08-22
Publication date: 2023-03-09
Also published as: CN115762108A

Abstract

A remote control method, a remote control device (100) and a controlled device (200). The controlled device (200) comprises a screen. The method comprises: receiving first information of a remote control device (102); determining, according to the first information, that the remote control device (100) directly faces a target position of a screen (104); and executing a first operation on the screen according to the target position (106). Therefore, a user conducts a remote operation on a controlled device without needing to perform frequent mechanical operations on five-way navigation buttons, thereby improving the experience of the user during a usage process.

Description

Remote control method, remote control device and controlled device

This application is required to be submitted to the China Patent Office on August 30, 2021, the application number is 202122060808.7, and the application name is "Remote Control Method, Remote Control Device, and Controlled Device", and it is submitted to the China Patent Office on March 14, 2022. The application number is 202210248607.6, the priority of a Chinese patent application titled "Remote Control Method, Remote Control Device, and Controlled Device", the entire contents of which are incorporated in this application by reference.

technical field

The present application relates to the technical field of remote control, and in particular to a remote control method, a remote control device and a controlled device.

Background technique

Currently, there are mainly two methods for remote control of the controlled device, one is remote control operation, and the other is using a mobile phone as the remote control of the controlled device.

Current remote controls, whether infrared remote controls or bluetooth remote controls, include five-way navigation keys. It is easy for users to use the remote control to control the controlled device, but compared with mobile phones and tablets as similar interactive products, the user's operating experience is poor. With the trend of large screens, more and more content can be displayed on the screen. When users use the basic five-guidance buttons, they often need to perform frequent mechanical operations to quickly locate the desired content. However, using a mobile phone as a large-screen remote control requires the mobile phone to have an infrared function, and corresponding software needs to be installed on the mobile phone to support it. In addition, using the mobile phone as the remote control of the controlled device will be limited by the distance and infrared emission angle, which does not fundamentally solve the problem of switching back and forth through the five-way navigation key to complete the interaction, resulting in a poor experience during use.

Therefore, the current remote control method of the controlled device requires the user to mechanically and frequently operate the five-way navigation key to realize the remote operation of the large screen, resulting in a poor experience during use.

Contents of the invention

In view of this, an embodiment of the present invention provides a remote control method, a remote control device, and a controlled device, which enable the user to realize remote operation of the controlled device without performing frequent mechanical operations on the five-way navigation key. Improve user experience during use.

In the first aspect, an embodiment of the present invention provides a remote control method applied to a controlled device, where the controlled device includes a screen, and the method includes:

receiving first information of the remote control device;

determining a target position where the remote control device is facing the screen according to the first information;

A first operation is performed on the screen according to the target position.

In a possible implementation manner, the first information includes position coordinates of the target position where the remote control device is facing the screen;

Determining the target position where the remote control device is facing the screen according to the first information includes:

According to the position coordinates, the target position corresponding to the position coordinates on the screen is determined.

In a possible implementation manner, the position coordinates of the target position are obtained according to an image including an outline of the screen captured by a binocular camera of the remote control device.

In a possible implementation manner, the first information includes an image including an outline of the screen captured by a binocular camera of the remote control device;

The determining the target position where the remote control device is facing the screen according to the first information includes: determining the target position on the screen corresponding to the position coordinates according to the image.

In a possible implementation manner, the determining the target position corresponding to the position coordinates on the screen according to the image includes:

Obtaining the position coordinates of the target position where the remote control device is facing the screen according to the image;

In a possible implementation manner, the first operation includes: displaying a pointer at the target position; or, highlighting or shadowing a control corresponding to the target position.

On the other hand, an embodiment of the present invention provides a remote control method, which is applied to a remote control device, and the method includes:

Acquiring first information, where the first information is used to determine the target position of the remote control device facing the screen of the controlled device;

sending the first information to the controlled device, so that the controlled device determines the target position according to the first information, and executes a first operation on the screen according to the target position.

In a possible implementation manner, the remote control device includes a binocular camera, and the first information includes position coordinates of the target position where the remote control device is facing the screen;

Get first information, including:

When the remote control device points to the screen, the binocular camera captures an image containing the outline of the screen;

The position coordinates of the target position where the remote control device is facing the screen are obtained according to the image.

In a possible implementation manner, the remote control device includes a binocular camera, and the first information includes an image including an outline of the screen captured by the binocular camera;

Get first information, including:

When the remote control device points to the screen, the image is captured by the binocular camera.

On the other hand, an embodiment of the present invention provides a remote control device, which is applied to a controlled device, where the controlled device includes a screen, and the device includes:

A receiving module, configured to receive the first information of the remote control device;

A determining module, configured to determine a target position where the remote control device is facing the screen according to the first information;

An executing module, configured to execute a first operation on the screen according to the target position.

The determining module is specifically configured to determine the target position corresponding to the position coordinates on the screen according to the position coordinates.

The determination module is specifically configured to determine the target position corresponding to the position coordinates on the screen according to the image.

In a possible implementation manner, the determination module is specifically configured to: obtain the position coordinates of the target position where the remote control device is facing the screen according to the image, and determine the position coordinates according to the position coordinates. The target position corresponding to the position coordinates on the screen.

On the other hand, an embodiment of the present invention provides a remote control device, which is applied to a remote control device, and the device includes:

An acquiring module, configured to acquire first information, and the first information is used to determine a target position where the remote control device faces the screen of the controlled device;

a sending module, configured to send the first information to the controlled device, so that the controlled device determines the target position according to the first information, and executes on the screen according to the target position first operation.

The acquisition module is specifically configured to, when the remote control device points to the screen, take an image containing the outline of the screen through the binocular camera, and obtain that the remote control device is facing the screen according to the image The location coordinates of the target location for .

The acquisition module is specifically configured to capture the image through the binocular camera when the remote control device points to the screen.

On the other hand, an embodiment of the present invention provides a controlled device, including a screen, a processor, and a memory, wherein the memory is used to store a computer program, and the computer program includes program instructions. When the processor runs the When the above-mentioned program instructions are used, the controlled equipment is made to perform the following steps:

receiving first information of the remote control device;

A first operation is performed on the screen according to the target position.

In a possible implementation manner, when the processor runs the program instruction, the controlled device is made to perform the following steps:

The first information includes position coordinates of the target position where the remote control device is facing the screen;

The first information includes an image including the outline of the screen captured by the binocular camera of the remote control device;

The determining the target position corresponding to the position coordinates on the screen according to the image includes:

On the other hand, an embodiment of the present invention provides a remote control device, including a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and when the processor runs the program instructions , make the remote control device perform the following steps:

In a possible implementation manner, the remote control device includes a binocular camera, and the first information includes position coordinates of the target position where the remote control device is facing the screen; when the processor runs the When the program instruction is used, the remote control device is made to perform the following steps:

Get first information, including:

In a possible implementation manner, the remote control device includes a binocular camera, and the first information includes an image including the outline of the screen captured by the binocular camera; when the processor runs the program When commanding, make the remote control device perform the following steps:

Get first information, including:

On the other hand, an embodiment of the present invention provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, the computer program includes program instructions, and when the program request is executed by the computer, the A computer executes the method as described above.

In the technical solutions of the remote control method, the remote control device and the controlled device provided by the embodiments of the present invention, the controlled device includes a screen, and the method includes: receiving first information of the remote control device; determining the The remote control device is facing a target position on the screen; and performing a first operation on the screen according to the target position. The embodiment of the present invention enables the user to realize the remote operation of the controlled device without performing frequent mechanical operations on the five-way navigation key, thereby improving the experience of the user during use.

Description of drawings

Fig. 1 is a structure diagram of a remote control system provided by an embodiment of the present invention;

Fig. 2 is a structure diagram of a remote control system provided by another embodiment of the present invention;

FIG. 3 is a schematic diagram of a remote control device pointing to a screen;

Fig. 4 is a schematic diagram of the outline of the screen itself and the outline information of the screen;

Fig. 5 is the schematic diagram of the workflow of algorithm module;

Fig. 6 is a flowchart of a remote control method provided by an embodiment of the present invention;

Fig. 7 is a specific flow chart of executing a first operation corresponding to the first information on the screen according to the first information in Fig. 6;

Fig. 8 is a flowchart of a remote control method provided by another embodiment of the present invention;

Fig. 9 is a specific flow chart of obtaining the first information in Fig. 8, the first information is used to determine the target position of the remote control device facing the screen of the controlled device;

FIG. 10 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention.

Detailed ways

In order to better understand the technical solutions of the present invention, the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings.

It should be clear that the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

Terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. As used in the embodiments of the present invention and the appended claims, the singular forms "a", "said" and "the" are also intended to include the plural forms unless the context clearly indicates otherwise.

It should be understood that the term "and/or" used herein is only an association relationship describing associated objects, which means that there may be three relationships, for example, A and/or B, which may mean that A exists alone, and A and B exist simultaneously. B, there are three situations of B alone. In addition, the character "/" in this article generally indicates that the contextual objects are an "or" relationship.

Because the controlled device has an open operating system and chip, and an open application platform, it can realize two-way human-computer interaction function, and people can freely choose to watch any program they like in their free time. Moreover, with the development of artificial intelligence technology, the controlled device can recommend programs that meet the user's viewing habits and preferences, greatly improving entertainment, making the controlled device gradually become the most important interactive center for living room entertainment. However, the interactive mode of the controlled device is realized by the remote control method.

In order to solve the above technical problems, embodiments of the present invention provide a remote control method, a remote control device and a controlled device.

Referring to FIG. 1 , FIG. 1 is a structural diagram of a remote control system provided by an embodiment of the present invention. The remote control system in the embodiment of the present invention includes two electronic devices, wherein the two electronic devices may include a controlled device and a remote control device, and the communication mode between the controlled device and the remote control device includes wireless transmission. As shown in FIG. 1 , the remote control system includes a remote control device 100 and a controlled device 200 . The remote control device 100 and the controlled device 200 can perform wireless transmission, for example, wireless transmission includes Bluetooth, infrared, mobile network or WLAN transmission. Wherein, the remote control device 100 includes a binocular camera and a sending module. The controlled device 200 includes a receiving module, an algorithm module and a screen.

In Fig. 1 , the binocular camera of the remote control device 100 is used to take an image of the screen of the controlled device 200, the image contains the outline of the screen, and sends the image to the sending module; the sending module is used to send the first information through wireless transmission to the controlled device 200. Wherein, the first information includes an image including the outline of the screen captured by the binocular camera of the remote control device 100 . The receiving module of the controlled device 200 is configured to receive the first information, and send the first information to the algorithm module. The algorithm module determines the position coordinates of the target position where the remote control device 100 is facing the screen according to the first information. Specifically, the algorithm module obtains the position coordinates of the target position where the remote control device 100 is facing the screen according to the image, and sends the position coordinates to the screen. The screen is used to determine a target position corresponding to the position coordinates on the screen according to the position coordinates, and perform a first operation on the screen according to the target position.

Referring to FIG. 2 , FIG. 2 is a structural diagram of a remote control system provided by another embodiment of the present invention. The remote control system in the embodiment of the present invention includes two electronic devices, wherein the two electronic devices may include a controlled device and a remote control device, and the communication mode between the controlled device and the remote control device includes wireless transmission. As shown in FIG. 1 , the remote control system includes a remote control device 100 and a controlled device 200 . The remote control device 100 and the controlled device 200 can perform wireless transmission, for example, wireless transmission includes Bluetooth, infrared, mobile network or WLAN transmission. Wherein, the remote control device 100 includes a binocular camera, an algorithm module and a sending module. The controlled device 200 includes a receiving module and a screen.

In FIG. 2 , the binocular camera of the remote control device 100 is used to capture an image of the screen of the controlled device 200 , the image includes the outline of the screen, and sends the image including the outline of the screen to the algorithm module. The algorithm module obtains the position coordinates of the target position where the remote control device 100 is facing the screen according to the image, and sends the first information to the sending module. Wherein, the first information includes the position coordinates of the target position where the remote control device 100 is facing the screen. The sending module is configured to send the first information to the controlled device 200 through wireless transmission. The receiving module of the controlled device 200 is configured to receive the first information and send the first information to the screen. The screen is used to determine a target position where the remote control device is facing the screen according to the first information; and perform a first operation on the screen according to the target position. Specifically, the screen is used to determine a target position corresponding to the position coordinates on the screen according to the position coordinates, and perform a first operation on the screen according to the target position.

In FIG. 1 and FIG. 2 , the remote control device 100 includes a binocular camera. Specifically, as shown in FIG. 3 , the binocular camera is located at the front end of the remote control device 100 and can recognize the outline of the screen. The screen is captured by a binocular camera to obtain two images containing the outline of the screen, as shown in Figure 4, the image on the left containing the outline of the screen is captured by the left-eye camera, and the image on the right containing the outline of the screen is captured by the right-eye camera of.

For example, binocular cameras include binocular depth-of-field cameras.

It should be noted that, as shown in FIG. 3 , when the remote control device 100 faces the screen, the line between the center point of the remote control device 100 and the target position on the screen is parallel to the two longer sides of the remote control device 100 .

Among them, as shown in Figure 5, the algorithm module is specifically used to: calibrate the binocular camera in the offline state; correct the two images on the screen to remove the distortion caused by the binocular camera; The disparity map is obtained by matching points between the two corrected images; according to the disparity map, the position coordinates of the remote control device 100 facing the screen are obtained through 3D coordinate calculation.

Among them, the purpose of calibrating the binocular camera in the offline state is to align the two cameras by obtaining the internal reference and external reference of the two cameras. First, calibrate the left-eye camera to obtain the internal and external parameters of the left-eye camera; secondly, calibrate the right-eye camera to obtain the internal and external parameters of the right-eye camera; finally, calibrate the binocular camera to obtain the translation and rotation between the left-eye camera and the right-eye camera relation. Among them, the internal parameters include focal length, image center, distortion coefficient, etc.; the external parameters include rotation matrix and translation matrix.

Wherein, by correcting the two images on the screen, the difference in the X direction of the two images can be obtained, which can improve the accuracy of the parallax calculation. Specifically, correcting the two images on the screen includes distortion correction and conversion into a standard form.

Among them, binocular matching is the core part of binocular depth estimation, and its main purpose is to calculate the relative matching relationship of pixels between two images. Specifically, binocular matching includes five steps: matching error calculation, error integration, disparity map calculation, disparity map optimization, and disparity map correction.

Wherein, the three-dimensional information of the screen in the picture is reconstructed according to the matching information in the disparity map and the principle of triangulation, and the position coordinates of the remote control device 100 facing the screen are obtained by calculating the 3D coordinates.

Wherein, the first operation includes: displaying the pointer at the target position; or highlighting or shadowing the control corresponding to the target position.

It should be noted that when the user moves the remote control device 100, the image including the outline of the screen collected by the remote control device 100 will change, and it is necessary to recalculate the position coordinates of the remote control device 100 facing the screen, and the controlled device 200 The position of the pointer is refreshed, and the pointer moves according to the pointing of the remote control device; or, the controlled device 200 refreshes the highlighted or shaded controls according to the new position coordinates.

Based on the architecture diagrams provided in FIG. 1 and FIG. 2 above, an embodiment of the present invention provides a remote control method, which is applied to a controlled device 200, and the control device 200 includes a screen. Fig. 6 is a flowchart of a remote control method provided by an embodiment of the present invention. As shown in Figure 6, the method includes:

Step 102, receiving first information of the remote control device.

Wherein, the first information includes the position coordinates of the target position where the remote control device is facing the screen when the remote control device points to the screen; or, the first information includes an image including the outline of the screen captured by the binocular camera of the remote control device.

In the embodiment of the present invention, as shown in FIG. 1, the first information includes an image including the outline of the screen captured by the binocular camera of the remote control device, and the receiving module of the controlled device 200 receives the first information of the remote control device 100, and then sends the The first information is sent to the algorithm module.

In the embodiment of the present invention, as shown in FIG. 2 , the first information includes the position coordinates of the target position where the remote control device 100 is facing the screen, and the receiving module of the controlled device 200 receives the first information sent by the remote control device 100 , and then sends the first The information is sent to the screen.

Wherein, the remote control device includes a binocular camera, and the position coordinates of the target position are obtained from an image including the outline of the screen captured by the binocular camera of the remote control device. Specifically, the binocular camera is located at the front end of the remote control device and can recognize the outline of the screen, and two images containing the outline of the screen are obtained by shooting the screen through the binocular camera.

For example, binocular cameras include binocular depth-of-field cameras.

Step 104: Determine the target position where the remote control device is facing the screen according to the first information.

As an optional solution, the first information includes position coordinates of the target position where the remote control device is facing the screen; step 104 includes: determining the target position on the screen corresponding to the position coordinates according to the position coordinates.

As shown in FIG. 2 , the receiving module of the controlled device 200 sends the first information to the screen, and the screen determines the target position corresponding to the position coordinates according to the position coordinates in the first information.

As another optional solution, the first information includes an image including the outline of the screen captured by the binocular camera of the remote control device; step 104 includes: determining the target position corresponding to the position coordinates on the screen according to the image.

Specifically, as shown in Figure 7, step 104 includes:

Step 1042, obtain the position coordinates of the target position where the remote control device is facing the screen according to the image.

As shown in FIG. 1 , the algorithm module of the controlled device 200 obtains the position coordinates of the target position where the remote control device 100 is facing the screen according to the image, and then sends the position coordinates to the screen.

Specifically, the algorithm module calibrates the binocular camera in an offline state; corrects the two images on the screen to remove the distortion caused by the binocular camera; calculates the difference between the corrected two images through binocular matching Match the points to obtain a disparity map; according to the disparity map, the position coordinates of the remote control device 100 facing the screen are obtained through 3D coordinate calculation.

Step 1044, according to the position coordinates, determine the target position corresponding to the position coordinates on the screen.

As shown in FIG. 1 , the algorithm module of the controlled device 200 sends the position coordinates to the screen, and the screen determines the target position corresponding to the position coordinates on the screen according to the position coordinates in the first information.

Step 106, perform a first operation on the screen according to the target position.

As shown in FIGS. 1 and 2 , the screen of the controlled device 200 executes the first operation at a target position corresponding to the position coordinates on the screen according to the position coordinates.

Wherein, the first operation includes: displaying a pointer at the target position; or highlighting or shadowing the control corresponding to the target position.

It should be noted that when the user moves the remote control device, the image collected by the remote control device changes, and the position coordinates need to be recalculated. The controlled device refreshes the pointer position according to the new position coordinates, and the pointer moves according to the direction of the remote control device; or, the controlled device The control device refreshes the highlighted or shaded controls according to the new position coordinates.

In the technical solution of the remote control method provided by the embodiment of the present invention, the first information of the remote control device is received; the target position of the remote control device facing the screen is determined according to the first information; and the first operation is performed on the screen according to the target position. The embodiment of the present invention enables the user to realize the remote operation of the controlled device without performing frequent mechanical operations on the five-way navigation key, thereby improving the experience of the user during use.

Based on the above architecture diagrams provided in FIG. 1 and FIG. 2 , an embodiment of the present invention provides a remote control method, which is applied to the remote control device 100 . Fig. 8 is a flow chart of a remote control method provided by another embodiment of the present invention. As shown in Figure 8, the method includes:

In step 202, first information is acquired, and the first information is used to determine a target position where the remote control device is directly facing the screen of the controlled device.

As an optional solution, the remote control device includes a binocular camera, and the first information includes the position coordinates of the target position of the remote control device facing the screen; as shown in FIG. 9, step 202 includes:

Step 2022, when the remote control device points to the screen, take an image including the outline of the screen through the binocular camera.

Wherein, the binocular camera is located at the front end of the remote control device and can recognize the outline of the screen, and two images containing the outline of the screen are obtained by shooting the screen through the binocular camera.

For example, binocular cameras include binocular depth-of-field cameras.

In the embodiment of the present invention, as shown in FIG. 2 , the remote control device 100 captures an image including the outline of the screen through a binocular camera, and sends the image to the algorithm module.

Step 2024, obtain the position coordinates of the target position where the remote control device is facing the screen according to the image.

In the embodiment of the present invention, as shown in FIG. 2 , the remote control device 100 captures an image including the outline of the screen through a binocular camera, and sends the image to the algorithm module. The algorithm module determines the position coordinates of the target position where the remote control device is facing the screen according to the image, and sends the position coordinates to the sending module. Specifically, the algorithm module calibrates the binocular camera in an offline state; corrects the two images on the screen to remove the distortion caused by the binocular camera; calculates the difference between the corrected two images through binocular matching Match the points to obtain a disparity map; according to the disparity map, the position coordinates of the remote control device 100 facing the screen are obtained through 3D coordinate calculation. The sending module generates first information according to the location coordinates.

As another optional solution, the remote control device includes a binocular camera, and the first information includes shooting the screen through the binocular camera to obtain two images containing the outline of the screen. Step 202 includes: when the remote control device points to the screen, through the binocular camera Take an image.

In the embodiment of the present invention, as shown in FIG. 1 , the remote control device 100 captures an image including the outline of the screen through a binocular camera, and sends the image to a sending module, and the sending module generates first information according to the image.

Step 204: Send the first information to the controlled device, so that the controlled device determines the target location according to the first information, and performs a first operation on the screen according to the target location.

As shown in FIG. 1 , the first information includes an image including the outline of the screen captured by the binocular camera of the remote control device 100 . The sending module of the remote control device 100 sends the first information to the receiving module of the controlled device 200 . The receiving module receives the first information and sends the first information to the algorithm module. The algorithm module determines the position coordinates of the target position where the remote control device 100 is facing the screen according to the first information, and sends the position coordinates to the screen. Specifically, the algorithm module calibrates the binocular camera in an offline state; corrects the two images on the screen to remove the distortion caused by the binocular camera; calculates the difference between the corrected two images through binocular matching Match the points to obtain a disparity map; according to the disparity map, the position coordinates of the remote control device 100 facing the screen are obtained through 3D coordinate calculation. The screen determines a target position corresponding to the position coordinates on the screen according to the position coordinates, and executes the first operation on the screen according to the target position.

As shown in FIG. 2 , the first information includes the position coordinates of the target position where the remote control device 100 is facing the screen. The sending module of the remote control device 100 sends the first information to the receiving module of the controlled device 200 . The receiving module receives the first information and sends the first information to the screen. The screen determines a target position corresponding to the position coordinates on the screen according to the position coordinates, and executes the first operation on the screen according to the target position.

It should be noted that when the user moves the remote control device, the image of the screen collected by the remote control device changes, and the position coordinates need to be recalculated, the controlled device refreshes the pointer position according to the new position coordinates, and the pointer moves according to the direction of the remote control device; or , the controlled device refreshes the highlighted or shaded controls according to the new position coordinates.

In the technical solution of the remote control method provided by the embodiment of the present invention, the remote control device sends the first information to the controlled device, so that the controlled device can determine the target position of the remote control device facing the screen according to the first information, and the target position on the screen according to the target position Perform the first action. The embodiment of the present invention enables the user to realize the remote operation of the controlled device without performing frequent mechanical operations on the five-way navigation key, thereby improving the experience of the user during use.

The remote control method provided by the embodiment of the present invention is described in detail above with reference to FIG. 1 to FIG. 9 , and the device embodiment of the present invention will be described in detail below in conjunction with FIG. 10 . It should be understood that the electronic device in the embodiment of the present invention can execute various methods in the foregoing embodiments of the present invention, that is, for specific working processes of the following various products, reference may be made to the corresponding processes in the foregoing method embodiments.

An embodiment of the present invention provides an electronic device, which may be a terminal device or a circuit device built in the terminal device. The electronic device may be used to execute the functions/steps in the foregoing method embodiments.

FIG. 10 is a schematic structural diagram of an electronic device 300 provided by an embodiment of the present invention. The electronic device 300 may include a processor 310, an external memory interface 320, an internal memory 321, a universal serial bus (universal serial bus, USB) interface 330, a charging management module 340, a power management module 341, a battery 342, an antenna 1, and an antenna 2 , mobile communication module 350, wireless communication module 360, audio module 370, speaker 370A, receiver 370B, microphone 370C, earphone jack 370D, sensor module 380, button 390, motor 391, indicator 392, camera 393, display screen 394, and A subscriber identification module (subscriber identification module, SIM) card interface 395 and the like. The sensor module 380 may include a pressure sensor 380A, a gyroscope sensor 380B, an air pressure sensor 380C, a magnetic sensor 380D, an acceleration sensor 380E, a distance sensor 380F, a proximity light sensor 380G, a fingerprint sensor 380H, a temperature sensor 380J, a touch sensor 380K, and an ambient light sensor. Sensor 380L, bone conduction sensor 380M, etc.

It can be understood that, the structure shown in the embodiment of the present invention does not constitute a specific limitation on the electronic device 300 . In other embodiments of the present invention, the electronic device 300 may include more or fewer components than shown, or combine certain components, or separate certain components, or arrange different components. The illustrated components can be realized in hardware, software or a combination of software and hardware.

The processor 310 may include one or more processing units, for example: the processor 310 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

The controller can generate an operation control signal according to the instruction opcode and timing signal, and complete the control of fetching and executing the instruction.

A memory may also be provided in the processor 310 for storing instructions and data. In some embodiments, the memory in processor 310 is a cache memory. The memory may hold instructions or data that the processor 310 has just used or recycled. If the processor 310 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 310 is reduced, thereby improving the efficiency of the system.

In some embodiments, processor 310 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transmitter (universal asynchronous receiver/transmitter, UART) interface, mobile industry processor interface (mobile industry processor interface, MIPI), general-purpose input and output (general-purpose input/output, GPIO) interface, subscriber identity module (subscriber identity module, SIM) interface, and /or universal serial bus (universal serial bus, USB) interface, etc.

The I2C interface is a bidirectional synchronous serial bus, including a serial data line (serial data line, SDA) and a serial clock line (derail clock line, SCL). In some embodiments, processor 310 may include multiple sets of I2C buses. The processor 310 can be respectively coupled to the touch sensor 380K, the charger, the flashlight, the camera 393 and so on through different I2C bus interfaces. For example, the processor 310 may be coupled to the touch sensor 380K through the I2C interface, so that the processor 310 and the touch sensor 380K communicate through the I2C bus interface to realize the touch function of the electronic device 300 .

The I2S interface can be used for audio communication. In some embodiments, processor 310 may include multiple sets of I2S buses. The processor 310 may be coupled to the audio module 370 through an I2S bus to implement communication between the processor 310 and the audio module 370 . In some embodiments, the audio module 370 can transmit audio signals to the wireless communication module 360 through the I2S interface, so as to realize the function of answering calls through the Bluetooth headset.

The PCM interface can also be used for audio communication, sampling, quantizing and encoding the analog signal. In some embodiments, the audio module 370 and the wireless communication module 360 may be coupled through a PCM bus interface. In some embodiments, the audio module 370 can also transmit audio signals to the wireless communication module 360 through the PCM interface, so as to realize the function of answering calls through the Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

The UART interface is a universal serial data bus used for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, a UART interface is generally used to connect the processor 310 and the wireless communication module 360 . For example: the processor 310 communicates with the Bluetooth module in the wireless communication module 360 through the UART interface to realize the Bluetooth function. In some embodiments, the audio module 370 can transmit audio signals to the wireless communication module 360 through the UART interface, so as to realize the function of playing music through the Bluetooth headset.

The MIPI interface can be used to connect the processor 310 with peripheral devices such as the display screen 394 and the camera 393 . MIPI interface includes camera serial interface (camera serial interface, CSI), display serial interface (display serial interface, DSI), etc. In some embodiments, the processor 310 communicates with the camera 393 through the CSI interface to realize the shooting function of the electronic device 300 . The processor 310 communicates with the display screen 394 through the DSI interface to realize the display function of the electronic device 300 .

The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 310 with the camera 393 , the display screen 394 , the wireless communication module 360 , the audio module 370 , the sensor module 380 and so on. The GPIO interface can also be configured as an I2C interface, I2S interface, UART interface, MIPI interface, etc.

The USB interface 330 is an interface conforming to the USB standard specification, specifically, it may be a Mini USB interface, a Micro USB interface, a USB Type C interface, and the like. The USB interface 330 can be used to connect a charger to charge the electronic device 300 , and can also be used to transmit data between the electronic device 300 and peripheral devices. It can also be used to connect headphones and play audio through them. This interface can also be used to connect other electronic devices, such as AR devices.

It can be understood that the interface connection relationship between the modules shown in the embodiment of the present invention is only a schematic illustration, and does not constitute a structural limitation of the electronic device 300 . In other embodiments of the present invention, the electronic device 300 may also adopt different interface connection manners in the foregoing embodiments, or a combination of multiple interface connection manners.

The charging management module 340 is configured to receive charging input from the charger. Wherein, the charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 340 can receive charging input from a wired charger through the USB interface 330 . In some wireless charging embodiments, the charging management module 340 may receive wireless charging input through a wireless charging coil of the electronic device 300 . While the charging management module 340 is charging the battery 342 , it can also supply power to the electronic device through the power management module 341 .

The power management module 341 is used for connecting the battery 342 , the charging management module 340 and the processor 310 . The power management module 341 receives the input from the battery 342 and/or the charging management module 340 to provide power for the processor 310 , the internal memory 321 , the display screen 394 , the camera 393 , and the wireless communication module 360 . The power management module 341 can also be used to monitor parameters such as battery capacity, battery cycle times, and battery health status (leakage, impedance). In some other embodiments, the power management module 341 may also be disposed in the processor 310 . In some other embodiments, the power management module 341 and the charging management module 340 may also be set in the same device.

The wireless communication function of the electronic device 300 can be realized by the antenna 1 , the antenna 2 , the mobile communication module 350 , the wireless communication module 360 , a modem processor, a baseband processor, and the like.

Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 300 may be used to cover single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example: Antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 350 can provide wireless communication solutions including 2G/3G/4G/5G applied on the electronic device 300 . The mobile communication module 350 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA) and the like. The mobile communication module 350 can receive electromagnetic waves through the antenna 1, filter and amplify the received electromagnetic waves, and send them to the modem processor for demodulation. The mobile communication module 350 can also amplify the signal modulated by the modem processor, convert it into electromagnetic wave and radiate it through the antenna 1 . In some embodiments, at least part of the functional modules of the mobile communication module 350 may be set in the processor 310 . In some embodiments, at least part of the functional modules of the mobile communication module 350 and at least part of the modules of the processor 310 may be set in the same device.

A modem processor may include a modulator and a demodulator. Wherein, the modulator is used for modulating the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low frequency baseband signal. Then the demodulator sends the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal is passed to the application processor after being processed by the baseband processor. The application processor outputs sound signals through audio equipment (not limited to speaker 370A, receiver 370B, etc.), or displays images or videos through display screen 394 . In some embodiments, the modem processor may be a stand-alone device. In some other embodiments, the modem processor may be independent from the processor 310, and be set in the same device as the mobile communication module 350 or other functional modules.

The wireless communication module 360 can provide wireless local area networks (wireless local area networks, WLAN) (such as wireless fidelity (Wireless Fidelity, Wi-Fi) network), bluetooth (bluetooth, BT), global navigation satellite, etc. applied on the electronic device 300. System (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication module 360 may be one or more devices integrating at least one communication processing module. The wireless communication module 360 receives electromagnetic waves via the antenna 2 , frequency-modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 310 . The wireless communication module 360 can also receive the signal to be sent from the processor 310 , frequency-modulate it, amplify it, and convert it into electromagnetic waves through the antenna 2 for radiation.

In some embodiments, the antenna 1 of the electronic device 300 is coupled to the mobile communication module 350, and the antenna 2 is coupled to the wireless communication module 360, so that the electronic device 300 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (general packet radio service, GPRS), code division multiple access (code division multiple access, CDMA), broadband Code division multiple access (wideband code division multiple access, WCDMA), time division code division multiple access (time-division code division multiple access, TD-SCDMA), long term evolution (long term evolution, LTE), BT, GNSS, WLAN, NFC , FM, and/or IR techniques, etc. The GNSS may include a global positioning system (global positioning system, GPS), a global navigation satellite system (global navigation satellite system, GLONASS), a Beidou navigation satellite system (beidou navigation satellite system, BDS), a quasi-zenith satellite system (quasi -zenith satellite system (QZSS) and/or satellite based augmentation systems (SBAS).

The electronic device 300 implements a display function through a GPU, a display screen 394, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 394 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 310 may include one or more GPUs that execute program instructions to generate or alter display information.

The display screen 394 is used to display images, videos and the like. Display 394 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light emitting diode or an active matrix organic light emitting diode (active-matrix organic light emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diodes (quantum dot light emitting diodes, QLED), etc. In some embodiments, the electronic device 300 may include 1 or N display screens 394, where N is a positive integer greater than 1.

The electronic device 300 can realize the shooting function through an ISP, a camera 393 , a video codec, a GPU, a display screen 394 , and an application processor.

The ISP is used for processing the data fed back by the camera 393 . For example, when taking a picture, open the shutter, the light is transmitted to the photosensitive element of the camera through the lens, and the light signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be located in the camera 393 .

Camera 393 is used to capture still images or video. The object generates an optical image through the lens and projects it to the photosensitive element. The photosensitive element can be a charge coupled device (charge coupled device, CCD) or a complementary metal-oxide-semiconductor (complementary metal-oxide-semiconductor, CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. DSP converts digital image signals into standard RGB, YUV and other image signals. In some embodiments, the electronic device 300 may include 1 or N cameras 393, where N is a positive integer greater than 1.

Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the electronic device 300 selects a frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.

Video codecs are used to compress or decompress digital video. The electronic device 300 may support one or more video codecs. In this way, the electronic device 300 can play or record videos in various encoding formats, for example: moving picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4 and so on.

The NPU is a neural-network (NN) computing processor. By referring to the structure of biological neural networks, such as the transfer mode between neurons in the human brain, it can quickly process input information and continuously learn by itself. Applications such as intelligent cognition of the electronic device 300 can be realized through the NPU, such as image recognition, face recognition, speech recognition, text understanding, and the like.

The external memory interface 320 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 300. The external memory card communicates with the processor 310 through the external memory interface 320 to implement a data storage function. Such as saving music, video and other files in the external memory card.

The internal memory 321 may be used to store computer-executable program code, which includes instructions. The internal memory 321 may include an area for storing programs and an area for storing data. Wherein, the stored program area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.) and the like. The storage data area can store data (such as audio data, phone book, etc.) created during the use of the electronic device 300 . In addition, the internal memory 321 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (universal flash storage, UFS) and the like. The processor 310 executes various functional applications and data processing of the electronic device 300 by executing instructions stored in the internal memory 321 and/or instructions stored in a memory provided in the processor.

The electronic device 300 can implement audio functions through an audio module 370 , a speaker 370A, a receiver 370B, a microphone 370C, an earphone interface 370D, and an application processor. Such as music playback, recording, etc.

The audio module 370 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 370 may also be used to encode and decode audio signals. In some embodiments, the audio module 370 can be set in the processor 310 , or some functional modules of the audio module 370 can be set in the processor 310 .

Speaker 370A, also called "horn", is used to convert audio electrical signals into sound signals. Electronic device 300 can listen to music through speaker 370A, or listen to hands-free calls.

Receiver 370B, also called "earpiece", is used to convert audio electrical signals into audio signals. When the electronic device 300 receives a call or a voice message, the receiver 370B can be placed close to the human ear to receive the voice.

The microphone 370C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can put his mouth close to the microphone 370C to make a sound, and input the sound signal to the microphone 370C. The electronic device 300 may be provided with at least one microphone 370C. In some other embodiments, the electronic device 300 may be provided with two microphones 370C, which may also implement a noise reduction function in addition to collecting sound signals. In some other embodiments, the electronic device 300 can also be provided with three, four or more microphones 370C, so as to collect sound signals, reduce noise, identify sound sources, and realize directional recording functions, etc.

The earphone interface 370D is used to connect wired earphones. The earphone interface 370D may be a USB interface 330, or a 3.5mm open mobile terminal platform (open mobile terminal platform, OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

The pressure sensor 380A is used to sense the pressure signal and convert the pressure signal into an electrical signal. In some embodiments, pressure sensor 380A may be located on display screen 394 .

There are many types of pressure sensors 380A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may be comprised of at least two parallel plates with conductive material. When a force is applied to pressure sensor 380A, the capacitance between the electrodes changes. The electronic device 300 determines the intensity of pressure according to the change in capacitance. When a touch operation acts on the display screen 394, the electronic device 300 detects the intensity of the touch operation according to the pressure sensor 380A. The electronic device 300 may also calculate the touched position according to the detection signal of the pressure sensor 380A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities may correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view short messages is executed. When a touch operation whose intensity is greater than or equal to the first pressure threshold acts on the icon of the short message application, the instruction of creating a new short message is executed.

The gyro sensor 380B can be used to determine the motion posture of the electronic device 300 . In some embodiments, the angular velocity of the electronic device 300 about three axes (ie, x, y and z axes) may be determined by the gyro sensor 380B. The gyro sensor 380B can be used for image stabilization. Exemplarily, when the shutter is pressed, the gyro sensor 380B detects the shaking angle of the electronic device 300, calculates the distance that the lens module needs to compensate according to the angle, and allows the lens to counteract the shaking of the electronic device 300 through reverse movement to achieve anti-shake. The gyroscope sensor 380B can also be used for navigation and somatosensory game scenes.

The air pressure sensor 380C is used to measure air pressure. In some embodiments, the electronic device 300 calculates the altitude based on the air pressure value measured by the air pressure sensor 380C to assist positioning and navigation.

The magnetic sensor 380D includes a Hall sensor. The electronic device 300 may use the magnetic sensor 380D to detect the opening and closing of the flip leather case. In some embodiments, when the electronic device 300 is a flip machine, the electronic device 300 can detect opening and closing of the flip according to the magnetic sensor 380D. Furthermore, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, features such as automatic unlocking of the flip cover are set.

The acceleration sensor 380E can detect the acceleration of the electronic device 300 in various directions (generally three axes). The magnitude and direction of gravity can be detected when the electronic device 300 is stationary. It can also be used to identify the posture of electronic devices, and can be used in applications such as horizontal and vertical screen switching, pedometers, etc.

The distance sensor 380F is used to measure the distance. The electronic device 300 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 300 can use the distance sensor 380F to measure distance to achieve fast focusing.

Proximity light sensor 380G may include, for example, light emitting diodes (LEDs) and light detectors, such as photodiodes. The light emitting diodes may be infrared light emitting diodes. The electronic device 300 emits infrared light through the light emitting diode. Electronic device 300 uses photodiodes to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it may be determined that there is an object near the electronic device 300. When insufficient reflected light is detected, the electronic device 300 may determine that there is no object near the electronic device 300 . The electronic device 300 can use the proximity light sensor 380G to detect that the user holds the electronic device 300 close to the ear to make a call, so as to automatically turn off the screen to save power. The proximity light sensor 380G can also be used in leather case mode, automatic unlock and lock screen in pocket mode.

The ambient light sensor 380L is used for sensing ambient light brightness. The electronic device 300 can adaptively adjust the brightness of the display screen 394 according to the perceived ambient light brightness. The ambient light sensor 380L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 380L can also cooperate with the proximity light sensor 380G to detect whether the electronic device 300 is in the pocket to prevent accidental touch.

The fingerprint sensor 380H is used to collect fingerprints. The electronic device 300 can use the collected fingerprint characteristics to implement fingerprint unlocking, access to the application lock, take pictures with the fingerprint, answer calls with the fingerprint, and the like.

The temperature sensor 380J is used to detect temperature. In some embodiments, the electronic device 300 uses the temperature detected by the temperature sensor 380J to implement a temperature treatment strategy. For example, when the temperature reported by the temperature sensor 380J exceeds the threshold, the electronic device 300 may reduce the performance of the processor located near the temperature sensor 380J, so as to reduce power consumption and implement thermal protection. In some other embodiments, when the temperature is lower than another threshold, the electronic device 300 heats the battery 342 to avoid abnormal shutdown of the electronic device 300 caused by the low temperature. In some other embodiments, when the temperature is lower than another threshold, the electronic device 300 boosts the output voltage of the battery 342 to avoid abnormal shutdown caused by low temperature.

Touch sensor 380K, also known as "touch device". The touch sensor 380K can be arranged on the display screen 394, and the touch sensor 380K and the display screen 394 form a touch screen, also called “touch screen”. The touch sensor 380K is used to detect a touch operation on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to touch operations can be provided through the display screen 394 . In other embodiments, the touch sensor 380K may also be disposed on the surface of the electronic device 300 , which is different from the position of the display screen 394 .

The bone conduction sensor 380M can acquire vibration signals. In some embodiments, the bone conduction sensor 380M can acquire the vibration signal of the vibrating bone mass of the human voice. The bone conduction sensor 380M can also contact the human pulse and receive the blood pressure beating signal. In some embodiments, the bone conduction sensor 380M can also be disposed in the earphone, combined into a bone conduction earphone. The audio module 370 can analyze the voice signal based on the vibration signal of the vibrating bone mass of the vocal part acquired by the bone conduction sensor 380M, so as to realize the voice function. The application processor can analyze the heart rate information based on the blood pressure beating signal acquired by the bone conduction sensor 380M, so as to realize the heart rate detection function.

The keys 390 include a power key, a volume key and the like. The key 390 may be a mechanical key. It can also be a touch button. The electronic device 300 may receive key input and generate key signal input related to user settings and function control of the electronic device 300 .

The motor 391 can generate a vibrating prompt. The motor 391 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations applied to different applications (such as taking pictures, playing audio, etc.) may correspond to different vibration feedback effects. The motor 391 can also correspond to different vibration feedback effects for touch operations acting on different areas of the display screen 394 . Different application scenarios (for example: time reminder, receiving information, alarm clock, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

The indicator 392 can be an indicator light, which can be used to indicate the charging status, the change of the battery capacity, and can also be used to indicate messages, missed calls, notifications and the like.

The SIM card interface 395 is used for connecting a SIM card. The SIM card can be connected and separated from the electronic device 300 by inserting it into the SIM card interface 395 or pulling it out from the SIM card interface 395 . The electronic device 300 may support 1 or N SIM card interfaces, where N is a positive integer greater than 1. SIM card interface 395 can support Nano SIM card, Micro SIM card, SIM card etc. Multiple cards can be inserted into the same SIM card interface 395 at the same time. The types of the multiple cards may be the same or different. The SIM card interface 395 is also compatible with different types of SIM cards. The SIM card interface 395 is also compatible with external memory cards. The electronic device 300 interacts with the network through the SIM card to implement functions such as calling and data communication. In some embodiments, the electronic device 300 adopts an eSIM, that is, an embedded SIM card. The eSIM card can be embedded in the electronic device 300 and cannot be separated from the electronic device 300 .

An embodiment of the present invention provides a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when the instructions are run on a terminal device, the terminal device is made to perform the functions/steps in the foregoing method embodiments.

The embodiment of the present invention also provides a computer program product containing instructions, and when the computer program product is run on a computer or any at least one processor, it causes the computer to execute the functions/steps in the above method embodiments.

In the embodiments of the present invention, "at least one" means one or more, and "multiple" means two or more. "And/or" describes the association relationship of associated objects, indicating that there may be three kinds of relationships, for example, A and/or B may indicate that A exists alone, A and B exist simultaneously, or B exists alone. Among them, A and B can be singular or plural. The character "/" generally indicates that the contextual objects are an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, and c may represent: a, b, c, a-b, a-c, b-c, or a-b-c, wherein a, b, and c may be single or multiple.

Those of ordinary skill in the art can appreciate that each unit and algorithm steps described in the embodiments disclosed herein can be realized by a combination of electronic hardware, computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present invention.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the several embodiments provided by the present invention, if any function is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make an electronic device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disc and other media that can store program codes. .

The above is only a specific implementation of the present invention, and any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present invention, which should be covered by the protection scope of the present invention. The protection scope of the present invention shall be determined by the protection scope of the claims.

Claims

A remote control method, characterized in that it is applied to a controlled device, the controlled device includes a screen, and the method includes:

receiving first information of the remote control device;

determining a target position where the remote control device is facing the screen according to the first information;

A first operation is performed on the screen according to the target position.
The method according to claim 1, wherein the first information includes position coordinates of the target position where the remote control device is facing the screen;

Determining the target position where the remote control device is facing the screen according to the first information includes:

According to the position coordinates, the target position corresponding to the position coordinates on the screen is determined.
The method according to claim 1, wherein the position coordinates of the target position are obtained from an image including the outline of the screen captured by a binocular camera of the remote control device.
The method according to claim 3, wherein the first information includes an image including the outline of the screen captured by the binocular camera of the remote control device;

The determining the target position where the remote control device is facing the screen according to the first information includes: determining the target position on the screen corresponding to the position coordinates according to the image.
The method according to claim 3, wherein the determining the target position on the screen corresponding to the position coordinates according to the image comprises:

Obtaining the position coordinates of the target position where the remote control device is facing the screen according to the image;

According to the position coordinates, the target position corresponding to the position coordinates on the screen is determined.
The method according to claim 2 or 4, wherein the first operation comprises: displaying a pointer at the target position; or highlighting or shadowing the control corresponding to the target position.
A remote control method, characterized in that it is applied to a remote control device, the method comprising:

Acquiring first information, where the first information is used to determine the target position of the remote control device facing the screen of the controlled device;

sending the first information to the controlled device, so that the controlled device determines the target position according to the first information, and executes a first operation on the screen according to the target position.
The method according to claim 7, wherein the remote control device includes a binocular camera, and the first information includes position coordinates of the target position where the remote control device is facing the screen;

Get first information, including:

When the remote control device points to the screen, the binocular camera captures an image containing the outline of the screen;

The position coordinates of the target position where the remote control device is facing the screen are obtained according to the image.
The method according to claim 7, wherein the remote control device includes a binocular camera, and the first information includes an image including the outline of the screen captured by the binocular camera;

Get first information, including:

When the remote control device points to the screen, the image is captured by the binocular camera.
The method according to claim 7, wherein the first operation comprises: displaying a pointer at the target position; or highlighting or shadowing the control corresponding to the target position.
A remote control device, characterized in that it is applied to a controlled device, the controlled device includes a screen, and the device includes:

A receiving module, configured to receive the first information of the remote control device;

A determining module, configured to determine a target position where the remote control device is facing the screen according to the first information;

An executing module, configured to execute a first operation on the screen according to the target position.
The apparatus according to claim 11, wherein the first information includes position coordinates of the target position where the remote control device is facing the screen;

The determining module is specifically configured to determine the target position corresponding to the position coordinates on the screen according to the position coordinates.
The device according to claim 11, wherein the position coordinates of the target position are obtained from an image including the outline of the screen captured by the binocular camera of the remote control device.
The device according to claim 13, wherein the first information includes an image including the outline of the screen captured by the binocular camera of the remote control device;

The determination module is specifically configured to determine the target position corresponding to the position coordinates on the screen according to the image.
The device according to claim 13, wherein the determination module is specifically configured to: obtain the position coordinates of the target position where the remote control device is facing the screen according to the image, and Coordinates, determining the target position on the screen corresponding to the position coordinates.
The device according to claim 12 or 14, wherein the first operation comprises: displaying a pointer at the target position; or highlighting or shading a control corresponding to the target position.
A remote control device is characterized in that it is applied to remote control equipment, and the device includes:

An acquiring module, configured to acquire first information, and the first information is used to determine a target position where the remote control device faces the screen of the controlled device;

a sending module, configured to send the first information to the controlled device, so that the controlled device determines the target position according to the first information, and executes on the screen according to the target position first operation.
The device according to claim 17, wherein the remote control device includes a binocular camera, and the first information includes position coordinates of the target position where the remote control device is facing the screen;

The acquisition module is specifically configured to, when the remote control device points to the screen, take an image containing the outline of the screen through the binocular camera, and obtain that the remote control device is facing the screen according to the image The location coordinates of the target location for .
The device according to claim 17, wherein the remote control device includes a binocular camera, and the first information includes an image including the outline of the screen captured by the binocular camera;

The acquisition module is specifically configured to capture the image through the binocular camera when the remote control device points to the screen.
The device according to claim 17, wherein the first operation comprises: displaying a pointer at the target position; or highlighting or shading a control corresponding to the target position.
A controlled device is characterized in that it includes a screen, a processor, and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and when the processor runs the program instructions, the The controlled device performs the following steps:

receiving first information of the remote control device;

determining a target position where the remote control device is facing the screen according to the first information;

A first operation is performed on the screen according to the target position.
The device according to claim 21, wherein when the processor executes the program instructions, the controlled device is made to perform the following steps:

The first information includes position coordinates of the target position where the remote control device is facing the screen;

Determining the target position where the remote control device is facing the screen according to the first information includes:

According to the position coordinates, the target position corresponding to the position coordinates on the screen is determined.
The device according to claim 21, wherein the position coordinates of the target position are obtained from an image including the outline of the screen captured by a binocular camera of the remote control device.
The device according to claim 23, wherein when the processor executes the program instructions, the controlled device is made to perform the following steps:

The first information includes an image including the outline of the screen captured by the binocular camera of the remote control device;

The determining the target position where the remote control device is facing the screen according to the first information includes: determining the target position on the screen corresponding to the position coordinates according to the image.
The device according to claim 23, wherein when the processor executes the program instructions, the controlled device is made to perform the following steps:

The determining the target position corresponding to the position coordinates on the screen according to the image includes:

Obtaining the position coordinates of the target position where the remote control device is facing the screen according to the image;

According to the position coordinates, the target position corresponding to the position coordinates on the screen is determined.
The device according to claim 22 or 24, wherein the first operation comprises: displaying a pointer at the target position; or, highlighting or shadowing a control corresponding to the target position.
A remote control device, characterized in that it includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and when the processor runs the program instructions, the remote control The device performs the following steps:

Acquiring first information, where the first information is used to determine the target position of the remote control device facing the screen of the controlled device;

sending the first information to the controlled device, so that the controlled device determines the target position according to the first information, and executes a first operation on the screen according to the target position.
The device according to claim 27, wherein the remote control device includes a binocular camera, and the first information includes the position coordinates of the target position where the remote control device is facing the screen; when the processing When the controller runs the program instructions, the remote control device is made to perform the following steps:

Get first information, including:

When the remote control device points to the screen, the binocular camera captures an image containing the outline of the screen;

The position coordinates of the target position where the remote control device is facing the screen are obtained according to the image.
The device according to claim 27, wherein the remote control device includes a binocular camera, and the first information includes an image including the outline of the screen captured by the binocular camera; when the processor runs the When the above program instructions are used, the remote control device is made to perform the following steps:

Get first information, including:

When the remote control device points to the screen, the image is captured by the binocular camera.
The device according to claim 27, wherein the first operation comprises: displaying a pointer at the target position; or highlighting or shadowing the control corresponding to the target position.