WO2023040449A1

WO2023040449A1 - Triggering of client operation instruction by using fitness action

Info

Publication number: WO2023040449A1
Application number: PCT/CN2022/106220
Authority: WO
Inventors: 蒋晨之; 宝鹏庆; 王维强
Original assignee: 支付宝(杭州)信息技术有限公司
Priority date: 2021-09-17
Filing date: 2022-07-18
Publication date: 2023-03-23
Also published as: CN113849065A

Abstract

Provided in one or more embodiments of the present specification are a method and apparatus for triggering a client operation instruction by using a fitness action. The method comprises: acquiring motion posture data of a user; recognizing a fitness action corresponding to the motion posture data; and determining, according to a preset correlation between operation instructions and fitness actions, an operation instruction corresponding to the fitness action, and executing the corresponding operation instruction. In the embodiments of the present specification, an operation instruction is triggered by using a fitness action, thereby increasing the modes of interaction between a user and a client, and improving the user experience.

Description

Use fitness actions to trigger client operation instructions

technical field

The embodiments of this specification relate to the field of computer applications, and in particular to a method and device for triggering client operation instructions by using fitness actions.

Background technique

The client, also known as the client, refers to an application that corresponds to the server and provides local services for users, and needs to cooperate with the server to run.

The client is generally installed on the user terminal, and the user generally triggers an operation command of the client by touching the screen or pressing a specific button on the mobile terminal. However, the current interaction method is too single, which affects the user experience.

Contents of the invention

In view of this, the embodiments of this specification provide a method and device for triggering an operation command of a client terminal by using a fitness action.

According to the first aspect of the embodiments of this specification, a method for triggering client operation instructions by using fitness actions is proposed, including: determining the correspondence between preset operation instructions and fitness actions; wherein each operation instruction corresponds to at least one Fitness action, the fitness action corresponding to any two operation instructions does not overlap; obtain the user's current exercise posture data; identify the fitness movement corresponding to the acquired exercise posture data; determine and execute according to the corresponding relationship between the preset operation instruction and fitness action Operation instructions corresponding to fitness actions.

According to the second aspect of the present specification, a device for triggering client operation instructions by using fitness actions is proposed, including: a correspondence determination module for determining the correspondence between preset operation instructions and fitness actions; wherein each operation instruction Corresponding to at least one fitness action, the fitness actions corresponding to any two operation instructions do not overlap; the exercise posture data acquisition module is used to obtain the user's current exercise posture data; the fitness action recognition module is used to identify the acquired exercise posture data corresponding to Fitness action; an operation instruction execution module, configured to determine and execute the operation instruction corresponding to the exercise action according to the preset corresponding relationship between the operation instruction and the exercise action.

According to a third aspect of the embodiments of this specification, there is provided a computer-readable storage medium, on which computer instructions are stored, and when the instructions are executed by a processor, the above-mentioned method for triggering an operation instruction of a client by using a fitness action is implemented.

According to a fourth aspect of the embodiments of this specification, there is provided an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein, the processor executes the executable instructions to realize the above-mentioned utilization of fitness Action triggers the method of client operation instruction.

In the embodiment of this specification, by acquiring the user's exercise posture data and identifying the fitness action corresponding to the exercise posture data, according to the corresponding relationship between the preset operation instruction and the fitness movement, determine the operation instruction corresponding to the fitness movement, and execute the corresponding operating instructions. The embodiment of this specification utilizes fitness actions to trigger operation instructions, increases the interaction mode between the user and the client, and improves user experience.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the specification.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and together with the description serve to explain the principles of the specification.

Fig. 1 is a flow chart of a method for triggering an operation instruction of a client by using a fitness action according to an exemplary embodiment of the present specification.

Fig. 2 is a schematic diagram of identifying a user's fitness action according to an embodiment of this specification.

Fig. 3 is a block diagram of an apparatus for triggering an operation instruction of a client by using a fitness action according to an exemplary embodiment of the present specification.

Fig. 4 is a hardware structural diagram of a computer device where an apparatus for triggering an operation command of a client terminal by using a fitness action according to an exemplary embodiment of this specification is located.

Detailed ways

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. Implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with aspects of one or more embodiments of the present specification as recited in the appended claims.

It should be noted that in other embodiments, the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification. In some other embodiments, the method may include more or less steps than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; multiple steps described in this specification may also be combined into a single step in other embodiments describe.

In related technologies, for the client software installed on the mobile terminal, the user generally interacts with the client software by touching the screen or pressing a specific button on the mobile terminal, in other words, the user triggers the client software through the above-mentioned method operating instructions to complete the interaction with the client software. For the client software installed on other user terminals (such as computers, TVs, etc.), the user generally completes the interaction with the client software through auxiliary tools (such as mouse keyboard or remote control, etc.). Tool to select a specific button on the page to trigger an operation command and complete the interaction.

It can be seen that the current interaction method is relatively single, which affects the user experience. In order to improve the user experience, it is considered that the fitness action can be combined with the trigger operation command, so that the user can exercise while using the client. Specifically, by acquiring the user's exercise posture data, and identifying the fitness action corresponding to the exercise posture data, according to the correspondence between the preset operation instruction and the fitness action, determine the operation instruction corresponding to the exercise action, and perform the corresponding operation instruction. The embodiment of this specification utilizes fitness actions to trigger operation instructions, increases the interaction mode between the user and the client, and improves user experience.

Next, a method for triggering an operation command of a client terminal provided by this specification will be described in detail.

As shown in Figure 1, this specification provides a method for using fitness actions to trigger client operation instructions, including the following steps: Step 101, determine the correspondence between preset operation instructions and fitness actions; where each operation instruction corresponds to At least one fitness movement, and the fitness movements corresponding to any two operation instructions do not overlap.

Wherein, the execution body of the method may be client software installed on the mobile terminal, such as applied to smart phones, tablet computers, smart watches and other wearable devices. In some cases, it can also be applied to client software installed on non-mobile user terminals, such as personal computers, TVs, etc. When users use the client, they can freely choose to use the traditional touch method. Trigger the client's operation instructions, or trigger the client's operation instructions through fitness actions. After enabling triggering operation instructions through fitness actions, in addition to performing step 101, if the user terminal where the client is located has a speaker or can connect to a wireless headset, it can also remind the user through voice, such as reminding the user of the action standard or reminder The placement of the user's mobile phone, etc.; if the picture of the user terminal where the client is located can be seen by the user in fitness, the user can also be reminded through the picture.

The user enables the operation command triggered by fitness action, which can be set on the setting page of the client; it can also be controlled by a specific gesture, such as pulling down the current page to the bottom to enable the operation command triggered by fitness action, or on the page Draw a big triangle on the top to trigger operation instructions through fitness actions; if the client or user terminal has a built-in voice assistant, the user can also enable the activation of exercise actions to trigger operation instructions by talking to the voice assistant; if the user terminal In the case of at least one button on the keyboard, it is also possible to trigger an operation command through a fitness action by pressing a specified button or a specified combination of buttons. Of course, the specified button or the combination of specified buttons has not been given other functions in advance. This specification of the user terminal does not limit the method of triggering the operation instruction of the client through fitness actions.

The user enables triggering operation commands through fitness actions, which can be enabled in the entire client to trigger operation commands through fitness actions, or only in a certain module of the client. For example, in the client Open it in an embedded applet, or open it in some interface of the client. For example, for an electronic wallet client, it can be opened on a certain carbon saving interface of the client. Of course, users can choose which interfaces to open.

After the user triggers the operation command through the fitness action, in order to determine what kind of operation command the user is currently doing to trigger the fitness action, the client needs to perform step 101 to determine the corresponding relationship between the preset operation command and the fitness action , so as to determine the operation instruction through the corresponding relationship.

If a fitness action corresponds to multiple operation instructions, it will be impossible to determine which operation instruction the user wants to trigger when the user performs the exercise action. Therefore, a fitness action will only correspond to one operation instruction. In addition, in order to further improve user experience, one operation instruction can be selected to correspond to multiple fitness actions.

Wherein, each operation instruction corresponds not only to a fitness movement, but also to a standard of the fitness movement. The fitness action standard can be the execution frequency of the fitness action, or the range of motion, etc. For example, if the fitness action is a high leg raise, you can set the frequency of the fitness action execution to 2 times per second. If the fitness action is a squat, You can set the range of motion for squatting.

The standard of fitness actions can be set according to the user's age and physical condition. For older users, the frequency of fitness actions should not be too high to prevent users from accidents during the completion of fitness actions; For the best users, the frequency and range of fitness movements should not be too high to prevent accidents.

When the user performs a fitness movement, but the frequency or amplitude of the fitness movement does not meet the standard of the corresponding fitness movement, the user may be prompted by voice, vibration or other forms that the current fitness movement is not standard. Specifically, when the client detects that the user has performed a certain fitness action, it judges whether the fitness action performed by the user meets the standard of the fitness action, and if it does not meet the standard of the fitness action, through a method To remind the user that the current fitness action is not standard.

In addition, how to obtain the corresponding relationship between specific operation instructions and fitness actions will be described in detail below, and will not be repeated here.

Step 103, acquiring the current motion posture data of the user.

Step 105, identifying the fitness action corresponding to the acquired exercise posture data.

Next, step 103-step 105 will be uniformly described. Specifically, the method of acquiring the user's motion posture data and identifying the user's fitness action, this manual provides the following two acquisition methods. Of course, these two methods do not represent limitations to this manual. If other methods can also obtain motion posture If the data is recognized and the fitness action is recognized, it is also possible to use other acquisition methods for acquisition and recognition.

The first method is to acquire through the camera module of the user terminal, and use a machine learning method to identify the fitness action performed by the user.

In other words, step 103 is specifically: acquiring motion posture data captured by the camera module of the user terminal. Step 105 is specifically: obtaining a pre-trained feature extraction model and a pre-trained action recognition model; inputting the motion gesture data of the user into the feature extraction model to extract the action features of the motion gesture data; Input the action recognition model to determine the fitness action corresponding to the user's exercise posture data.

Wherein, the camera module of the user terminal may be a self-contained camera module of the user terminal, or may be an external camera module of the user terminal. After the user triggers the operation command through fitness actions and the camera module is in place, the user can be reminded to stand at the position where the camera module can capture the whole body through sound, vibration or image. The range that the module can capture can also remind the user that it has exceeded the range through sound, vibration, or text or images on the display interface, and prompt the user to move to enter the shooting range of the camera module.

In addition, since the user interacts with the client through fitness actions, if the camera module of the mobile terminal is used, the user may be too far away from the mobile terminal in order to capture the whole body of the user, and cannot see the image on the mobile terminal. The content cannot be well interacted. Therefore, when the screen of the mobile terminal is small, after the user activates the fitness action to trigger the operation command, the user may be prompted to use the screen projection method to project the content of the mobile terminal client to a larger screen; or the user may be prompted to use Take pictures with an external camera; or prompt the user to stand in front of the mirror, make the screen face the user, and use the rear camera of the mobile terminal to capture the mirrored action of the user in the mirror. Through the above method, on the basis of the complete image of the user obtained by the camera module, the user can also see the content on the screen of the mobile terminal, so that the user can have a better experience.

Wherein, the acquired motion posture data is the real-time motion posture data of the user, not the motion posture data recorded in the past. In order to ensure that the acquired motion posture data is real real-time data, it is also possible to add a live body detection function when obtaining motion posture data. For example, live body detection can be used while shooting data to detect whether the currently captured video is the user's Real-time motion posture data. In the case that the currently captured video does not include a living body, the current data is not analyzed, and the user is prompted that there is no living body currently by voice or other methods.

The feature extraction of the user's motion posture data can be realized by various methods, and the extracted action features can also include various forms, such as, action features, which are the positional features of human body key points in each frame image, and the human body key points It is used to identify human actions. In this case, the extracted action features can be shown in FIG. 2 . It can be seen that the key points of the human body can include joint points such as ankles, knees, shoulders, elbows, hip bones, and elbows, as well as key points used to identify facial movements.

Identifying key points can be identified by gesture detection models MoveNet, BlazePose or PoseNet. Of course, in order to reduce the number of functions integrated on the client side, key points can also be detected through auxiliary tools, for example, special stickers can be pasted on key points, and the user's posture can be determined by the position of the special sticker on the device.

In addition, in addition to the above-mentioned acquisition of motion posture data by means of photography, if the user terminal is a mobile terminal or a wearable device, the user's motion can also be detected according to the sensor on the mobile terminal to obtain the user's motion posture data .

Specifically, step 103 includes: obtaining a measurement sequence of a motion sensor of the mobile terminal within a preset time period, and using the measurement sequence as motion posture data; the motion sensor is a sensor for detecting dynamic movement of the mobile terminal. Step 105 includes: acquiring a pre-trained action recognition model; inputting the measurement sequence into the action recognition model, and determining an action identifier corresponding to the measurement sequence.

Wherein, the motion sensor may be an acceleration sensor, a gyroscope, and other sensors already installed in the mobile terminal itself. Currently, most smart mobile terminals and wearable devices are equipped with an acceleration sensor, a gyroscope, and the like. Through the measurement sequence of the motion sensor, the magnitude and direction of the velocity can be calculated through the velocity formula, and then the motion trajectory of the mobile terminal or wearable device can be obtained through the displacement formula. The measurement sequence can be the measured value of the sensor obtained directly, or It is the motion trajectory obtained according to the velocity formula and displacement formula.

In the case that the user terminal is a mobile terminal, in order to enable the motion sensor to obtain a measurement sequence, the client needs to be fixed on a certain part of the body, such as the user's forearm, forearm or thigh. If it is fixed, it can be fixed by a strap. If the mobile terminal is not large, you can also put the mobile terminal in the user's tight pocket (to prevent it from shaking in the pocket and affect the final measurement result). Of course, you can also place the mobile terminal The mobile terminal is held in the hand for measurement.

In addition, considering that the method provided in this specification is a method for the user to interact with the client, and when the motion sensor on the mobile terminal or wearable device is used to measure, the user cannot see the motion of the mobile terminal or wearable device. For the display interface, in order to enable the user to interact with the client, the user can be prompted to cast the screen, the display interface of the mobile terminal or wearable device can be displayed on the screen, or the visual interaction can be replaced by voice broadcast to complete the user and Client interaction.

In the above method, the action recognition model is a pre-trained neural network model used to identify fitness actions. Different users have different actions. For each user, in order to enable the action recognition model to successfully identify the specific fitness actions of the user, the user can be prompted to record in advance before the recognition. Videos/measurement sequences of users doing different fitness actions, and the action/sensor sequence recognition model is updated through these videos, so that the action recognition model of each user's client can be more suitable for the user.

Step 107, according to the preset correspondence between the operation instruction and the exercise action, determine and execute the operation instruction corresponding to the exercise action.

After the instruction is completed, the user may also be prompted by voice that the operation instruction has been completed, so that the user can get feedback.

In addition, in order to increase users' willingness to use this function, some rewards can be set to attract users to use this function. For example, if this function is used in the takeaway applet of an e-wallet client, every time the user completes 1000 squat actions (Here is just an example, and does not represent a limitation on fitness actions), reward users with certain points (points can be exchanged for red envelopes). If this function is used on a certain carbon saving statistics interface of an e-wallet client, the user can increase the amount of stolen energy in "stealing energy" through fitness actions.

In order to improve the user's willingness to use this function, it is also possible to: add a ranking function to this function. The rankings that each user can see include other users who have a friend relationship with the user, and each of the user and these other users. Daily exercise time/number of times of exercise, for example, the ranking that can be seen by the user may include the daily number of squats of users who have a friend relationship with the user.

Further, in order to get the ranking, it is necessary to count the exercise time/number of times of each user in real time, but in order to reduce the communication pressure of the client, the client can be allowed to obtain the data required for the ranking at regular intervals, or only when the user opens the ranking Get the user's data.

In addition, it should be noted that the methods for obtaining the corresponding relationship between the operation instruction and the fitness action may include the following methods.

Determining the correspondence between preset operation instructions and fitness actions may be: determining the correspondence between user-defined operating instructions and fitness actions. In other words, the correspondence between operation instructions and fitness actions is preset by the user, and determining the correspondence between the preset operation instructions and fitness actions is the acquisition of the correspondence between the user's preset operation instructions and fitness actions.

The user-defined setting may be that the user selects several fitness movements from all the operation instructions and all fitness movements that can be selected, and selects several operation instructions, and sets the corresponding relationship between the operation instructions and the fitness movements.

User-defined settings can also be selected according to content recommended by the client. For the client, the client recommendation can be based on the user's operation instruction usage requirements, or based on the user's exercise needs, or based on the user's friend relationship.

If the recommendation is made based on the user's operation instruction usage requirements or exercise requirements (collectively referred to as usage requirements), the client needs to first recommend to the user the operation instructions, fitness actions, or the corresponding relationship between the operation instructions and the fitness actions that meet the user's usage needs, so that The user prefers what the client recommends. In other words, the method for customizing settings includes: determining the user's usage requirements; the usage requirements include at least one of the following: the execution requirements of operation instructions, and the user's exercise requirements; according to the user's usage requirements, preferentially recommending The operation instruction, exercise action corresponding to the use requirement, or the correspondence between the operation instruction and the exercise action; determining the correspondence between the operation instruction selected by the user and the exercise exercise.

According to the user's use needs, the priority recommendation is performed, that is, the operation instructions, fitness actions or the corresponding relationship between the two that meet the user's use needs are preferentially displayed, so that the user can preferentially choose the recommended ones. In addition, since the correspondence between the operation instructions and the exercise actions is user-defined, the correspondence between the operation instructions and the exercise actions customized by different users may also be different.

If the client makes recommendations based on the user's operating instruction usage requirements, then determining the user's usage requirements includes: determining the common operating instructions that the user executes more frequently than the preset frequency threshold, that is, determining the user's most frequently used operating instructions. According to the user's needs, recommend to the user the operation instructions, fitness actions corresponding to the use needs, or the correspondence between the operation instructions and the fitness actions, including: displaying common operation instructions to the user on the corresponding relationship selection page, that is, in On the corresponding relationship selection page, common operation instructions are in front of other operation instructions.

If the client makes recommendations based on the user's exercise needs, determining the user's use needs includes: determining the user's exercise needs. Among them, the user's exercise needs can be determined according to the questionnaire filled out by the user, or determined according to the user's exercise type selection. The exercise type can include fat loss, plasticity, muscle gain, etc., and can also be selected Areas for fat loss, plasticity and muscle gain. In addition, according to the user's needs, preferentially recommend to the user the corresponding operation instructions, fitness actions, or the correspondence between the operation instructions and the fitness actions, including: determining a preset number of recommended actions according to the user's exercise needs ; The preset number is greater than the number of configurable operation instructions; The recommended actions are preferentially displayed to the user on the corresponding relationship selection page. The recommended actions are in line with the user's exercise needs. For example, if the user wants to increase abdominal muscles, the recommended actions may include planks, sit-ups, and so on.

In addition, in the case where there are standards for fitness actions set by the user, the user's exercise needs may also include exercise levels, which may be selected by the user according to his or her own circumstances, or may be determined based on a questionnaire filled out by the user. . The standard of fitness action can be comprehensively considered according to the user's exercise level, age and gender.

If the client makes comprehensive recommendations based on the user's usage needs, then according to the user's usage needs, it will preferentially recommend to the user the operation instructions, fitness actions corresponding to the usage needs, or the corresponding relationship between the operation instructions and the fitness actions, including: According to the usage requirements of the user, determine a user group that has similar usage requirements to the user; determine the common correspondence of the user group; and preferentially display the common correspondence to the user on the correspondence selection page.

Among them, the user's usage requirements can include the user's exercise requirements and operation instruction usage requirements. The exercise requirements here can be exercise types, such as fat loss, plasticity, or muscle gain. The operation instruction usage requirements can be a certain page or Small program needs. Similar use requirements may refer to the same type of exercise and similar use requirements for operation instructions.

The commonly used corresponding relationship of the user group may refer to a corresponding relationship in which the number of users is greater than a preset threshold among the corresponding relationships between operating instructions and fitness actions that have been customized in the user group. When recommending to the user, it is also possible to mark the recommended corresponding relationship, for example, adding the label of "popularity TPO1", so that the user can preferentially select these corresponding relationships and improve the user experience.

Among them, the commonly used corresponding relationship can be obtained from the corresponding relationship recommended by the user uploaded by the user, or the corresponding relationship can be uploaded to the server after the client determines the corresponding relationship, and the commonly used corresponding relationship can be obtained from the corresponding relationship stored in the server obtained from .

In addition, if the client recommends based on the user's friend relationship, then the client can recommend to the user, among the users who have a friend relationship with the user, the corresponding relationship that uses the most. Specifically, the client needs to first confirm the user's friend relationship, determine the corresponding relationship used by users who have a friend relationship with the user, count the used corresponding relationships, and calculate the number of users who have a friend relationship with the user The most used sets of correspondences are used, and these correspondences are recommended to users first.

Through the above method, the user can customize the corresponding relationship between operation instructions and fitness actions, and combine fitness with daily use of the client, so that the user can complete fitness while using the client. In addition, since the client will consider the user's use needs or the user's friend relationship and make recommendations, the user can give priority to the operation instructions and fitness actions recommended by the client, or the corresponding relationship between the operation instructions and the fitness actions, so that the operation selected by the user The corresponding relationship between instructions and fitness actions is in line with the user's needs for operating instructions or fitness needs, which improves the user experience.

Determining the correspondence between preset operation instructions and fitness actions can be determined not only according to user-defined settings, but also by determining the correspondence between system preset operation instructions and fitness actions. In other words, the client is preset with the corresponding relationship between operation instructions and fitness actions. After the user triggers the operation instructions through fitness actions, the client will obtain the corresponding relationship between the operation instructions and fitness actions written in the client.

The preset correspondence between the operation instruction and the fitness action may be a correspondence that can enhance the interest. Specifically, the realization effect of the fitness action in the correspondence corresponds to the effect after the operation instruction is triggered. For example, for the carbon saving statistics interface of an electronic wallet client, its commonly used functions include watering others, stealing energy, and switching to the next friend user. Then, because watering requires a lot of strength, you can trigger watering through strength-based actions (such as squats, push-ups); because energy grows like fruit at a higher position on the tree, you need to jump up to pick it, you can use Jumping actions (such as raising your legs high) trigger energy stealing. Since you need to slide left and right to switch to the next friend, you can switch by bending to the left or bending to the right. For example, for a news browsing client, you can turn pages down by squatting, and turn pages up by jumping. The content of Wuqinxi. Such a setting increases the interest of users.

Next, a specific embodiment will be used to describe in detail a method for using fitness actions to trigger client operation instructions.

For example, on the interface of an e-wallet client that counts carbon savings (take Ant Forest as an example), operation instructions can be triggered through fitness actions. After the user activates the operation command to trigger the client through the fitness action on the setting page of the client, the client obtains the correspondence between the preset operation command and the fitness action. The specific correspondence may include:

Steal Energy - Knee High,

watering - squats/pushups/situps,

switch next user - side bend,

Enter the joint species that you participate in - one-leg belly lift.

Among them, each action can set fitness action standards. The specific fitness action standards are set according to the user's gender and age. For example, the user is 30 years old and male, and the set fitness action standards are: high leg raise, ten seconds 15 times, sit-ups, 10 in 10 seconds, 10 push-ups in 10 seconds, side bending at an angle greater than 60°, and 15 consecutive abdominal lifts with one leg. In order to improve the user's exercise effect, it is also possible to give the user a certain reward when the fitness action performed by the user exceeds the fitness action standard to a certain extent. The proportion of the user's energy.

After the user turns it on, the client is a smart phone, which prompts the user to put the smart phone upright on the table and the ground through voice, and prompts the user to go far away. The client obtains the captured image through the camera module of the smart phone, and detects the The location of the user in the module, when the location of the user is appropriate, the user will be prompted by voice that the user has arrived at the appropriate location.

After the user arrives at a suitable location, the client obtains the video captured by the camera module, inputs the captured video into the pre-trained feature extraction model, extracts the action features of the video, and inputs the action features into the action recognition model to identify whether there is a preset Fitness actions (the aforementioned high leg lifts, squats, push-ups, sit-ups, side bends, single-leg abdominal lifts, etc.).

In the case of determining that there is a preset fitness action, such as determining that there is a high leg raising action, further determine the frequency of the user performing the high leg raising action, if the user frequency exceeds the set high leg raising standard (15 in ten seconds), Then perform the energy steal corresponding to the leg raising action. After the energy is stolen, the user can be prompted by voice that the energy stealing action has been completed. For example, the user can be prompted "Congratulations, you have obtained the user's A10g energy". Whether there are still energy balls that have not been stolen, etc. For example, the user can be prompted, "There are still 5 energy balls that have not been picked on the current page".

If the action performed by the user is to bend to the left and meets the preset standard of bending to the left, after switching to the interface of the next user, the user can be prompted by voice, "Switched to the page of user B" . If the action performed by the user is single-leg abdominal lifting, and it meets the preset standard of single-leg abdominal lifting, the user can be prompted by voice, "You have entered your hybrid xx tree". After entering the combined species, if the user performs sit-ups and satisfies the standard of sit-ups, the user can be prompted by voice "You have watered xx tree 10g".

Corresponding to the foregoing embodiment of the method for triggering a client operation command by using a fitness action, this specification also provides an embodiment of a device for triggering a client operation command by using a fitness action and a terminal to which it is applied.

As shown in FIG. 3, FIG. 3 is a block diagram of a device for triggering client operation instructions by using fitness actions according to an exemplary embodiment of this specification. The device includes: a correspondence determination module 310, configured to determine a preset The corresponding relationship between the operation instruction and the fitness action; wherein, each operation instruction corresponds to at least one fitness action, and the fitness actions corresponding to any two operation instructions do not overlap.

The exercise posture data acquisition module 320 is configured to acquire the current exercise posture data of the user.

The exercise action recognition module 330 is configured to identify the exercise action corresponding to the acquired exercise posture data.

The operation instruction execution module 340 is configured to determine and execute the operation instruction corresponding to the exercise action according to the preset correspondence between the operation instruction and the exercise exercise.

Wherein, the correspondence determining module 310 is specifically configured to: determine the correspondence between user-defined operating instructions and fitness actions.

In the case that the corresponding relationship is a user-defined setting, the device may also include a custom setting module 300, configured to: determine the user's use requirements; the use requirements include at least one of the following: the execution requirements of operation instructions, the user's exercise Requirements; according to the user's use needs, on the corresponding relationship selection page, recommend to the user the operation instructions, fitness actions corresponding to the use needs, or the corresponding relationship between the operation instructions and the fitness actions; determine the relationship between the operation instructions selected by the user and the fitness actions Correspondence between.

In the custom setting module 300, the determination of the user's usage requirements may include: determining that the user executes common operation instructions whose frequency is greater than a preset frequency threshold; The operation instruction, exercise action, or the correspondence between the operation instruction and the exercise action may include: preferentially displaying common operation instructions to the user on the correspondence selection page.

In addition, in the custom setting module 300, the determination of the user's use needs may also include: determining the user's exercise needs; according to the user's use needs, preferentially recommending to the user the operation instructions and fitness actions corresponding to the use needs Or the correspondence between operation instructions and fitness actions may also include: determining a preset number of recommended actions according to the user's exercise needs; the preset number is greater than the number of configurable operation instructions; Users are given priority to display recommended actions.

In the custom setting module 300, according to the use needs of the user, the operation instructions and fitness actions corresponding to the use needs are preferentially recommended to the user, or the corresponding relationship between the operation instructions and the fitness actions can also include: Determine the user group that has similar usage requirements as the user; determine the common correspondence of the user group; and preferentially display the common correspondence to the user on the correspondence selection page.

Wherein, the motion posture data acquisition module 320 is specifically configured to obtain motion posture data captured by the camera module of the user terminal. The fitness action recognition module 330 is specifically used to obtain a pre-trained feature extraction model and a pre-trained action recognition model; input the user's motion posture data into the feature extraction model, and extract the motion features of the motion posture data; The action features are input into the action recognition model to determine the fitness action corresponding to the user's exercise posture data. Wherein, the action feature may be the position feature of the key points of the human body in each frame of image; the key points of the human body are used to identify the action of the human body.

In addition, the motion posture data acquisition module 320 can also be specifically used to obtain the measurement sequence of the motion sensor of the mobile terminal within a preset time, and use the measurement sequence as motion posture data; the motion sensor is used to detect the motion of the mobile terminal. Mobile sensors. The fitness action recognition module 330 can also be specifically configured to obtain a pre-trained action recognition model; input the measurement sequence into the action recognition model, and determine the action identifier corresponding to the measurement sequence.

For the implementation process of the functions and functions of each module in the above-mentioned device for triggering client operation instructions by using fitness actions, please refer to the implementation process of corresponding steps in the above-mentioned method for triggering client operation instructions by using exercise actions, which will not be repeated here. repeat.

As for the device embodiment, since it basically corresponds to the method embodiment, for related parts, please refer to the part description of the method embodiment. The device embodiments described above are only illustrative, and the modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in One place, or it can be distributed to multiple network modules. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution in this specification. It can be understood and implemented by those skilled in the art without creative effort.

As shown in Figure 4, Figure 4 shows a hardware structure diagram of a computer device where the embodiment=uses a fitness action to trigger a client operation instruction device, the device may include: a processor 1010, a memory 1020, an input/output interface 1030 , a communication interface 1040 and a bus 1050 . The processor 1010 , the memory 1020 , the input/output interface 1030 and the communication interface 1040 are connected to each other within the device through the bus 1050 .

The processor 1010 may be implemented by a general-purpose CPU (Central Processing Unit, central processing unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute related programs to realize the technical solutions provided by the embodiments of this specification.

The memory 1020 can be implemented in the form of ROM (Read Only Memory, read-only memory), RAM (Random Access Memory, random access memory), static storage device, dynamic storage device, and the like. The memory 1020 can store operating systems and other application programs. When implementing the technical solutions provided by the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 1020 and invoked by the processor 1010 for execution.

The input/output interface 1030 is used to connect the input/output module to realize information input and output. The input/output/module can be configured in the device as a component (not shown in the figure), or can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, and the like.

The communication interface 1040 is used to connect a communication module (not shown in the figure), so as to realize the communication interaction between the device and other devices. The communication module can realize communication through wired means (such as USB, network cable, etc.), and can also realize communication through wireless means (such as mobile network, WIFI, Bluetooth, etc.).

Bus 1050 includes a path that carries information between the various components of the device (eg, processor 1010, memory 1020, input/output interface 1030, and communication interface 1040).

It should be noted that although the above device only shows the processor 1010, the memory 1020, the input/output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components. In addition, those skilled in the art can understand that the above-mentioned device may only include components necessary to implement the solutions of the embodiments of this specification, and does not necessarily include all the components shown in the figure.

The embodiment of this specification also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned method for triggering an operation instruction of a client by using a fitness action is implemented.

Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can be implemented by any method or technology for storage of information. Information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic tape cartridge, tape magnetic disk storage or other magnetic storage device or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media excludes transitory computer-readable media, such as modulated data signals and carrier waves.

It should also be noted that the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes Other elements not expressly listed, or elements inherent in the process, method, commodity, or apparatus are also included. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

Claims

A method for using fitness actions to trigger client operation instructions, comprising:

Determine the correspondence between preset operation instructions and fitness actions; wherein, each operation instruction corresponds to at least one fitness action, and the fitness actions corresponding to any two operation instructions do not overlap;

Obtain the user's current motion posture data;

Identify the fitness action corresponding to the acquired motion posture data;

According to the preset corresponding relationship between the operation instruction and the exercise action, the operation instruction corresponding to the exercise action is determined and executed.
According to the method according to claim 1, said determining the corresponding relationship between preset operation instructions and fitness actions comprises:

Determine the correspondence between the user-defined operating instructions and fitness actions.
According to the method described in claim 2, the method for custom setting comprises:

Determine the user's usage requirements; the usage requirements include at least one of the following: execution requirements of operation instructions, user's exercise requirements;

According to the user's use needs, on the corresponding relationship selection page, preferentially recommend to the user the operation instructions, exercise actions corresponding to the use needs, or the correspondence between the operation instructions and the exercise actions;

Determine the correspondence between the operation instruction selected by the user and the exercise action.
According to the method described in claim 3,

The determination of the user's use requirements includes: determining that the user executes common operation instructions whose frequency is greater than a preset frequency threshold;

According to the usage needs of the user, the preferential recommendation to the user of the operation instructions and fitness actions corresponding to the usage needs, or the correspondence between the operation instructions and the fitness actions, includes: preferentially displaying common operation instructions to the user on the corresponding relationship selection page.
According to the method described in claim 3,

The determination of the user's use needs includes: determining the user's exercise needs;

According to the user's use needs, the priority recommendation to the user is the operation instruction, fitness action corresponding to the use demand, or the corresponding relationship between the operation instruction and the exercise action, including: according to the user's exercise demand, determining a preset number of recommended actions ; The preset number is greater than the number of configurable operation instructions; The recommended actions are preferentially displayed to the user on the corresponding relationship selection page.
According to the method of claim 3, according to the use needs of the user, the priority recommendation to the user is the operation instruction, fitness action corresponding to the use demand, or the corresponding relationship between the operation instruction and the exercise action, including:

According to the user's use needs, determine the user group that has similar use needs with the user;

Determine the common correspondence of the user group;

On the corresponding relationship selection page, frequently used corresponding relationships are preferentially displayed to the user.
According to the described method of claim 1,

The acquisition of the motion posture data of the user includes: obtaining the motion posture data captured by the camera module of the user terminal;

The fitness action corresponding to the exercise posture data acquired by the identification includes:

Get pre-trained feature extraction models and pre-trained action recognition models;

Inputting the motion posture data of the user into the feature extraction model, and extracting the action features of the motion posture data;

Input the action feature into the action recognition model to determine the fitness action corresponding to the user's exercise posture data.
According to the method according to claim 7, the action feature is the position feature of the key points of the human body in each frame of image; the key points of the human body are used to identify the action of the human body.
According to the described method of claim 1,

The acquisition of the user's motion posture data includes: obtaining a measurement sequence of the motion sensor of the mobile terminal within a preset time, and using the measurement sequence as motion posture data; the motion sensor is a sensor used to detect the dynamic movement of the mobile terminal ;

The fitness action corresponding to the exercise posture data acquired by the identification includes:

Obtain a pre-trained action recognition model;

Input the measurement sequence into the action recognition model, and determine the action identifier corresponding to the measurement sequence.
A device for triggering client operation instructions by fitness actions, comprising:

A correspondence determination module, configured to determine the correspondence between preset operation instructions and fitness actions; wherein, each operation instruction corresponds to at least one fitness movement, and the fitness movements corresponding to any two operation instructions do not overlap;

The movement posture data acquisition module is used to obtain the user's current movement posture data;

A fitness action recognition module, used to identify the fitness action corresponding to the acquired motion posture data;

The operation instruction execution module is used to determine and execute the operation instruction corresponding to the exercise action according to the preset correspondence between the operation instruction and the exercise exercise.
An electronic device comprising:

processor;

memory for storing processor-executable instructions;

Wherein, the processor implements the method according to any one of claims 1-9 by running the executable instructions.
A computer-readable storage medium storing computer instructions, which implement the method according to any one of claims 1-9 when the computer instructions are executed by a processor.