CN115942105A - Method, device and camera for scheduling and running AI model in camera - Google Patents

Abstract

In the scheduling and operating method of the artificial intelligent AI model in the camera, after the camera receives an AI model rotation instruction sent by electronic equipment, in each rotation period, on a time slice allocated to a first AI process, a first AI model is operated, and image data in a frame corresponding to the current rotation period is processed through the first AI model. Therefore, the camera side can simultaneously operate at least two frame rate insensitive AI models, and data frames processed by the simultaneously operated AI models in the same cycle are consistent, so that the consistency of AI model results can be improved.

Description

Method, device and camera for scheduling and running AI model in camera

technical field

The embodiments of the present application relate to the technical field of smart terminals, and in particular to a method and device for scheduling and running an AI model in a camera, and a camera.

Background technique

Currently, when a large screen with a camera provides camera-related artificial intelligence (AI) capabilities, the following method can be used: a built-in AI model on the side of the large screen, at runtime, according to the needs of the application, through the remote network The socket (socket) interface of the driver interface specification (remote network driver interface specification, RNDIS) sends the AI model to the camera side, and the AI model runs on the camera side. Among them, the large screen and the camera are connected through a physical universal serial bus (USB) transmission line; RNDIS uses the USB protocol as the lower layer transmission, and provides a virtual Ethernet connection to the upper layer.

However, in this implementation, at runtime, according to the needs of the application, after the large screen transmits the AI model to the camera, the camera can only process one AI model at a time, and does not support multiple AI models running at the same time.

Contents of the invention

Embodiments of the present application provide a method and device for scheduling and running an AI model in a camera, and a camera. The embodiments of the present application also provide a computer-readable storage medium, so that at least two frame rate-insensitive AI models can be simultaneously run on the camera side. AI models, and the data frames processed by the AI models running at the same time in the same cycle are consistent, which can improve the consistency of the results of the AI models.

In the first aspect, an embodiment of the present application provides a method for scheduling and running an artificial intelligence AI model in a camera, which is applied to the camera, the camera is connected to an electronic device, and at least two AI models are loaded in the camera. The method includes: receiving an AI model rotation instruction sent by the electronic device; in each rotation cycle, in the time slice allocated to the first AI process, running the first AI model, and using the first AI model to update the current rotation cycle Processing the image data in the corresponding frame; wherein, the first AI model is any AI model in a combination of AI models capable of time slice rotation, and the first AI model runs in the first AI process, The image data is collected by the camera.

In the method for scheduling and running the artificial intelligence AI model in the above camera, after the camera receives the AI model rotation instruction sent by the electronic device, in each rotation cycle, the first AI model is run on the time slice allocated to the first AI process, through The first AI model processes the image data in the frame corresponding to the current rotation period. Therefore, at least two frame rate-insensitive AI models can be run simultaneously on the camera side, and the data frames processed by the AI models running at the same time are consistent in the same rotation cycle, which can improve the consistency of AI model results.

In one possible implementation manner, the time slice allocated to the first AI process is determined according to the time required by the first AI process to process one frame of image data.

In one possible implementation manner, the frame number of the frame corresponding to the current rotation cycle is determined according to the sequence number of the current rotation cycle and the time required for each AI process included in the current rotation cycle to process a frame of image data.

In one possible implementation manner, before receiving the AI model rotation instruction sent by the electronic device, it also includes: after the electronic device is started and the camera is started, receiving at least Two AI models; divide the at least two AI models into a frame rate-sensitive AI model and a frame rate-insensitive AI model according to the frame rate requirements of the AI model and the calculation method of the video frame; from the frame rate In the rate-insensitive AI model, select the AI model combination that can perform time slice rotation.

In one possible implementation, the at least two AI models are divided into a frame rate sensitive AI model and a frame rate insensitive AI model according to the frame rate requirements of the AI model and the calculation method for video frames. The model includes: for any AI model in the at least two AI models, if the frame rate of the video frame required by the AI model is greater than a predetermined frame rate threshold, and the calculation result of the AI model depends on the video frame forward , it is determined that the AI model is a frame rate-sensitive AI model; if the video frame rate required by the AI model is less than or equal to a predetermined frame rate threshold, and the calculation results of the AI model for video frames are independent of each other, then It is determined that the AI model is a frame rate insensitive AI model.

In one possible implementation manner, the selecting an AI model combination capable of time slice rotation from the frame rate-insensitive AI models includes: The required minimum frame rate, and the time required for the AI process to process a frame of data, from the frame rate insensitive AI model, select an AI model combination that can perform time slice rotation.

In the second aspect, the embodiment of the present application provides a device for scheduling and running the artificial intelligence AI model in the camera, the device is included in the camera, and the device has the function of implementing the camera behavior in the first aspect and possible implementations of the first aspect. The functions may be implemented by hardware, or may be implemented by executing corresponding software through hardware. Hardware or software includes one or more modules or units corresponding to the functions described above. For example, a receiving module or unit, a processing module or unit, a sending module or unit, and the like.

In a third aspect, an embodiment of the present application provides a camera, the camera is connected to an electronic device, and the camera includes: one or more processors; memory; multiple application programs; and one or more computer programs, wherein the The one or more computer programs are stored in the memory, the one or more computer programs include instructions that, when executed by the camera, cause the camera to perform the following steps: receiving the electronic device The AI model rotation instruction sent; in each rotation cycle, in the time slice allocated to the first AI process, run the first AI model, and use the first AI model to process the image data in the frame corresponding to the current rotation cycle ; Wherein, the first AI model is any AI model in the AI model combination that can perform time slice rotation, the first AI model runs in the first AI process, and the image data is the camera Collected.

In one possible implementation manner, the time slice allocated to the first AI process is determined according to the time required by the first AI process to process one frame of image data.

In one possible implementation manner, the frame number of the frame corresponding to the current rotation cycle is determined according to the sequence number of the current rotation cycle and the time required for each AI process included in the current rotation cycle to process a frame of image data.

In one possible implementation manner, when the instruction is executed by the camera, before the camera executes the step of receiving the AI model rotation instruction sent by the electronic device, the following steps are further performed: After the electronic device is started and the camera is started, at least two AI models sent by the electronic device are received; according to the frame rate requirement of the AI model and the calculation method for the video frame, the at least two AI models are divided into It is a frame rate sensitive AI model and a frame rate insensitive AI model; from the frame rate insensitive AI models, select an AI model combination that can perform time slice rotation.

In one possible implementation manner, when the instruction is executed by the camera, the camera executes the calculation method of the at least two AI The step of classifying the model into a frame rate sensitive AI model and a frame rate insensitive AI model includes: for any AI model in the at least two AI models, if the video frame rate required by the AI model is greater than a predetermined Frame rate threshold, and the AI model is forward dependent on the calculation result of the video frame, then it is determined that the AI model is a frame rate sensitive AI model; if the video frame rate required by the AI model is less than or equal to the predetermined frame rate threshold, and the calculation results of the AI model for video frames are independent of each other, then it is determined that the AI model is a frame rate insensitive AI model.

In one possible implementation manner, when the instruction is executed by the camera, the camera is made to perform the step of selecting a combination of AI models capable of time slice rotation from the frame rate insensitive AI models. The steps include: according to the minimum frame rate required by the AI process running the frame rate insensitive AI model, and the time required for the AI process to process one frame of data, from the frame rate insensitive AI model, Select an AI model combination that can perform time slice rotation.

It should be understood that the second aspect and the third aspect of the embodiment of the present application are consistent with the technical solution of the first aspect of the embodiment of the present application, and the beneficial effects obtained in each aspect and the corresponding feasible implementation manners are similar, so details are not repeated here.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, where a computer program is stored in the above-mentioned computer-readable storage medium, and when running on a computer, causes the computer to execute the method provided in the first aspect.

In a fifth aspect, an embodiment of the present application provides a computer program, which is used to execute the method provided in the first aspect when the above computer program is executed by a computer.

In a possible design, the programs in the fourth aspect and the fifth aspect may be stored in whole or in part on a storage medium packaged with the processor, or stored in part or in whole in a storage medium not packaged with the processor. on memory.

Description of drawings

FIG. 1 is a schematic structural diagram of a camera provided by an embodiment of the present application;

FIG. 2 is a schematic diagram of an application scenario provided in an embodiment of the present application;

FIG. 3 is a flowchart of a method for scheduling and running an AI model in a camera according to an embodiment of the present application;

4 is a flowchart of a method for scheduling and running an AI model in a camera according to another embodiment of the present application;

FIG. 5 is a flow chart of a method for scheduling and running an AI model in a camera according to yet another embodiment of the present application;

FIG. 6 is a schematic diagram of frames processed by each AI process provided by an embodiment of the present application;

FIG. 7 is a schematic structural diagram of a camera provided by another embodiment of the present application.

Detailed ways

The terms used in the embodiments of the present application are only used to explain specific embodiments of the present application, and are not intended to limit the present application.

Currently, there are three ways to provide camera-related AI capabilities for large screens with cameras:

1) An AI model is built into the camera system, and the AI model runs on the camera side;

However, when the AI model is built into the camera system, on the one hand, due to the large video-related AI model, limited by the small storage capacity of the camera system, the number of built-in AI models is limited, and the result presented externally is a large-screen The AI function of the camera is limited; on the other hand, when the AI model is built into the camera system, the update of the AI model depends on the update of the camera system, and the cost of updating the AI model is relatively high.

2) Built-in AI model on the motherboard side of the large screen, and the AI model runs on the motherboard side.

However, when the AI model is built on the motherboard side, hardware modules such as video encoding, image processing, and video decoding need to be integrated on the motherboard side to decode the video stream sent from the camera, apply the AI model, image processing, and video encoding. high.

3) Built-in AI model on the large screen side, at runtime, according to the needs of the application, send the AI model to the camera side through the socket interface of RNDIS (using the USB protocol as the lower layer transmission, providing a virtual Ethernet connection to the upper layer), The AI model runs on the camera side.

When the built-in AI model is built on the large screen side, when the AI model is transmitted to the camera according to the needs of the application during operation, on the one hand, due to the large size of the AI model, the process of transmitting the AI model to the camera and loading the AI model to the camera takes a long time. time, leading to poor application experience; on the other hand, the AI model is not classified and scheduled on the camera side, and only one AI model can be processed at a time, and multiple AI models are not supported to run at the same time.

On the basis of the camera solution of the dual-drive protocol, in view of the problem that the camera does not schedule the AI model and does not support the simultaneous operation of multiple AI models, the embodiment of this application provides a method for scheduling and running the AI model in the camera. Based on the camera solution of the protocol, the camera can support multiple AI models to run simultaneously:

1. According to the frame rate requirement of the AI model and the calculation method of the video frame, the AI model is divided into a frame rate sensitive AI model and a frame rate insensitive AI model.

2. For the frame rate insensitive AI model, judge whether it can perform multi-AI rotation through the given constraints, and calculate its rotation period.

The method for scheduling and running the AI model in the camera provided in the embodiment of the present application can be applied to the camera, and the above-mentioned camera can include: one or more processors; memory; multiple application programs; and one or more computer programs, wherein one or more One or more computer programs are stored in the memory, and one or more computer programs include instructions. When the instructions are executed by the camera, the above-mentioned camera can be made to execute the method for scheduling and running the AI model in the camera provided by the embodiment of the present application.

Exemplarily, FIG. 1 is a schematic structural diagram of a camera provided by an embodiment of the present application. As shown in FIG. 1 , the camera 100 may include a processor 110 and a memory 120 .

It can be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the camera 100 . In some other embodiments of the present application, the camera 100 may include more or fewer components than shown in the figure, or combine certain components, or separate certain components, or arrange different components. The illustrated components can be realized in hardware, software or a combination of software and hardware.

The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor ( image signal processor (ISP), controller, video codec, digital signal processor (digital signal processor, DSP), and/or neural network processor (neural-network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

The memory 120 may include computer system readable media in the form of volatile memory, such as random access memory (random access memory, RAM) and/or cache memory. The memory 120 may include at least one program product, and the program product has a set of (for example, at least one) program modules configured to execute the method provided by the embodiment of the present application.

A program/utility having a set (at least one) of program modules may be stored in memory 120, such program modules including - but not limited to - an operating system, one or more application programs, other program modules, and program data , each or some combination of these examples may include implementations of network environments. The program modules generally perform the functions and/or methods in the embodiments described herein.

The processor 110 executes various functional applications and data processing by running the programs stored in the memory 120 , for example, implementing the scheduling and running method of the artificial intelligence AI model in the camera provided by the embodiments shown in this application.

Further, the camera 100 may further include a lens 130 , a photosensitive element 140 and a communication interface 150 .

The object generates an optical image through the lens 130 and projects it to the photosensitive element 140 . The photosensitive element 140 may be a charge coupled device (charge coupled device, CCD) or a complementary metal-oxide-semiconductor (complementary metal-oxide-semiconductor, CMOS) phototransistor. The photosensitive element 140 converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. DSP converts digital image signals into standard RGB, YUV and other image signals.

The communication interface 150 is used for interaction between the camera 100 and electronic equipment.

The scheduling and running method of the artificial intelligence AI model in the camera provided by the embodiment of the present application can be applied in the solution of dual-protocol communication between the camera and the electronic device (for example: large screen). As shown in FIG. 2 , the camera 100 and the electronic device 22 communicate through the USB video specification (USB video-class, UVC)+RNDIS, and the camera proxy (cameraproxy) is an agent program that runs on the electronic device and communicates with the camera. Fig. 2 is a schematic diagram of an application scenario provided in an embodiment of the present application.

In FIG. 2 , the camera 100 is connected to the electronic device 22 through a USB cable, which supports USB1.0/2.0/3.0.

Wherein, the above-mentioned electronic devices may be large screens, smart phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (augmented reality, AR)/virtual reality (virtual reality, VR) equipment, notebook computers, super mobile personal computers ( ultra-mobile personal computer, UMPC), netbook or personal digital assistant (personal digital assistant, PDA); the embodiment of the present application does not make any limitation on the specific type of electronic equipment. The following embodiments of the present application take a large screen as an example for illustration.

For ease of understanding, the following embodiments of the present application will use the camera with the structure shown in FIG. 1 in combination with the application scenario shown in FIG. 2 to specifically describe the method for scheduling and running the AI model in the camera provided by the embodiments of the present application.

FIG. 3 is a flow chart of a method for scheduling and running an AI model in a camera provided by an embodiment of the present application, which is applied to the camera 100 and the camera 100 is connected to the electronic device 22 . In this embodiment, at least two AI models are loaded in the camera 100, as shown in FIG. 3 , the method for scheduling and running the artificial intelligence AI model in the camera may include:

Step 301 , receiving an AI model rotation instruction sent by the electronic device 22 .

Step 302, in each rotation period, in the time slice allocated to the first AI process, run the first AI model, and use the first AI model to process the image data in the frame corresponding to the current rotation period; wherein, the above-mentioned first The AI model is any AI model in the combination of AI models capable of time slice rotation, the first AI model runs in the first AI process, and the above image data is collected by the camera 100 .

Wherein, the time slice allocated to the first AI process may be determined according to the time required by the first AI process to process one frame of image data.

The frame number of the frame corresponding to the current rotation period is determined according to the sequence number of the current rotation period and the time required for each AI process included in the current rotation period to process a frame of image data. It can be seen from this that within a rotation period, the frame numbers of the frames processed by each AI model are the same, so that the data frames processed by the AI models running at the same time are consistent in the same rotation period, which can improve the accuracy of the AI model results. consistency.

In the method for scheduling and running the artificial intelligence AI model in the above-mentioned camera, after the camera 100 receives the AI model rotation command sent by the electronic device 22, in each rotation cycle, it runs the first AI model on the time slice allocated to the first AI process. , using the first AI model to process the image data in the frame corresponding to the current rotation period. Therefore, at least two frame rate-insensitive AI models can be run simultaneously on the camera side, and the data frames processed by the AI models running at the same time are consistent in the same rotation cycle, which can improve the consistency of AI model results.

Fig. 4 is a flowchart of a method for scheduling and running an AI model in a camera according to another embodiment of the present application. As shown in Fig. 4, in the embodiment shown in Fig. 3 of the present application, before step 301, it may also include:

Step 401 , after the electronic device 22 is started and the camera 100 is started, the camera 100 receives at least two AI models sent by the electronic device 22 .

In step 402, the camera 100 classifies at least two AI models into a frame rate sensitive AI model and a frame rate insensitive AI model according to the frame rate requirement of the AI model and the calculation method of the video frame.

Specifically, the camera 100 divides at least two AI models into a frame rate sensitive AI model and a frame rate insensitive AI model according to the frame rate requirement of the AI model and the calculation method of the video frame. For any AI model in the AI model, if the frame rate of the video frame required by the AI model is greater than the predetermined frame rate threshold, and the calculation result of the AI model is forward dependent on the video frame, it is determined that the above AI model is a frame rate sensitive AI model; if the frame rate of the video frame required by the AI model is less than or equal to the predetermined frame rate threshold, and the calculation results of the AI model on the video frames are independent of each other, then the above AI model is determined to be a frame rate insensitive AI model.

Wherein, the predetermined frame rate threshold above can be set by itself according to system performance and/or implementation requirements during specific implementation. This embodiment does not limit the size of the predetermined frame rate threshold above. For example, the above predetermined frame rate The rate threshold may be 15 frames per second.

In step 403, the camera 100 selects a combination of AI models capable of time slice rotation from the aforementioned frame rate insensitive AI models.

Specifically, the camera 100 selects the combination of AI models that can perform time slice rotation from the aforementioned frame rate insensitive AI models, which may be: according to the minimum frame rate required by the AI process for running the aforementioned frame rate insensitive AI model, and The time required for the above-mentioned AI process to process one frame of data. From the above-mentioned frame rate-insensitive AI models, select an AI model combination that can perform time slice rotation.

Taking the electronic device as a large screen as an example, the method for scheduling and running the AI model in the camera provided by the embodiment of the present application will be described below. FIG. 5 is a flow chart of the method for scheduling and running the AI model in the camera provided in another embodiment of the present application. As shown in Figure 5, the scheduling and running method of the AI model in the camera may include:

Step 501 , after the large screen is started, and after the camera 100 is started, the large screen sends all AI models to the camera 100 through the socket interface of RNDIS.

Step 502 , the camera 100 groups the AI models, and divides the AI models into two types: frame rate sensitive AI and frame rate insensitive AI according to the frame rate requirement of the AI model and the calculation method of the video frame.

Specifically, frame rate-sensitive AI model: this type of AI model has high real-time requirements for video frames (for example: greater than 15 frames per second), and the calculation results of video frames are forward-dependent;

Frame rate insensitive AI model: This type of AI model has low real-time requirements for video frames (for example: less than or equal to 15 frames per second), and the calculation results of video frames are independent of each other.

Step 503, for the frame rate insensitive AI model, calculate the combination of AI models that can perform time slice rotation.

Specifically, the following two conditions can be used to select the AI model combination that can perform time slice rotation from the frame rate insensitive AI model:

1) FPS _AI is used to represent the AI frame rate of the AI model combination for time slice rotation, and the lowest frame rate of the i-th AI process AI _i is FPS _AIi , then FPS _AI = max(FPS _AI1 , FPS _AI2 ,..., FPS _AIn ).

2) Assuming that n AI processes can be loaded at the same time, the time required for the i-th AI process AI _i to process one frame of data is: time _AIi , then these n processes need to satisfy formula (1).

Step 504 , during operation, after the camera 100 receives the AI model rotation instruction sent from the large screen side, time slice rotation is performed on each AI model in the AI model combination capable of time slice rotation.

In one rotation, the time slice allocated by the i-th AI process AI _i is:

In the i-th rotation cycle, the frames processed by each AI process are the same, and the frame number can be determined according to formula (2).

Fig. 6 is a schematic diagram of frames processed by each AI process provided by an embodiment of the present application. Referring to Fig. 6, the frame processed by processes AI1, AI2 and AI3 in the first rotation cycle is frame 1, and in the second rotation cycle The processed frame is frame 6.

上，运行AI_i模型，通过AI_i模型对当前轮转周期对应帧中的图像数据进行处理。而第i个轮转周期对应的帧的帧序号可以按照式(2)确定。In specific implementation, the time slice rotation for each AI model in the AI model combination that can perform time slice rotation can be: in the time slice allocated to the i-th AI process

, run the AI _i model, and process the image data in the frame corresponding to the current rotation cycle through the AI _i model. The frame sequence number of the frame corresponding to the i-th rotation period can be determined according to formula (2).

It should be noted that the AI model rotation instruction received by the camera 100 from the side of the large screen may be: a function switch for AI model rotation is set on the large screen, and the user puts the function switch in an on state, and then, when the camera 100 is running, , the camera 100 receives the AI model rotation instruction sent by the large screen side.

In the scheduling operation method of the AI model in the camera provided by the embodiment of the present application, the AI model is divided into a frame rate sensitive AI model and a frame rate insensitive AI model, and for the frame rate insensitive AI model, it is time sliced Round-robin scheduling, the camera 100 supports multiple AI models running at the same time. In addition, in this embodiment of the application, after the electronic device is turned on and the camera 100 is turned on, the electronic device sends the AI model to the camera 100, and the camera 100 loads the AI model after receiving the AI model, instead of sending it when the application calls , Load the AI model, avoiding the effect delay caused by the need to load the AI model when the application starts, and improving the application experience.

It can be understood that some or all of the steps or operations in the foregoing embodiments are only examples, and other operations or modifications of various operations may also be performed in the embodiment of the present application. In addition, various steps may be performed in different orders presented in the above embodiments, and it may not be necessary to perform all operations in the above embodiments.

It can be understood that, in order to realize the above functions, the camera includes hardware and/or software modules corresponding to each function. In combination with the algorithm steps of the examples described in the embodiments disclosed in the present application, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or computer software drives hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions in combination with the embodiments for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

In this embodiment, the functional modules of the camera may be divided according to the above method embodiments. For example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one module. The above integrated modules may be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic, and is only a logical function division, and there may be other division methods in actual implementation.

FIG. 7 is a schematic structural diagram of a camera provided in another embodiment of the present application. In the case of dividing each functional module corresponding to each function, FIG. 7 shows a possible composition diagram of the camera 700 involved in the above-mentioned embodiment. As shown in FIG. 7, the camera 700 may include: a receiving unit 701, a processing unit 702, and a sending unit 703;

Wherein, the receiving unit 701 can be used to support the camera 700 to perform step 301, step 401 and step 504, etc., and/or other processes for the technical solution described in the embodiment of this application;

The processing unit 702 may be used to support the camera 700 to execute step 302, steps 402-403, and steps 502-504, etc., and/or other processes for the technical solutions described in the embodiments of this application.

It should be noted that all relevant content of the steps involved in the above method embodiments can be referred to the function description of the corresponding function module, and will not be repeated here.

The camera 700 provided in this embodiment is used to implement the above method for scheduling and running the AI model in the camera, so the same effect as the above method can be achieved.

It should be understood that the camera 700 may correspond to the camera 100 shown in FIG. 1 . Wherein, the functions of the receiving unit 701 and the sending unit 703 can be realized by the communication interface 150 in the camera 100 shown in FIG. 1 ; the functions of the processing unit 702 can be realized by the processor 110 in the camera 100 shown in FIG. 1 .

In the case of an integrated unit, the camera 700 may include a processing module, a storage module and a communication module.

Wherein, the processing module can be used to control and manage the actions of the camera 700 , for example, can be used to support the camera 700 to execute the above steps performed by the receiving unit 701 , the processing unit 702 and the sending unit 703 . The storage module can be used to support the camera 700 to store program codes and data. The communication module can be used to support the communication between the camera 700 and other devices.

Wherein, the processing module may be a processor or a controller, which may realize or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present application. The processor can also be a combination of computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (digital signal processing, DSP) and a microprocessor, and the like. The storage module may be a memory. Specifically, the communication module may be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip, and/or a Wi-Fi chip.

In an embodiment, when the processing module is a processor and the storage module is a memory, the camera 700 involved in this embodiment may be a device with the structure shown in FIG. 1 .

The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When it is run on a computer, the computer executes the program provided by the embodiments shown in FIGS. 3 to 5 of the present application. method.

The embodiment of the present application also provides a computer program product, the computer program product includes a computer program, and when it is run on a computer, it causes the computer to execute the method provided in the embodiments shown in FIG. 3 to FIG. 5 of the present application.

In the embodiments of the present application, "at least one" means one or more, and "multiple" means two or more. "And/or" describes the association relationship of associated objects, indicating that there may be three kinds of relationships, for example, A and/or B may indicate that A exists alone, A and B exist simultaneously, or B exists alone. Among them, A and B can be singular or plural. The character "/" generally indicates that the contextual objects are an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c or a and b and c, where a, b, c can be single, or Can be multiple.

Those of ordinary skill in the art can appreciate that each unit and algorithm steps described in the embodiments disclosed herein can be realized by a combination of electronic hardware, computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In several embodiments provided in this application, if any function is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disc and other media that can store program codes. .

The foregoing is only a specific implementation of the present application. Any person skilled in the art within the technical scope disclosed in the present application can easily think of changes or substitutions, which should be covered by the protection scope of the present application. The protection scope of the present application shall be based on the protection scope of the claims.

Claims (14)

1. A scheduling operation method of an artificial intelligent AI model in a camera is applied to the camera, the camera is connected with an electronic device, the method is characterized in that at least two AI models are loaded in the camera, and the method comprises the following steps:

receiving an AI model rotation instruction sent by the electronic equipment;

in each rotation period, operating a first AI model on a time slice allocated to a first AI process, and processing image data in a corresponding frame of the current rotation period through the first AI model; the first AI model is any one of AI model combinations capable of time slice rotation, the first AI model runs in the first AI process, and the image data is acquired by the camera.

2. The method of claim 1, wherein the time slice allocated to the first AI process is determined based on the time required for the first AI process to process one frame of image data.

3. The method according to claim 1, wherein the frame number of the frame corresponding to the current round-robin period is determined based on the number of the current round-robin period and a time required for each AI process included in the current round-robin period to process one frame of image data.

4. The method according to claim 1, wherein before receiving the AI model rotation command sent by the electronic device, the method further comprises:

after the electronic equipment is started and the camera is started, receiving at least two AI models sent by the electronic equipment;

dividing the at least two AI models into a frame rate sensitive AI model and a frame rate insensitive AI model according to the frame rate requirements of the AI models and the calculation mode of the video frames;

and selecting an AI model combination capable of time slice rotation from the frame rate insensitive AI models.

5. The method of claim 4, wherein the classifying the at least two AI models into frame rate sensitive AI models and frame rate insensitive AI models according to AI model frame rate requirements and video frame computation comprises:

for any one of the at least two AI models, if the video frame rate required by the AI model is greater than a predetermined frame rate threshold and the AI model depends on the calculation result of the video frame forward, determining that the AI model is a frame rate sensitive AI model;

and if the video frame rate required by the AI model is less than or equal to a preset frame rate threshold value and the calculation results of the AI model on the video frames are mutually independent, determining that the AI model is a frame rate insensitive AI model.

6. The method of claim 4, wherein selecting the time slice round robin capable AI model combination from the frame rate insensitive AI models comprises:

and selecting an AI model combination capable of performing time slice rotation from the frame rate insensitive AI model according to the lowest frame rate required by the AI process for operating the frame rate insensitive AI model and the time required by the AI process for processing one frame of data.

7. An apparatus for scheduling an artificial intelligence AI model in a camera, characterized by being configured to perform the method of any one of claims 1 to 6.

8. A camera, wherein the camera is connected to an electronic device, the camera comprising:

one or more processors; a memory; a plurality of application programs; and one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs comprising instructions that, when executed by the camera, cause the camera to perform the steps of:

receiving an AI model rotation instruction sent by the electronic equipment;

in each cycle, running a first AI model on a time slice allocated to a first AI process, and processing image data in a frame corresponding to the current cycle through the first AI model; the first AI model is any one of AI model combinations capable of time slice rotation, the first AI model runs in the first AI process, and the image data is acquired by the camera.

9. The camera of claim 8, wherein the time slice allocated to the first AI process is determined according to the time required for the first AI process to process one frame of image data.

10. The camera according to claim 8, wherein the frame number of the frame corresponding to the current round-robin period is determined according to the number of the current round-robin period and a time required for each AI process included in the current round-robin period to process one frame of image data.

11. The camera of claim 8, wherein the instructions, when executed by the camera, cause the camera to perform the following steps before the step of receiving AI model rotation instructions sent by the electronic device:

after the electronic equipment is started and the camera is started, receiving at least two AI models sent by the electronic equipment;

dividing the at least two AI models into a frame rate sensitive AI model and a frame rate insensitive AI model according to the requirements of the AI models on the frame rate and the calculation mode of the AI models on the video frames;

and selecting an AI model combination capable of time slice rotation from the frame rate insensitive AI models.

12. The camera of claim 11, wherein the instructions, when executed by the camera, cause the camera to perform the step of separating the at least two AI models into a frame rate sensitive AI model and a frame rate insensitive AI model according to AI model frame rate requirements and video frame calculation methods, comprises:

for any AI model in the at least two AI models, if the video frame rate required by the AI model is greater than a predetermined frame rate threshold value and the AI model depends on the calculation result of the video frame forward, determining that the AI model is a frame rate sensitive AI model;

and if the video frame rate required by the AI model is less than or equal to a preset frame rate threshold value and the calculation results of the AI model on the video frames are mutually independent, determining that the AI model is a frame rate insensitive AI model.

13. The camera of claim 11, wherein the instructions, when executed by the camera, cause the camera to perform the selecting a time slice round robin combination of AI models from the frame rate insensitive AI models comprises:

and selecting an AI model combination capable of performing time slice rotation from the frame rate insensitive AI model according to the lowest frame rate required by the AI process for operating the frame rate insensitive AI model and the time required by the AI process for processing one frame of data.

14. A computer-readable storage medium, in which a computer program is stored which, when run on a computer, causes the computer to carry out the method according to any one of claims 1 to 6.

Images