WO2021174389A1

WO2021174389A1 - Video processing method and apparatus

Info

Publication number: WO2021174389A1
Application number: PCT/CN2020/077469
Authority: WO
Inventors: 肖旭; 侯弘毅; 陈策
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2020-03-02
Filing date: 2020-03-02
Publication date: 2021-09-10
Also published as: CN112544070A

Abstract

Provided are a video processing method and apparatus. The method comprises: acquiring a video stream; and performing special effects processing on each frame of image comprising a target object in the video stream, such that the target object is surrounded by a special effect, wherein the three-dimensional degree of the special effect is determined according to the attitude of a photographic apparatus when collecting the image currently superimposed with the special effect. According to the present application, a special effect is superimposed onto a real target object in an image according to position information and contour information of the target object in the image, or the attitude of a photographic apparatus when collecting an image currently superimposed with the special effect, thereby facilitating interaction between the special effect and the target object, bringing about a better special effect display, and enriching the understanding of a video stream, and thus improving the viewing effect and understanding of video stream presentation. Moreover, the combination of virtuality and reality can simplify the audience's understanding of a video stream, thereby improving the visual perception of the audience.

Description

Video processing method and device

Technical field

This application relates to the field of image processing, and in particular to a video processing method and device.

Background technique

The adversarial game time is short, the game rules are complex, the game process is intense and exciting, and there are many status information updates. If the data information is used to present the status information to the audience, the audience cannot adapt and deal with so much in a short time. It’s easy for viewers to get tired when watching game videos, especially live game videos, and easily overlook important information. With the rapid development of online games and mobile games, people of different ages have some entry-level special effects understanding of the game. For example, the health bar special effect of each target object represents the current life value information of the target object, if it appears Drop blood or increase experience can display the blood deduction value or gold coin bonus value above the target object. Therefore, transforming the status information into the presentation method of visual special effects can improve the audience's understanding of the game situation and viewing experience.

Generally, the content of the live broadcast screen changes frequently, and the shooting position changes greatly. If the live broadcast screen is rendered with AR special effects, a fixed angle or area will be selected, and it is impossible to perform high-compatibility real-time AR special effects rendering for the entire scene; moreover, the rendered special effects The type is also relatively simple, and special effects can only be rendered in a relatively open location, but virtual special effects cannot interact with real target objects in the scene.

Summary of the invention

This application provides a video processing method and device.

Specifically, this application is implemented through the following technical solutions:

In the first aspect, an embodiment of the present application provides a video processing method, and the method includes:

Get the video stream;

Processing each frame of the image including the target object in the video stream to obtain position information and contour information of the target object in the image;

According to the position information and the contour information, superimpose special effects on the target object in the image.

In a second aspect, an embodiment of the present application provides a video processing device, and the device includes:

Storage device for storing program instructions; and

One or more processors call program instructions stored in the storage device, and when the program instructions are executed, the one or more processors are individually or collectively configured to implement the following operations:

Get the video stream;

In a third aspect, an embodiment of the present application provides a video processing method, and the method includes:

Get the video stream;

Performing special effect processing on each frame of image including the target object in the video stream, so that the target object is surrounded by special effects;

Wherein, the three-dimensional degree of the special effect is determined according to the posture of the photographing device when the image on which the special effect is currently superimposed is collected.

In a fourth aspect, an embodiment of the present application provides a video processing device, and the device includes:

Storage device for storing program instructions; and

Get the video stream;

According to the technical solution provided by the embodiments of the present application, the present application superimposes the real target object in the image according to the position information and contour information of the target object in the image, or the posture of the camera when the image is currently superimposed with the special effect. Facilitate the interaction between special effects and target objects, bringing better special effects display, enriching the understanding of the video stream, thereby increasing the viewing and understanding of the video stream presentation, and this combination of virtual and real can simplify the audience The understanding of the video stream improves the visual experience of the audience.

Description of the drawings

In order to more clearly describe the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings that need to be used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained from these drawings without creative labor.

FIG. 1 is a schematic diagram of an application scenario of a video processing method in an embodiment of the present application;

FIG. 2 is a schematic diagram of a method flow of a video processing method in an embodiment of the present application;

FIG. 3 is a schematic flowchart of an implementation manner of superimposing a target object in an image according to position information and contour information in an embodiment of the present application;

4 is a schematic diagram of a process of superimposing a stereoscopic special effect on a target object in a foreground image in an embodiment of the present application;

FIG. 5 is a schematic diagram of a method flow of a video processing method in another embodiment of the present application;

Fig. 6 is a structural block diagram of a video processing device in an embodiment of the present application.

Detailed ways

Most of the existing AR live broadcast technologies are still based on the screen level, that is, a variety of simple special effects appear on the screen. This kind of special effects does not require an understanding of the scene content, but only needs to set the presentation position of the special effect on the screen and the special effect playback The animation is fine. Portrait-level AR special effects rely on AR recognition and tracking algorithms, such as facial feature recognition, superimposed virtual props and stable tracking. In addition, there are similar AR effects triggered by gesture recognition, and AR effects triggered by facial gesture recognition. However, this type of special effects method is suitable for small videos and shooting with people as the mainstay. The accuracy of face detection is not high. If it is not accurate the first time, it can be shot again. The user's error tolerance rate It is also relatively high. Some TV programs use scene-level AR special effects. The shooting device for shooting TV shows is usually at a fixed angle. Fusion effects such as AR model special effects are not easy to wear, and the positions of models and special effects can be registered and tested in advance. The AR live broadcast is mainly designed for the rendering of background scenes, and currently there is no rendering of independent target objects.

According to the position information and contour information of the target object in the image, or the posture of the shooting device when the image is currently superimposed with the special effect, the application superimposes the real target object in the image with special effect, which is beneficial to the interaction between the special effect and the target object. Brings a better display of special effects, enriches the understanding of the video stream, thereby increasing the viewing and comprehensibility of the video stream presentation, and this combination of virtual and real can simplify the audience’s understanding of the video stream and improve the audience’s vision Feel.

The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of this application.

It should be noted that, in the case of no conflict, the following embodiments and features in the implementation can be combined with each other.

The video processing method of the embodiment of this application can be applied to games in real scenes, such as games with real competitive objects, real competitive players (controllable competitive objects), or games without competitive objects, but including real competitive players. Competition (such as boxing). The competitive game can also be a virtual competitive game (such as MOBA), the competitive object is a virtual competitive object, and the competitive player is a real competitive player. The target object may include various moving objects on the playing field, such as competitive objects (such as robots, drones, mobile cars, etc.) and/or competitive players; the target object may also include stationary objects on the playing field, such as those on the playing field. Prop equipment (such as a base). It should be understood that the video processing method of the embodiment of the present application can also be applied to other types of competitions. In addition, the video processing method of the embodiment of the present application can also be applied to real-time video recording, programs, etc. in a non-competitive state.

Taking a video processing method applied to a robot competition in a real scene as an example, please refer to Fig. 1. The robot competition system may include a robot, a competition server, and a shooting device. Among them, there may be multiple robots, and each robot communicates with a game server, and the game server can obtain real-time status information (such as blood volume, attack power, defense power, skills, or others) of each robot and data from sensors on each robot. The photographing device may include one or more. The photographing device in this embodiment includes a plurality of photographing devices, which are respectively connected to the game server in communication, and the photographing device can shoot from different angles to obtain game images from different angles. The game server may include one or more. For example, the number of game servers is two. The game server includes a game server and an AR (Augmented Reality) server. The game server is used to communicate with robots on the game field. , Used to obtain the real-time status information of the robot, the data information of the sensors on the robot and other competition information; the AR server communicates with the shooting device, and is used to control the shooting device and obtain the data information of the shooting device, such as the video stream collected by the shooting device, and Process the video stream based on the game information, etc. Exemplarily, the number of game servers is one. The game server includes an AR server. The game server is independent of the AR server and communicates with the AR server. That is, the AR server obtains the game information through the game server and the video stream through the camera. Process and output the video stream according to the game information.

Optionally, the robot competition system may also include a referee system. The referee system communicates with the competition server and each robot respectively. The referee system obtains real-time status information of each robot and data from sensors on each robot and other competition information, and transmits it to the competition. server.

The video processing method of the embodiment of the present application can be applied to online processing of videos, and can also be applied to offline processing of videos.

Figure 2 is a schematic diagram of the method flow of the video processing method in an embodiment of the present application; the execution subject of the video processing method in the embodiment of the present application may be a game server or other devices capable of data processing, such as a personal terminal . Referring to FIG. 2, the video processing method of the embodiment of the present invention may include S201 to S203.

Among them, in S201, a video stream is obtained.

In the embodiments of this application, the video stream may be a live video stream or a non-live video stream; the video stream may be a video stream of a competitive game or a video stream of a non-competitive game.

The video stream in the embodiment of the present application may be collected by a camera, and the camera can be mounted on a pan-tilt or on a movable platform such as an unmanned aerial vehicle.

In S202, each frame of image including the target object in the video stream is processed to obtain position information and contour information of the target object in the image.

In some embodiments, the video stream is a video stream of a competitive game, and the target object includes one or more of a moving object and a stationary object. Optionally, the target object is a moving object, such as a robot, a drone, a mobile car, etc.; optionally, the target object is a stationary object, such as a prop device on the playing field; optionally, the target object includes a moving object and a stationary object object.

In some embodiments, the implementation process of S202 may include: processing the video stream based on the instance segmentation algorithm to obtain the position information and contour information of the target object in each image of the video stream. Exemplarily, an instance segmentation algorithm based on deep learning is used to process each frame image of the video stream including the target object to obtain the position information and area mask map (ie mask image) of the target object in each image of the video stream. ), that is, the area mask map is used to characterize the contour information of the target object in each image of the video stream. After being processed by the instance segmentation algorithm, each target object on the image is classified at the pixel level. There is a clear segmentation boundary between all target objects, and the mask image can effectively solve the occlusion problem between multiple target objects. The target object in the image can be divided by a rectangular frame, which pixels in the rectangular frame belong to the target object, and which target object the pixels in the overlapping area between multiple target objects belong to will also be clearly divided. Examples The results of the segmentation algorithm can determine the position information and edge information of the target object on the image, especially for the target objects with occlusion and overlap.

It should be understood that traditional vision segmentation algorithms, or a combination of model matching and edge extraction algorithms can also be used to process the video stream to obtain the position information and contour information of the target object in each image of the video stream.

In S203, special effects are superimposed on the target object in the image according to the position information and the contour information.

Based on the position information and contour information of the target object in the image, it provides accurate occlusion relationship and pixel position for the special effect, which is beneficial to reflect the occlusion relationship between the special effect and the target object, and at the same time, allows the special effect to be displayed on the real target object It is more wrapped, and the positioning of special effects is more accurate. In addition, the mask map of the target object obtained by the result of the instance segmentation algorithm can superimpose special effects with occlusion relationship on the target object in the image, which is conducive to the design of richer and more appropriate special effects for the target object.

The special effects in the embodiments of the present application may be three-dimensional special effects, such as three-dimensional special effects. The three-dimensional effect of the three-dimensional special effect lies in the occlusion effect. The special effect will be blocked by specific pixels in the image during the movement to form a three-dimensional effect; of course, the special effect may also be two Dimensional special effects. For multiple target objects that are occluded and overlapped, the use of two-dimensional special effects may display the special effects of one target object in the area of other target objects, and the use of three-dimensional special effects can more accurately position the target objects and distinguish different degrees of occlusion Therefore, the special effects of each target object are only displayed in their respective areas, which effectively prevents the special effects from being superimposed on other target objects and causing misunderstandings in viewing. In addition, the special effect can also be a frame animation special effect, which is a special effect displayed by sequentially playing a group of pictures at a specific position in the image.

The virtual engine for superimposing special effects in the embodiments of this application can be the Unity rendering engine, or Unreal (a kind of rendering software), Blender (an open source 3D creation suite) or 3ds Max (used for designing visualization, games, and animations). 3D modeling and rendering software) and other rendering engines.

In addition, one or more special effects can be superimposed on each target object in the image to show different content to the audience.

Fig. 3 is a schematic flow diagram of an implementation manner of superimposing a target object in an image according to position information and contour information in an embodiment of the present application; please refer to Fig. 3, according to the position information and contour information, in the image The implementation manner of superimposing special effects on the target object may include steps S301 to S303.

Among them, in S301, the target object in the image and the background are segmented according to the position information and the contour information to obtain a foreground image;

The position information and contour information in S301 respectively refer to the position information of the target object in the image and the contour information of the target object in the image.

The foreground image of this embodiment only contains the target object, and the remaining pixels are transparent. In addition, according to the position information and contour information, the target object and the background in the image are segmented, and a background image can also be obtained.

In S302, special effects are superimposed on the target object in the foreground image according to the position information and the contour information.

The position information and contour information in S302 respectively refer to the position information of the target object in the foreground image and the contour information of the target object in the foreground image. In the embodiment of the present application, the size of the foreground image is equal to the size of the image, the position of the target object in the foreground image is the same as the position of the target object in the image, and the contour information of the target object in the foreground image is the same as that of the target object. The contour information in the image is also the same.

A realization process of superimposing special effects on the target object in the foreground image according to the position information and the contour information may include:

Step 1. Determine the position of the special effect superimposed on the foreground image according to the position information;

The relationship between the position of the special effect superimposed in the foreground image and the position of the target object superimposed with the special effect in the foreground image can be set according to needs, so that the package between the special effect and the target object superimposed on the special effect is more suitable, and the special effect is positioned Is more accurate. Optionally, the position of the special effect superimposed in the foreground image is the same as the position of the target object superimposed on the special effect in the foreground image; optionally, the position of the special effect superimposed in the foreground image is the same as the target object superimposed on the special effect The deviation between the positions in the foreground image is less than the preset deviation threshold, and the size of the preset deviation threshold can be set as required, for example, the preset deviation threshold can be set to 1 mm or others.

Step 2. Determine the size of the special effect superimposed on the foreground image according to the contour information;

When implementing step 2, optionally, according to the contour information of the target object in the foreground image, determine the size of the target object in the foreground image; according to the size of the target object in the foreground image, determine the special effect superimposed on the foreground image Size.

The relationship between the size of the special effect superimposed in the foreground image and the size of the target object superimposed with the special effect in the foreground image can also be set as needed, so that the package between the special effect and the target object superimposed with the special effect is more suitable Optionally, the size of the special effect superimposed in the foreground image is equal to the size of the target object superimposed on the special effect in the foreground image; optionally, the size of the special effect superimposed in the foreground image is larger than the target object superimposed on the special effect The size in the foreground image, and the ratio between the size of the special effect superimposed in the foreground image and the size of the target object superimposed on the special effect in the foreground image (that is, the size of the special effect superimposed in the foreground image/superimposed the size of the The size of the target object of the special effect in the foreground image) is less than the preset ratio threshold. The preset ratio threshold can be set as needed. For example, the preset ratio threshold can be set to 1.2 or others.

It should be noted that the embodiment of the present application does not specifically limit the execution sequence of step 1 and step 2. For example, in some embodiments, step 1 and step 2 can be executed in a sequential order, and step 1 can be executed first, and then step can be executed. 2; Or, perform step 2 first, and then perform step 1. In other embodiments, step 1 and step 2 can be performed simultaneously.

Step 3. According to the position of the special effect superimposed in the foreground image and the size of the special effect superimposed in the foreground image, the special effect is superimposed on the target object in the foreground image.

When there are multiple target objects in the foreground image, the special effects can be superimposed synchronously on the multiple target objects in the foreground image; of course, the special effect superimpositions on the multiple target objects in the foreground image can also be asynchronous.

A realization process of superimposing the special effect on the target object in the foreground image according to the position of the special effect superimposed on the foreground image and the size of the special effect superimposed on the foreground image may include but not limited to the following steps:

(1) Determine the pixel occlusion relationship between the special effect and the target object according to the position of the special effect superimposed in the foreground image, the size of the special effect superimposed in the foreground image, and the size of the target object, where the size of the target object is Determined according to the contour information of the target object.

In this step (1), the size of the target object is the size of the target object in the foreground image, and the contour information of the target object is the contour information of the target object in the foreground image. Since the size of the foreground image in the embodiment of the present application is equal to the size of the image, the contour information of the target object in the foreground image is the same as the contour information of the target object in the image, that is, the target object is in the foreground image. The size is equal to the size of the target object in the image.

In some embodiments, according to the position of the special effect superimposed in the foreground image, the size of the special effect superimposed in the foreground image, and the size of the target object, the process of determining the pixel occlusion relationship between the special effect and the target object may include: According to the position of the special effect superimposed in the foreground image, the size of the special effect superimposed in the foreground image, and the size of the target object, determine the front and back position relationship of each pixel of the special effect relative to the target object when the special effect is superimposed on the foreground image; according to the front and back position relationship , To determine the pixel occlusion relationship between the special effect and the target object. Fig. 4(1) is a schematic diagram of the stereoscopic special effect 10; Fig. 4(1) is the front and back positional relationship between the stereoscopic special effect 10 and the target object 20 when the stereoscopic special effect 10 is superimposed on the target object 20 in the foreground image Schematic. Please refer to Figure 4(1)-(2). When the 3D special effect 10 is superimposed on the target object 20 in the foreground image, the area 11 of the 3D special effect 10 (the part in bold in Figure 4(2)) is located on the target object 20 Back; the area 12 of the three-dimensional special effect 10 is located in front of the target object 20.

Optionally, the process of determining the pixel occlusion relationship between the special effect and the target object according to the front and back position relationship may include: in the foreground image, if the pixel of the special effect is located behind the target object, determining the pixel occlusion at the corresponding position of the target object Pixels for special effects. Optionally, according to the front and back position relationship, the process of determining the pixel occlusion relationship between the special effect and the target object includes: in the foreground image, if the pixel of the special effect is located in front of the target object, determining the corresponding position where the pixel of the special effect occludes the target object Of pixels. Please refer to Figure 4(2). When the stereoscopic special effect 10 is superimposed on the target object 20 in the foreground image, since the area 11 is located behind the target object 20, the pixels of the area 11 are corresponding to the position of the target object 20 (the target object 20 and the area 11 overlapped by the pixel); in addition, because the area 12 is located in front of the target object 20, the pixels of the area 12 will block the corresponding position of the target object (the position of the overlapped portion on the target object 20 and the area 12) Of pixels.

(2) According to the pixel occlusion relationship, superimpose special effects on the target object in the foreground image;

Fig. 4(3) is a stereoscopic special effect foreground image obtained by superimposing a stereoscopic special effect on the target object in the foreground image. Please refer to Figures 4(2)-(3). When the stereoscopic effect 10 is superimposed on the target object 20 in the foreground image, the pixels in the area 11 are blocked by the pixels corresponding to the target object 20, so Figure 4(3) The area 11 will not be displayed; the pixels of the area 12 will block the pixels of the corresponding position of the target object. Therefore, in FIG. 4(3), the area 12 is wrapped and displayed around the target object 20.

It should be noted that, according to the position information and the contour information, the implementation manner of superimposing special effects on the target object in the foreground image may also be implemented in other manners, and is not limited to steps 1 to 3.

In S303, the foreground image after the special effect is superimposed and the image are superimposed.

Exemplarily, the foreground image is placed in the field of view of the virtual camera to ensure that the field of view of the virtual camera just contains the entire foreground image. Since the foreground image only contains the target object, the remaining pixels are transparent, that is, the alpha channel (alpha channel, used for recording The special layer of transparency information) is 0, therefore, the occlusion relationship between the stereoscopic special effect and the foreground image is consistent with the occlusion relationship between the target object and the stereoscopic special effect. The 3D special effect foreground image with occlusion is obtained through the depth rendering method of the virtual camera, and the 3D special effect foreground image is superimposed on the image captured by the real camera to obtain the 3D special effect superimposed image with the occlusion effect.

In the embodiment of the present application, after superimposing the foreground image with the special effect superimposed on the image, the pixels of the special effect block the pixels at the corresponding position of the background of the image, and the corresponding position of the background is the position of the overlapping part of the special effect and the background. Please follow the embodiment shown in FIG. 4, the pixels in the area 12 block the pixels in the corresponding position of the background of the image (the position where the area 12 overlaps with the background).

It should be noted that, according to the position information and the contour information, the implementation manner of superimposing special effects on the target object in the image may also be implemented in other manners, which is not limited to S301-S303.

The display content of the special effect can be set as required. For example, in some embodiments, the display content of the special effect is determined according to the status information of the target object. Exemplarily, the video stream is a video stream of a competitive game, and the status information of the target object includes one or more of blood volume, attack power, defense power, and skills (for example, additional game duration, accelerated movement skills); it should be understood Yes, the status information of the target object is not limited to the status information of the target object listed above, and can also be other.

Optionally, the display contents of special effects corresponding to different types of status information are different. For example, the special effect corresponding to the blood volume of the target object can be characterized by the blood bar. The blood bar is used to indicate the real-time blood volume of the target object. When the blood volume of the target object drops to 0, the broken blood bar can be simulated by special effects. , For another example, when the blood volume of the target object is reduced, the blood return effect is displayed when the blood return condition is met. The blood return effect can refer to the effect of surrounding characters in general online games. In this embodiment, the surrounding target can be displayed around the target object. For example, when the current target object is attacked by other target objects and loses blood, the special effect of fire is displayed around the current target object, and the corresponding blood loss value will pop up on the special effect of the blood bar. For another example, when the attack power of the target object increases, a whirlwind special effect surrounding the target object can be displayed around the target object, as shown in Figure 4(3). For another example, when the defensive power of the target object is increased, the special effect of the shield surrounding the target object can be displayed around the target object. For another example, when an accelerated movement skill is added to the target object, a special effect of accelerated movement surrounding the target object can be displayed around the bottom of the target object. For another example, when the target object is invincible, a three-dimensional protective cover is displayed around the target object.

Optionally, the display content of the special effect includes at least one of the pattern and color of the special effect, and different special effects can be distinguished by the pattern. For example, when the blood volume drops to 0, the corresponding special effect pattern is broken; when the blood returns, The corresponding special effect pattern is the blood return effect surrounding the target object; different special effects can also be distinguished by color, or different special effects can be distinguished by pattern and color. Of course, the display content of the special effect can also be other, not limited to patterns and colors.

It is understandable that, in addition to the status information of the target object, some other information of the target object can also be displayed around the corresponding target object through special effects simulation, such as the identity information of the target object.

Optionally, the posture of the special effect is determined according to the posture of the camera when the image on which the special effect is currently superimposed is collected, and the presentation effect of the special effect can be enriched by combining the posture of the camera. Among them, the posture of the special effect is used to characterize the three-dimensional degree of the special effect, that is, after the special effect is superimposed on the image, the special effect displays the three-dimensional effect.

The posture of the special effect in the embodiment of the present application changes with the posture of the shooting device, so as the shooting angle of the image is switched, the angle of the target object in the image will change, and the special effect presents the corresponding angle change, making the special effect and the target The object fits more closely, and the visual effect is as if there are real special effects around the target object.

Optionally, the posture of the shooting device includes the posture of the shooting device relative to the scene where the target object is located. Based on the posture of the shooting device relative to the scene where the target object is located, the presentation effect of the stereo special effect in the image can be determined.

Among them, the posture of the shooting device relative to the scene where the target object is located is the translation and rotation relationship between the coordinate system of the shooting device and the scene coordinate system (the scene coordinate system may be a world coordinate system). Optionally, the shooting device is relative to the target object. The posture of the scene includes the pitch posture of the shooting device relative to the scene where the target object is located. When the pitching attitude of the shooting device relative to the scene where the target object is located changes, the special effect also produces the attitude change in the pitching direction. For example, when the shooting device rotates vertically upwards, the target object moves down in the image, and the special effect is also in the image. Corresponds to downward movement, and the distance that the special effect moves downward is equal to the distance that the target object moves downward; when the camera is turned vertically downward, the target object moves upward in the image, and the special effect also moves upward in the image. And the distance moved by the special effect is equal to the distance moved upward by the target object.

In some embodiments, the camera is mounted on the pan-tilt, and the posture of the camera is acquired through the pan-tilt. Exemplarily, the camera is equipped with a posture sensor (such as a visual odometer), the PTZ obtains the data of the posture sensor, and determines the posture of the camera according to the data of the posture sensor; of course, the way the PTZ obtains the posture of the camera is different. Limited to this. In addition, in the embodiment of the present application, the pan-tilt is fixedly installed at a preset position relative to the scene where the target object is located, and the acquisition of the posture of the camera is easier. Among them, the preset position can be selected according to needs.

In some embodiments, the camera is mounted on the drone, and the posture of the camera is acquired by the drone. Exemplarily, the camera is provided with an attitude sensor, and the drone obtains the data of the attitude sensor, and determines the attitude of the camera according to the data of the attitude sensor; of course, the manner in which the drone acquires the attitude of the camera is not limited to this.

Figure 5 is a schematic diagram of the method flow of the video processing method in another embodiment of the application; the execution subject of the video processing method in the embodiment of the application may be a game server or other devices capable of data processing, such as individuals terminal. Referring to FIG. 5, the video processing method of the embodiment of the present invention may include S501 to S502.

Among them, in S501, a video stream is obtained.

In S502, special effect processing is performed on each frame of image including the target object in the video stream, so that the target object surrounds the special effect; wherein the stereoscopic degree of the special effect is determined according to the posture of the camera when capturing the image with the current superimposed special effect.

Optionally, the posture of the special effect changes with the posture of the shooting device, and the posture of the special effect is used to represent the three-dimensional degree of the special effect.

Optionally, the posture of the shooting device includes the posture of the shooting device relative to the scene where the target object is located.

Optionally, the posture of the shooting device relative to the scene where the target object is located includes the pitch posture of the shooting device relative to the scene where the target object is located.

Optionally, the camera is mounted on a pan-tilt, and the posture of the camera is acquired through the pan-tilt.

Optionally, the pan/tilt is fixedly installed at a preset position relative to the scene where the target object is located.

Optionally, the camera is mounted on the drone, and the posture of the camera is acquired by the drone.

Optionally, special effect processing is performed on each frame of image including the target object in the video stream to make the target object surround with special effects, including: processing each frame of image including the target object in the video stream to obtain the target object in the image According to the position information and contour information, special effects are superimposed on the target object in the image, so that the target object is surrounded by special effects.

Optionally, according to the position information and contour information, superimpose special effects on the target object in the image to make the target object surround with special effects, including: segmenting the target object and the background in the image according to the position information and contour information to obtain Foreground image; according to the position information and contour information, superimpose special effects on the target object in the foreground image, so that the target object surrounds with special effects; superimpose the foreground image and the image after the special effects are superimposed.

Optionally, according to the position information and the contour information, superimpose the special effect on the target object in the foreground image, including: determining the position of the special effect superimposed in the foreground image according to the position information; determining the special effect superimposed in the foreground image according to the contour information Size; according to the position of the special effect superimposed in the foreground image and the size of the special effect superimposed in the foreground image, the special effect is superimposed on the target object in the foreground image.

Optionally, superimpose the special effect on the target object in the foreground image according to the position of the special effect superimposed in the foreground image and the size of the special effect superimposed in the foreground image, including: superimposing the special effect on the foreground image according to the position of the special effect superimposed in the foreground image Determine the pixel occlusion relationship between the special effect and the target object in the size of the foreground image and the size of the target object; according to the pixel occlusion relationship, superimpose the special effect on the target object in the foreground image; among them, the size of the target object To be determined based on the contour information of the target object.

Optionally, determine the pixel occlusion relationship between the special effect and the target object according to the position of the special effect superimposed in the foreground image, the size of the special effect superimposed in the foreground image, and the size of the target object, including: superimposing the special effect on the foreground image according to the special effect The position, the size of the special effect superimposed on the foreground image, and the size of the target object in the foreground image, determine the front and back position relationship of each pixel of the special effect relative to the target object when the special effect is superimposed on the foreground image; determine the special effect and the target object according to the front and back position relationship The pixel occlusion relationship between.

Optionally, determining the pixel occlusion relationship between the special effect and the target object according to the front and back position relationship includes: in the foreground image, if the pixel of the special effect is located behind the target object, determining that the pixel at the corresponding position of the target object occludes the pixel of the special effect .

Optionally, determining the pixel occlusion relationship between the special effect and the target object according to the front and back position relationship includes: in the foreground image, if the pixel of the special effect is located in front of the target object, determining that the pixel of the special effect occludes the pixel at the corresponding position of the target object .

Optionally, the pixels of the special effect block the pixels of the corresponding position of the background of the image, and the corresponding position of the background is the position of the overlapping part of the special effect and the background.

Optionally, the display content of the special effect is determined according to the status information of the target object.

Optionally, the video stream is a video stream of a competitive game, and the status information of the target object includes one or more of blood volume, attack power, defense power, and skill.

Optionally, the display contents of special effects corresponding to different types of status information are different.

Optionally, the display content of the special effect includes at least one of a pattern and color of the special effect.

Optionally, processing the video stream to obtain the position information and contour information of the target object in each image of the video stream includes: processing the video stream based on an instance segmentation algorithm to obtain each target object in the video stream Position information and contour information in an image.

Optionally, the video stream is a video stream of a competitive game, and the target object includes one or more of a moving object and a stationary object.

For the implementation process of the video processing method of the embodiment shown in FIG. 5, reference may be made to the description of the corresponding part of the implementation process of the video processing method of the embodiment shown in FIG. 2.

Corresponding to the video processing method of the foregoing embodiment, an embodiment of the present application also provides a video processing device. Fig. 6 is a structural block diagram of a video processing device in an embodiment of the present application; referring to Fig. 6, the video processing device may include a storage device and one or more processors.

Among them, the storage device is used to store program instructions; one or more processors call the program instructions stored in the storage device.

In some embodiments, when the program instructions are executed, one or more processors are individually or collectively configured to perform the following operations: obtain a video stream; perform operations on each frame of the video stream including the target object. Processing to obtain the position information and contour information of the target object in the image; according to the position information and contour information, special effects are superimposed on the target object in the image. The processor of this embodiment can implement the interaction method of the embodiment shown in FIG. 2 and FIG. 3 of the present invention, and the movable platform of this embodiment can be described with reference to the interaction method of the foregoing embodiment.

In some embodiments, when the program instructions are executed, one or more processors are individually or collectively configured to perform the following operations: obtain a video stream; Special effect processing is performed to make the target object surround with special effects; wherein, the three-dimensional degree of the special effect is determined according to the posture of the camera when the image on which the special effect is currently superimposed is collected. The processor of this embodiment can implement the interaction method of the embodiment shown in FIG. 5 of the present invention. The movable platform of this embodiment can be described with reference to the interaction method of the foregoing embodiment.

In the foregoing embodiment, the storage device may include a volatile memory (volatile memory), such as random-access memory (RAM); the storage device may also include a non-volatile memory (non-volatile memory), such as Flash memory (flash memory), hard disk drive (HDD) or solid-state drive (SSD); the storage device may also include a combination of the foregoing types of memory.

The processor may be a central processing unit (CPU). The processor may further include a hardware chip. The aforementioned hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The above-mentioned PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

In addition, an embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the video processing method in the first embodiment or the second embodiment are realized.

The computer-readable storage medium may be an internal storage unit of the video processing apparatus described in any of the foregoing embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of the video processing device, such as a plug-in hard disk, a smart media card (SMC), an SD card, and a flash card (Flash Card) equipped on the device. )Wait. Further, the computer-readable storage medium may also include both an internal storage unit of the video processing apparatus and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the video processing device, and can also be used to temporarily store data that has been output or will be output.

A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing relevant hardware through a computer program. The program can be stored in a computer readable storage medium. During execution, it may include the procedures of the above-mentioned method embodiments. Wherein, the storage medium can be a magnetic disk, an optical disc, a read-only memory (Read-Only Memory, ROM), or a random access memory (Random Access Memory, RAM), etc.

The above-disclosed are only some of the embodiments of this application, which of course cannot be used to limit the scope of rights of this application. Therefore, equivalent changes made in accordance with the claims of this application still fall within the scope of this application.

Claims

A video processing method, characterized in that the method includes:

Get the video stream;

Processing each frame of the image including the target object in the video stream to obtain position information and contour information of the target object in the image;

According to the position information and the contour information, superimpose special effects on the target object in the image.
The method according to claim 1, wherein the special effect is a three-dimensional special effect.
The method according to claim 1 or 2, wherein the performing special effect superposition on the target object in the image according to the position information and the contour information comprises:

Segmenting the target object and background in the image according to the position information and the contour information to obtain a foreground image;

Performing special effect superposition on the target object in the foreground image according to the position information and the contour information;

The foreground image after the special effect is superimposed and the image are superimposed.
The method according to claim 3, wherein the performing special effect superposition on the target object in the foreground image according to the position information and the contour information comprises:

Determine the position where the special effect is superimposed on the foreground image according to the position information;

Determine the size of the special effect superimposed on the foreground image according to the contour information;

According to the superimposed position of the special effect in the foreground image and the size of the special effect superimposed in the foreground image, the special effect is superimposed on the target object in the foreground image.
The method according to claim 4, wherein the special effect is superimposed on the foreground image according to the position of the special effect superimposed in the foreground image and the size of the special effect superimposed in the foreground image. The target object in the foreground image includes:

Determine the pixel occlusion between the special effect and the target object according to the position where the special effect is superimposed in the foreground image, the size of the special effect superimposed in the foreground image, and the size of the target object relation;

Superimpose the special effect on the target object in the foreground image according to the pixel occlusion relationship;

Wherein, the size of the target object is determined according to the contour information of the target object.
The method according to claim 5, wherein the location of the special effect superimposed in the foreground image, the size of the special effect superimposed in the foreground image, and the size of the target object , Determining the pixel occlusion relationship between the special effect and the target object includes:

According to the position where the special effect is superimposed in the foreground image, the size of the special effect superimposed in the foreground image, and the size of the target object, it is determined that when the special effect is superimposed on the foreground image, the The front and back positional relationship of each pixel of the special effect with respect to the target object;

According to the front and rear position relationship, the pixel occlusion relationship between the special effect and the target object is determined.
The method according to claim 6, wherein the determining the pixel occlusion relationship between the special effect and the target object according to the front and back position relationship comprises:

In the foreground image, if the pixel of the special effect is located behind the target object, it is determined that the pixel of the corresponding position of the target object blocks the pixel of the special effect.
The method according to claim 6, wherein the determining the pixel occlusion relationship between the special effect and the target object according to the front and back position relationship comprises:

In the foreground image, if the pixel of the special effect is located in front of the target object, it is determined that the pixel of the special effect occludes the pixel at the corresponding position of the target object.
The method according to claim 3, wherein the pixels of the special effect block the pixels of the corresponding position of the background of the image, and the corresponding position of the background is the position of the overlapping part of the special effect and the background.
The method according to claim 1 or 2, wherein the display content of the special effect is determined according to the status information of the target object.
The method according to claim 10, wherein the video stream is a video stream of a competitive game, and the status information of the target object includes one or more of blood volume, attack power, defense power, and skill.
The method according to claim 10, wherein the display content of the special effects corresponding to the different types of the state information is different.
The method according to claim 10, wherein the display content of the special effect includes at least one of a pattern and a color of the special effect.
The method according to claim 1 or 2, wherein the posture of the special effect is determined according to the posture of the photographing device when the image on which the special effect is currently superimposed is collected.
The method according to claim 14, wherein the posture of the special effect changes with the posture of the photographing device.
The method according to claim 15, wherein the posture of the shooting device comprises the posture of the shooting device relative to the scene where the target object is located.
The method according to claim 16, wherein the posture of the photographing device relative to the scene where the target object is located comprises a pitch posture of the photographing device relative to the scene where the target object is located.
The method according to any one of claims 15 to 17, wherein the camera is mounted on a pan/tilt, and the posture of the camera is acquired through the pan/tilt.
The method according to claim 18, wherein the pan-tilt is fixedly installed at a preset position relative to the scene where the target object is located.
The method according to any one of claims 15 to 17, wherein the camera is mounted on a drone, and the posture of the camera is acquired by the drone.
The method according to claim 1 or 2, wherein the processing the video stream to obtain the position information and contour information of the target object in each image of the video stream comprises:

The video stream is processed based on an instance segmentation algorithm to obtain the position information and contour information of the target object in each image of the video stream.
The method according to claim 1 or 2, wherein the video stream is a video stream of a competitive game, and the target object includes one or more of a moving object and a stationary object.
A video processing device, characterized in that the device includes:

Storage device for storing program instructions; and

One or more processors call program instructions stored in the storage device, and when the program instructions are executed, the one or more processors are individually or collectively configured to implement the following operations:

Get the video stream;

Processing each frame of the image including the target object in the video stream to obtain position information and contour information of the target object in the image;

According to the position information and the contour information, superimpose special effects on the target object in the image.
The device according to claim 23, wherein the special effect is a three-dimensional special effect.
The device according to claim 23 or 24, wherein the one or more processors perform special effects superimposition on the target object in the image according to the position information and the contour information, Separately or collectively are further configured to implement the following operations:

Segmenting the target object and background in the image according to the position information and the contour information to obtain a foreground image;

Performing special effect superposition on the target object in the foreground image according to the position information and the contour information;

The foreground image after the special effect is superimposed and the image are superimposed.
The device according to claim 25, wherein the one or more processors independently superimpose the target object in the foreground image according to the position information and the contour information. Ground or collectively is further configured to perform the following operations:

Determine the position where the special effect is superimposed on the foreground image according to the position information;

Determine the size of the special effect superimposed on the foreground image according to the contour information;

According to the superimposed position of the special effect in the foreground image and the size of the special effect superimposed in the foreground image, the special effect is superimposed on the target object in the foreground image.
The device according to claim 26, wherein the one or more processors are based on the position of the special effect superimposed in the foreground image and the size of the special effect superimposed in the foreground image, When the special effect is superimposed on the target object in the foreground image, it is separately or collectively further configured to perform the following operations:

Determine the pixel occlusion between the special effect and the target object according to the position where the special effect is superimposed in the foreground image, the size of the special effect superimposed in the foreground image, and the size of the target object relation;

Superimpose the special effect on the target object in the foreground image according to the pixel occlusion relationship;

Wherein, the size of the target object is determined according to the contour information of the target object.
28. The device according to claim 27, wherein the one or more processors are superimposed on the foreground image according to the position of the special effect, the size of the special effect superimposed in the foreground image, and The size of the target object, when determining the pixel occlusion relationship between the special effect and the target object, is separately or collectively further configured to implement the following operations:

According to the position where the special effect is superimposed in the foreground image, the size of the special effect superimposed in the foreground image, and the size of the target object, it is determined that when the special effect is superimposed on the foreground image, the The front and back positional relationship of each pixel of the special effect with respect to the target object;

According to the front and rear position relationship, the pixel occlusion relationship between the special effect and the target object is determined.
The device according to claim 28, wherein when the one or more processors determine the pixel occlusion relationship between the special effect and the target object according to the front and back position relationship, individually or collectively The ground is further configured to perform the following operations:

In the foreground image, if the pixel of the special effect is located behind the target object, it is determined that the pixel of the corresponding position of the target object blocks the pixel of the special effect.
The device according to claim 28, wherein when the one or more processors determine the pixel occlusion relationship between the special effect and the target object according to the front and back position relationship, individually or collectively The ground is further configured to perform the following operations:

In the foreground image, if the pixel of the special effect is located in front of the target object, it is determined that the pixel of the special effect occludes the pixel at the corresponding position of the target object.
22. The device of claim 25, wherein the pixels of the special effect block the pixels at the corresponding position of the background of the image, and the corresponding position of the background is the position of the overlapping portion of the special effect and the background.
The device according to claim 23 or 24, wherein the display content of the special effect is determined according to the status information of the target object.
The device according to claim 32, wherein the video stream is a video stream of a competitive game, and the status information of the target object includes one or more of blood volume, attack power, defense power, and skills.
The device according to claim 32, wherein the display contents of the special effects corresponding to the different types of the status information are different.
The device according to claim 32, wherein the display content of the special effect includes at least one of a pattern and a color of the special effect.
The device according to claim 23 or 24, wherein the posture of the special effect is determined according to the posture of the photographing device when the image on which the special effect is currently superimposed is collected.
The device according to claim 36, wherein the posture of the special effect changes with the posture of the photographing device.
The device according to claim 37, wherein the posture of the photographing device comprises the posture of the photographing device relative to the scene where the target object is located.
The device according to claim 38, wherein the posture of the photographing device relative to the scene where the target object is located comprises a pitch posture of the photographing device relative to the scene where the target object is located.
The device according to any one of claims 37 to 39, wherein the camera is mounted on a pan-tilt, and the posture of the camera is acquired through the pan-tilt.
The device according to claim 40, wherein the pan-tilt is fixedly installed at a preset position relative to the scene where the target object is located.
The device according to any one of claims 37 to 39, wherein the camera is mounted on a drone, and the posture of the camera is acquired by the drone.
The device according to claim 23 or 24, wherein the one or more processors are processing the video stream to obtain the position information of the target object in each image of the video stream and When the profile information, individually or collectively, it is further configured to implement the following operations:

The video stream is processed based on an instance segmentation algorithm to obtain the position information and contour information of the target object in each image of the video stream.
The device according to claim 23 or 24, wherein the video stream is a video stream of a competitive game, and the target object includes one or more of a moving object and a stationary object.
A video processing method, characterized in that the method includes:

Get the video stream;

Performing special effect processing on each frame of image including the target object in the video stream, so that the target object is surrounded by special effects;

Wherein, the three-dimensional degree of the special effect is determined according to the posture of the photographing device when the image on which the special effect is currently superimposed is collected.
The method according to claim 45, wherein the posture of the special effect changes with the posture of the shooting device, and the posture of the special effect is used to represent the three-dimensional degree of the special effect.
The method according to claim 45 or 46, wherein the posture of the photographing device comprises the posture of the photographing device relative to the scene in which the target object is located.
The method according to claim 47, wherein the posture of the photographing device relative to the scene where the target object is located comprises a pitch posture of the photographing device relative to the scene where the target object is located.
The method according to claim 45, wherein the camera is mounted on a pan-tilt, and the posture of the camera is acquired through the pan-tilt.
The method according to claim 49, wherein the pan-tilt is fixedly installed at a preset position relative to the scene where the target object is located.
The method according to claim 45, wherein the camera is mounted on a drone, and the posture of the camera is acquired by the drone.
The method according to claim 45, wherein the performing special effect processing on each frame of image including the target object in the video stream, so that the target object surrounds with special effects, comprises:

Processing each frame of image including the target object in the video stream to obtain position information and contour information of the target object in the image;

According to the position information and the contour information, a special effect is superimposed on the target object in the image, so that the target object is surrounded by a special effect.
The method according to claim 52, wherein the special effect is superimposed on the target object in the image according to the position information and the contour information, so that the target object surrounds the target object with special effects, include:

Segmenting the target object and background in the image according to the position information and the contour information to obtain a foreground image;

Performing special effect superposition on the target object in the foreground image according to the position information and the contour information, so that the target object surrounds the target object with special effects;

The foreground image after the special effect is superimposed and the image are superimposed.
The method according to claim 53, wherein the superimposing special effects on the target object in the foreground image according to the position information and the contour information comprises:

Determine the position where the special effect is superimposed on the foreground image according to the position information;

Determine the size of the special effect superimposed on the foreground image according to the contour information;

According to the superimposed position of the special effect in the foreground image and the size of the special effect superimposed in the foreground image, the special effect is superimposed on the target object in the foreground image.
The method according to claim 54, characterized in that, the special effect is superimposed on the foreground image according to the position of the special effect superimposed in the foreground image and the size of the special effect superimposed in the foreground image. The target object in the foreground image includes:

Determine the pixel occlusion between the special effect and the target object according to the position where the special effect is superimposed in the foreground image, the size of the special effect superimposed in the foreground image, and the size of the target object relation;

Superimpose the special effect on the target object in the foreground image according to the pixel occlusion relationship;

Wherein, the size of the target object is determined according to the contour information of the target object.
55. The method according to claim 55, wherein the position of the special effect superimposed in the foreground image, the size of the special effect superimposed in the foreground image, and the size of the target object are , Determining the pixel occlusion relationship between the special effect and the target object includes:

According to the position where the special effect is superimposed in the foreground image, the size of the special effect superimposed in the foreground image, and the size of the target object, it is determined that when the special effect is superimposed on the foreground image, the The front and back positional relationship of each pixel of the special effect with respect to the target object;

According to the front and rear position relationship, the pixel occlusion relationship between the special effect and the target object is determined.
The method of claim 56, wherein the determining the pixel occlusion relationship between the special effect and the target object according to the front and back position relationship comprises:

In the foreground image, if the pixel of the special effect is located behind the target object, it is determined that the pixel of the corresponding position of the target object blocks the pixel of the special effect.
The method of claim 56, wherein the determining the pixel occlusion relationship between the special effect and the target object according to the front and back position relationship comprises:

In the foreground image, if the pixel of the special effect is located in front of the target object, it is determined that the pixel of the special effect occludes the pixel at the corresponding position of the target object.
The method according to claim 53, wherein the pixels of the special effect block the pixels of the corresponding position of the background of the image, and the corresponding position of the background is the position of the overlapping portion of the special effect and the background.
The method according to claim 45, wherein the display content of the special effect is determined according to the status information of the target object.
The method according to claim 60, wherein the video stream is a video stream of a competitive game, and the status information of the target object includes one or more of blood volume, attack power, defense power, and skill.
The method according to claim 60, wherein the display contents of the special effects corresponding to the different types of the state information are different.
The method according to claim 61, wherein the display content of the special effect includes at least one of a pattern and a color of the special effect.
The method according to claim 52, wherein the processing the video stream to obtain the position information and contour information of the target object in each image of the video stream comprises:

The video stream is processed based on an instance segmentation algorithm to obtain the position information and contour information of the target object in each image of the video stream.
The method according to claim 45, wherein the video stream is a video stream of a competitive game, and the target object includes one or more of a moving object and a stationary object.
A video processing device, characterized in that the device includes:

Storage device for storing program instructions; and

One or more processors call program instructions stored in the storage device, and when the program instructions are executed, the one or more processors are individually or collectively configured to implement the following operations:

Get the video stream;

Performing special effect processing on each frame of image including the target object in the video stream, so that the target object is surrounded by special effects;

Wherein, the three-dimensional degree of the special effect is determined according to the posture of the photographing device when the image on which the special effect is currently superimposed is collected.
The device according to claim 66, wherein the posture of the special effect changes with the posture of the photographing device, and the posture of the special effect is used to represent the three-dimensional degree of the special effect.
The device according to claim 66 or 67, wherein the posture of the photographing device comprises the posture of the photographing device relative to the scene in which the target object is located.
The device according to claim 68, wherein the posture of the photographing device relative to the scene where the target object is located comprises a pitch posture of the photographing device relative to the scene where the target object is located.
The device according to claim 66, wherein the camera is mounted on a pan/tilt, and the posture of the camera is acquired through the pan/tilt.
The device according to claim 70, wherein the pan-tilt is fixedly installed at a preset position relative to the scene where the target object is located.
The device according to claim 66, wherein the camera is mounted on a drone, and the posture of the camera is acquired by the drone.
The device according to claim 66, wherein the one or more processors perform special effect processing on each frame of image including the target object in the video stream, so that the target object surrounds with special effects. , Individually or collectively, are further configured to implement the following operations:

Processing each frame of image including the target object in the video stream to obtain position information and contour information of the target object in the image;

According to the position information and the contour information, a special effect is superimposed on the target object in the image, so that the target object is surrounded by a special effect.
The device according to claim 73, wherein the one or more processors superimpose special effects on the target object in the image according to the position information and the contour information, so that all When the target object is surrounded by special effects, it is further configured individually or collectively to implement the following operations:

Segmenting the target object and background in the image according to the position information and the contour information to obtain a foreground image;

Performing special effect superposition on the target object in the foreground image according to the position information and the contour information, so that the target object surrounds the target object with special effects;

The foreground image after the special effect is superimposed and the image are superimposed.
The device according to claim 74, wherein the one or more processors independently superimpose the target object in the foreground image according to the position information and the contour information. Ground or collectively is further configured to perform the following operations:

Determine the position where the special effect is superimposed on the foreground image according to the position information;

Determine the size of the special effect superimposed on the foreground image according to the contour information;

According to the superimposed position of the special effect in the foreground image and the size of the special effect superimposed in the foreground image, the special effect is superimposed on the target object in the foreground image.
The device according to claim 75, wherein the one or more processors are based on the position of the special effect superimposed in the foreground image and the size of the special effect superimposed in the foreground image, When the special effect is superimposed on the target object in the foreground image, it is separately or collectively further configured to perform the following operations:

Determine the pixel occlusion between the special effect and the target object according to the position where the special effect is superimposed in the foreground image, the size of the special effect superimposed in the foreground image, and the size of the target object relation;

Superimpose the special effect on the target object in the foreground image according to the pixel occlusion relationship;

Wherein, the size of the target object is determined according to the contour information of the target object.
The device according to claim 76, wherein the one or more processors perform the superimposition of the special effect in the foreground image according to the position of the special effect, the size of the special effect superimposed in the foreground image, and The size of the target object, when determining the pixel occlusion relationship between the special effect and the target object, is separately or collectively further configured to implement the following operations:

According to the position where the special effect is superimposed in the foreground image, the size of the special effect superimposed in the foreground image, and the size of the target object, it is determined that when the special effect is superimposed on the foreground image, the The front and back positional relationship of each pixel of the special effect with respect to the target object;

According to the front and rear position relationship, the pixel occlusion relationship between the special effect and the target object is determined.
The device according to claim 77, wherein when the one or more processors determine the pixel occlusion relationship between the special effect and the target object according to the front-to-rear position relationship, individually or collectively The ground is further configured to perform the following operations:

In the foreground image, if the pixel of the special effect is located behind the target object, it is determined that the pixel of the corresponding position of the target object blocks the pixel of the special effect.
The device according to claim 77, wherein when the one or more processors determine the pixel occlusion relationship between the special effect and the target object according to the front-to-rear position relationship, individually or collectively The ground is further configured to perform the following operations:

In the foreground image, if the pixel of the special effect is located in front of the target object, it is determined that the pixel of the special effect occludes the pixel at the corresponding position of the target object.
The device according to claim 74, wherein the pixels of the special effect block the pixels at the corresponding position of the background of the image, and the corresponding position of the background is the position of the overlapping portion of the special effect and the background.
The device according to claim 66, wherein the display content of the special effect is determined according to the status information of the target object.
The device according to claim 81, wherein the video stream is a video stream of a competitive game, and the status information of the target object includes one or more of blood volume, attack power, defense power, and skill.
The device according to claim 81, wherein the display contents of the special effects corresponding to the different types of the state information are different.
The device according to claim 82, wherein the display content of the special effect includes at least one of a pattern and a color of the special effect.
The device according to claim 73, wherein the one or more processors are processing the video stream to obtain position information and contour information of the target object in each image of the video stream When, individually or collectively, it is further configured to implement the following operations:

The video stream is processed based on an instance segmentation algorithm to obtain the position information and contour information of the target object in each image of the video stream.
The device according to claim 66, wherein the video stream is a video stream of a competitive game, and the target object includes one or more of a moving object and a stationary object.