Nothing Special   »   [go: up one dir, main page]

CN112866507B - Intelligent panoramic video synthesis method and system, electronic device and medium - Google Patents

Intelligent panoramic video synthesis method and system, electronic device and medium Download PDF

Info

Publication number
CN112866507B
CN112866507B CN202110043474.4A CN202110043474A CN112866507B CN 112866507 B CN112866507 B CN 112866507B CN 202110043474 A CN202110043474 A CN 202110043474A CN 112866507 B CN112866507 B CN 112866507B
Authority
CN
China
Prior art keywords
foreground
video
signal
panoramic video
panoramic
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110043474.4A
Other languages
Chinese (zh)
Other versions
CN112866507A (en
Inventor
叶龙
冯晨曦
钟微
方力
胡飞
张勤
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Communication University of China
Original Assignee
Communication University of China
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Communication University of China filed Critical Communication University of China
Priority to CN202110043474.4A priority Critical patent/CN112866507B/en
Publication of CN112866507A publication Critical patent/CN112866507A/en
Application granted granted Critical
Publication of CN112866507B publication Critical patent/CN112866507B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00Details of television systems
    • H04N5/222Studio circuitry; Studio devices; Studio equipment
    • H04N5/2224Studio circuitry; Studio devices; Studio equipment related to virtual studio applications
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70Circuitry for compensating brightness variation in the scene
    • H04N23/741Circuitry for compensating brightness variation in the scene by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00Details of television systems
    • H04N5/222Studio circuitry; Studio devices; Studio equipment
    • H04N5/262Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/265Mixing

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Image Processing (AREA)
  • Studio Devices (AREA)

Abstract

本发明提供一种智能化的全景视频合成方法、系统、电子设备及介质包括:在全景视频的三维全景图像空间中自适应调整前景视频位置;采集全景视频的光照信息,对前景视频的光照信息进行自适应调整,所述光照信息包括亮度和色调;将自适应调整后的前景视频叠加在所述全景视频中。上述方法及装置提升全景视频合成的真实感。

Figure 202110043474

The present invention provides an intelligent panoramic video synthesis method, system, electronic equipment and medium including: adaptively adjusting the position of the foreground video in the three-dimensional panoramic image space of the panoramic video; Adaptive adjustment is performed, the illumination information includes brightness and hue; and the adaptively adjusted foreground video is superimposed on the panoramic video. The above method and device enhance the sense of reality of panoramic video synthesis.

Figure 202110043474

Description

智能化的全景视频合成方法、系统、电子设备及介质Intelligent panoramic video synthesis method, system, electronic equipment and medium

技术领域technical field

本发明涉及视频合成领域,更为具体地,涉及一种智能化的全景视频合成方法、系统、电子设备及介质。The present invention relates to the field of video synthesis, and more specifically, to an intelligent panoramic video synthesis method, system, electronic equipment and media.

背景技术Background technique

全景视频背景叠加技术以预先拍摄的全景视频作为合成背景,同用户在绿幕前拍摄的前景视频实时渲染融合,实现真实前景视频“转移”到真实全景视频背景中的跨时空效果。全景视频由多个鱼眼镜头组成的全景摄像机拍摄,前景视频由线性镜头拍摄,由于拍摄设备和拍摄环境的差异,直接将前景与背景进行叠加无法满足前景对象与背景的透视关系和光照一致性,进而导致合成画面与现实世界中观看到的画面不符,引发真实度下降。The panoramic video background overlay technology uses the pre-shot panoramic video as the synthetic background, and real-time rendering and fusion with the foreground video shot by the user in front of the green screen, so as to realize the cross-temporal effect of "transferring" the real foreground video to the real panoramic video background. The panoramic video is shot by a panoramic camera composed of multiple fisheye lenses, and the foreground video is shot by a linear lens. Due to the differences in shooting equipment and shooting environments, directly superimposing the foreground and background cannot satisfy the perspective relationship and illumination consistency between foreground objects and background , which in turn leads to the inconsistency between the synthesized picture and the picture viewed in the real world, causing a decrease in authenticity.

全景视频背景叠加技术脱胎于虚拟演播室技术。虚拟演播室主要包括绿幕抠像、摄像机跟踪等技术手段,可将计算机制作的虚拟三维场景与电视摄像机现场拍摄的人物活动图像进行数字化合成,使人物与虚拟背景同步变化,呈现出现实生活中无法看到的奇幻画面,突破了传统布景、道具、灯光、场地等演播室制作工艺的限制,为用户提供了新颖的视觉体验。Panoramic video background overlay technology was born out of virtual studio technology. The virtual studio mainly includes technical means such as green screen keying and camera tracking, which can digitally synthesize the virtual three-dimensional scene produced by the computer and the moving images of the characters captured by the TV camera on the spot, so that the characters and the virtual background change synchronously, presenting the real life. The fantasy picture that cannot be seen breaks through the limitations of traditional studio production techniques such as sets, props, lighting, and venues, and provides users with a novel visual experience.

尽管虚拟演播室技术已得到广泛运用,然而绝大多数的研究人员及相关领域从业者都倾向于利用虚拟引擎设计“虚幻”的CG场景并同现实的前景视频进行融合,这固然可以创造出令人惊叹的虚拟特效,但如果将抠像得到的前景视频与不同时空的真实背景视频进行融合,这种“真实的虚幻”可能对观众更具有冲击力。Although virtual studio technology has been widely used, most researchers and practitioners in related fields tend to use virtual engines to design "illusory" CG scenes and integrate them with realistic foreground videos. Amazing virtual special effects, but if the foreground video obtained by keying is integrated with the real background video of different time and space, this "real illusion" may have more impact on the audience.

增强现实(Augmented Reality,简称AR)技术是一种将虚拟信息与真实世界巧妙融合的技术,其将原本在现实世界的空间范围中比较难以进行体验的信息在电脑等科学技术的基础上,实施模拟仿真处理,叠加在真实世界中,并且在这一过程中能够被人类感官所感知,从而实现超越现实的感官体验。但AR是真实环境与虚拟物体之间的融合,并未涉及真实环境与真实对象之间的融合。Augmented reality (Augmented Reality, referred to as AR) technology is a technology that ingeniously integrates virtual information with the real world. Simulation processing is superimposed on the real world, and in this process can be perceived by human senses, so as to achieve a sensory experience beyond reality. However, AR is the fusion between the real environment and virtual objects, and does not involve the fusion between the real environment and real objects.

Long Ye等人在2019年提出了一种简单易行的全景视频背景叠加方案,该方案基于UE4引擎,将全景视频粘贴于三维球体内侧播放,用放在球心处的虚拟摄像机来捕获背景画面。手机拍摄的前景画面放置于HUD用户控件上,抠像后实现了与全景视频的叠加。该方案将虚拟演播室技术同全景视频相结合,实现了真实的前景视频与不同时空真实背景视频进行融合,这种“虚幻的真实感”可以给观众带来更加新鲜的观看体验。Long Ye et al. proposed a simple and easy panoramic video background overlay scheme in 2019. Based on the UE4 engine, the scheme pastes the panoramic video inside the three-dimensional sphere for playback, and uses a virtual camera placed at the center of the sphere to capture the background image. . The foreground picture taken by the mobile phone is placed on the HUD user control, and after keying, it is superimposed with the panoramic video. This solution combines virtual studio technology with panoramic video to realize the integration of real foreground video and real background video of different time and space. This "illusory sense of reality" can bring a fresher viewing experience to the audience.

由于全景视频和前景视频拍摄设备参数不同,直接将两者叠加会产生明显的不协调,Long Ye等人尝试通过手动调节前景视频各色彩通道增益实现色调一致性。但该方法针对不同内容输入视频需要分别手动调整,无法实现智能化适配。另外,该方案并不能有效保证前景与背景的透视关系及位置一致性。Due to the different shooting equipment parameters of the panoramic video and the foreground video, directly superimposing the two will produce obvious incongruity. Long Ye et al. tried to achieve tone consistency by manually adjusting the gain of each color channel of the foreground video. However, this method needs to be manually adjusted for different content input videos, which cannot achieve intelligent adaptation. In addition, this solution cannot effectively guarantee the perspective relationship and position consistency between the foreground and the background.

传统的虚拟演播室技术将真实前景合成到由计算机建模生成的三维虚拟场景中,AR则将虚拟模型与真实背景相结合,这两种技术固然可以给用户提供新奇的视觉体验,但虚拟模型的纹理、色泽、以及轮廓和真实物体有着明显区别,这在一定程度上降低了用户的临场感。The traditional virtual studio technology synthesizes the real foreground into the 3D virtual scene generated by computer modeling, while AR combines the virtual model with the real background. These two technologies can certainly provide users with a novel visual experience, but the virtual model The texture, color, and outline of the real object are obviously different from the real object, which reduces the user's sense of presence to a certain extent.

Long Ye等人针对以上问题提出了面向全景视频的背景叠加方案,根据前景摄像机位姿信息自动渲染对应全景视频的子区域作为背景视频,实现前景人物和真实背景的有机融合。但该方案有以下问题没有解决:1、由于全景视频和前景视频拍摄设备参数不同,导致合成后的视频前景背景光照不一致,尽管Long Ye等人的方案中设计了HSV色彩调节功能,但手动调节的方式太过费事费力,不能实时的智能合成理想的画面;2、另一方面,LongYe等人方案将全景视频放置于三维球体内侧,通过先验结构信息自动实现鱼眼变换,最后将变换后的二维视频同二维前景视频叠加。然而该方法不能有效描述前景视频和背景子视频的透视关系,当观看视角变化显著时,前景对象同背景视频的正常透视关系无法维持,影响合成画面真实感。Aiming at the above problems, Long Ye et al. proposed a background overlay scheme for panoramic video, which automatically renders the sub-region corresponding to the panoramic video as the background video according to the pose information of the foreground camera, so as to realize the organic fusion of foreground characters and real background. However, this solution has the following problems that have not been resolved: 1. Due to the different parameters of the panoramic video and foreground video shooting equipment, the foreground and background lighting of the synthesized video are inconsistent. Although the HSV color adjustment function is designed in the solution of Long Ye et al., manual adjustment 2. On the other hand, LongYe et al. proposed to place the panoramic video inside the 3D sphere, automatically realize the fisheye transformation through the prior structure information, and finally convert the transformed The 2D video of is overlaid with the 2D foreground video. However, this method cannot effectively describe the perspective relationship between the foreground video and the background sub-video. When the viewing angle changes significantly, the normal perspective relationship between the foreground object and the background video cannot be maintained, which affects the realism of the composite picture.

发明内容Contents of the invention

鉴于上述问题,本发明主要针对全景视频背景叠加技术中前景与背景位置不一致、光照不一致这两个问题提供一种提升全景视频合成的真实感的智能化的全景视频合成方法、系统、电子设备及介质。In view of the above-mentioned problems, the present invention mainly provides an intelligent panoramic video synthesis method, system, electronic equipment, and method for improving the realism of panoramic video synthesis for the two problems of inconsistency between foreground and background positions and inconsistent illumination in panoramic video background overlay technology. medium.

根据本发明的一个方面,提供一种智能化的全景视频合成方法,包括:According to one aspect of the present invention, a kind of intelligent panoramic video synthesis method is provided, comprising:

在全景视频的三维全景图像空间中自适应调整前景视频位置;Adaptively adjust the position of the foreground video in the three-dimensional panoramic image space of the panoramic video;

采集全景视频的光照信息,对前景视频的光照信息进行自适应调整,所述光照信息包括亮度和色调;Collecting the lighting information of the panoramic video, and adaptively adjusting the lighting information of the foreground video, the lighting information including brightness and hue;

将自适应调整后的前景视频叠加在所述全景视频中。The adaptively adjusted foreground video is superimposed on the panoramic video.

可选地,所述采集全景视频的光照信息,对前景视频的色调进行自适应调整的步骤包括:Optionally, the step of collecting the lighting information of the panoramic video and adaptively adjusting the hue of the foreground video includes:

利用逆色调映射方法提高全景视频光照信息的动态范围,得到HDR图像;Using the inverse tone mapping method to improve the dynamic range of the lighting information of the panoramic video, and obtain the HDR image;

采集逆色调映射得到的HDR图像所包含的光照信息;Collect the lighting information contained in the HDR image obtained by inverse tone mapping;

调整前景视频的光照信息,使得前景视频和HDR图像的光照信息相似。Adjust the lighting information of the foreground video so that the lighting information of the foreground video and the HDR image are similar.

可选地,所述采集全景视频的光照信息,对前景视频的色调进行自适应调整的步骤还包括:Optionally, the step of collecting the lighting information of the panoramic video and adaptively adjusting the hue of the foreground video also includes:

采集得到的光照信息在三维模型中生成环境光。The collected lighting information generates ambient light in the 3D model.

可选地,所述利用逆色调映射方法提高全景视频光照信息的动态范围的步骤包括:Optionally, the step of using the inverse tone mapping method to improve the dynamic range of the lighting information of the panoramic video includes:

根据下式将普通动态范围的全景视频映射至高动态范围The panoramic video of normal dynamic range is mapped to high dynamic range according to the following formula

Figure BDA0002896760540000031
Figure BDA0002896760540000031

其中,

Figure BDA0002896760540000032
R为扩展阈值,大于R的亮度值都会被扩展,Iw,max为扩展后的最大亮度,α为控制色调曲线伸展的衰减指数,Ld(x)为原亮度,Lw(x)为扩展后的亮度。in,
Figure BDA0002896760540000032
R is the expansion threshold, the brightness value greater than R will be expanded, I w,max is the maximum brightness after expansion, α is the attenuation index that controls the extension of the tone curve, L d (x) is the original brightness, L w (x) is Extended brightness.

可选地,所述在全景视频的三维全景图像空间中自适应调整前景视频位置的步骤包括:Optionally, the step of adaptively adjusting the position of the foreground video in the three-dimensional panoramic image space of the panoramic video includes:

获得三维极坐标系中的全景视频信号;Obtain the panoramic video signal in the three-dimensional polar coordinate system;

获得前景视频在全景视频中的前景附着点,构成附着点集;Obtain the foreground attachment point of the foreground video in the panoramic video to form an attachment point set;

通过坐标变换映射将前景视频信号及其掩膜信号变换至三维极坐标上,获得掩膜信号中所有前景视频信号组成的前景信号点集;transforming the foreground video signal and its mask signal to three-dimensional polar coordinates through coordinate transformation mapping, and obtaining a foreground signal point set composed of all foreground video signals in the mask signal;

通过阈值判断方法获得前景信号点集中的底部前景信号点集;Obtaining the bottom foreground signal point set in the foreground signal point set by a threshold judgment method;

根据底部前景信号点集获得前景视频信号的定位点;Obtain the anchor point of the foreground video signal according to the bottom foreground signal point set;

随机选取附着点集中的前景附着点,根据前景附着点和定位点的距离,将前景视频信号在三维极坐标中进行旋转,使得旋转后的前景视频信号的定位点同前景附着点重合;Randomly select the foreground attachment point in the attachment point set, and rotate the foreground video signal in three-dimensional polar coordinates according to the distance between the foreground attachment point and the anchor point, so that the anchor point of the rotated foreground video signal coincides with the foreground attachment point;

根据三维极坐标系中拍摄装置的高度和前景附着点获得映射半径;Obtain the mapping radius according to the height of the shooting device and the attachment point of the foreground in the three-dimensional polar coordinate system;

将旋转后的三维极坐标下的前景视频信号缩放至以映射半径为半径的三维球体内表面上;Scaling the rotated foreground video signal in three-dimensional polar coordinates to the inner surface of a three-dimensional sphere whose radius is the mapping radius;

其中,所述将自适应调整后的前景视频叠加在所述全景视频中的步骤包括:Wherein, the step of superimposing the adaptively adjusted foreground video on the panoramic video includes:

在三维球体内合成掩膜信号、前景视频信号和全景视频信号。A mask signal, a foreground video signal and a panoramic video signal are synthesized within a three-dimensional sphere.

根据本发明的另一个方面,提供一种智能化的全景视频合成系统,包括:According to another aspect of the present invention, a kind of intelligent panoramic video synthesis system is provided, comprising:

位置自适应调整模块,在全景视频的三维全景图像空间中自适应调整前景视频位置;A position adaptive adjustment module, adaptively adjusts the position of the foreground video in the three-dimensional panoramic image space of the panoramic video;

采集模块,采集位置自适应调整模块前景视频自适应调整后的全景视频的光照信息,所述光照信息包括亮度和色调;The acquisition module collects the lighting information of the panoramic video after the position adaptive adjustment module foreground video adaptive adjustment, and the lighting information includes brightness and hue;

光照自适应调整模块,对前景视频的光照信息根据采集模块采集的全景视频的光照信息进行自适应调整;The illumination adaptive adjustment module is adapted to adjust the illumination information of the foreground video according to the illumination information of the panoramic video collected by the acquisition module;

三维合成模块,在全景视频的三维全景图像空间中叠加经过光照自适应调整模块调整后的前景视频。The three-dimensional synthesis module superimposes the foreground video adjusted by the light adaptive adjustment module in the three-dimensional panoramic image space of the panoramic video.

可选地,所述采集模块包括:Optionally, the collection module includes:

映射单元,利用逆色调映射方法提高全景视频光照信息的动态范围,得到HDR图像;A mapping unit that utilizes an inverse tone mapping method to improve the dynamic range of the panoramic video lighting information to obtain an HDR image;

信息采集单元,采集逆色调映射得到的HDR图像所包含的光照信息;an information collection unit, for collecting illumination information contained in the HDR image obtained by inverse tone mapping;

其中,所述光照自适应调整模块调整前景视频的光照信息,使得前景视频和HDR图像的光照信息相似。Wherein, the illumination adaptive adjustment module adjusts the illumination information of the foreground video, so that the illumination information of the foreground video and the HDR image are similar.

可选地,所述位置自适应调整模块包括:Optionally, the position adaptive adjustment module includes:

第一采集单元,获得三维极坐标系中的全景视频信号;The first acquisition unit obtains the panoramic video signal in the three-dimensional polar coordinate system;

附着点集构建单元,获得前景视频在全景视频中的前景附着点,构成附着点集;The attachment point set construction unit is used to obtain the foreground attachment point of the foreground video in the panoramic video to form the attachment point set;

前景信号点集构建单元,通过坐标变换映射将前景视频信号及其掩膜信号变换至三维极坐标上,获得掩膜信号中所有前景视频信号组成的前景信号点集;The foreground signal point set construction unit transforms the foreground video signal and its mask signal to three-dimensional polar coordinates through coordinate transformation mapping, and obtains a foreground signal point set composed of all foreground video signals in the mask signal;

底部前景信号点集构建单元,通过阈值判断方法获得前景信号点集中的底部前景信号点集;The bottom foreground signal point set construction unit obtains the bottom foreground signal point set in the foreground signal point set by a threshold judgment method;

定位点获得单元,根据底部前景信号点集获得前景视频信号的定位点;The positioning point obtaining unit obtains the positioning point of the foreground video signal according to the bottom foreground signal point set;

旋转单元,随机选取附着点集中的前景附着点,根据前景附着点和定位点的距离,将前景视频信号在三维极坐标中进行旋转,使得旋转后的前景视频信号的定位点同前景附着点重合;The rotation unit randomly selects the foreground attachment point in the attachment point set, and rotates the foreground video signal in three-dimensional polar coordinates according to the distance between the foreground attachment point and the anchor point, so that the anchor point of the rotated foreground video signal coincides with the foreground attachment point ;

映射半径获得单元,根据三维极坐标系中拍摄装置的高度和前景附着点获得映射半径;The mapping radius obtaining unit obtains the mapping radius according to the height of the shooting device in the three-dimensional polar coordinate system and the attachment point of the foreground;

缩放单元,将旋转后的三维极坐标下的前景视频信号缩放至以映射半径为半径的三维球体内表面上;A scaling unit, which scales the rotated foreground video signal in three-dimensional polar coordinates to the inner surface of a three-dimensional sphere with the mapping radius as the radius;

其中,三维合成模块在三维球体内合成掩膜信号、前景视频信号和全景视频信号。Wherein, the three-dimensional synthesis module synthesizes the mask signal, the foreground video signal and the panoramic video signal in the three-dimensional sphere.

此外,本发明还提供一种电子设备,所述电子设备包括:In addition, the present invention also provides an electronic device, which includes:

至少一个处理器;以及,at least one processor; and,

与所述至少一个处理器通信连接的存储器;其中,a memory communicatively coupled to the at least one processor; wherein,

所述存储器存储有可被所述至少一个处理器执行的指令,所述指令被所述至少一个处理器执行,以使所述至少一个处理器能够执行上述智能化的全景视频合成方法。The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor can execute the above-mentioned intelligent panoramic video synthesis method.

此外,本发明还提供一种计算机可读存储介质,存储有计算机程序,其特征在于,所述计算机程序被处理器执行时实现上述智能化的全景视频合成方法。In addition, the present invention also provides a computer-readable storage medium storing a computer program, wherein the above-mentioned intelligent panoramic video synthesis method is realized when the computer program is executed by a processor.

本发明所述智能化的全景视频合成方法、系统、电子设备及介质基于HDR照度采集和三维空间下位置一致性校准,高效的提升前景视频和全景视频合成的真实性。The intelligent panoramic video synthesis method, system, electronic equipment and medium of the present invention are based on HDR illuminance acquisition and position consistency calibration in three-dimensional space, and efficiently improve the authenticity of foreground video and panoramic video synthesis.

附图说明Description of drawings

通过参考以下结合附图的说明,并且随着对本发明的更全面理解,本发明的其它目的及结果将更加明白及易于理解。在附图中:Other objects and results of the present invention will become clearer and easier to understand by referring to the following description in conjunction with the accompanying drawings, and with a more comprehensive understanding of the present invention. In the attached picture:

图1为本发明一实施例提供的智能化的全景视频合成方法的流程示意图;Fig. 1 is a schematic flow chart of an intelligent panoramic video synthesis method provided by an embodiment of the present invention;

图2本发明另一实施例提供的智能化的全景视频合成系统的模块示意图;Fig. 2 is a block diagram of an intelligent panoramic video synthesis system provided by another embodiment of the present invention;

图3为本发明一实施例提供的实现智能化的全景视频合成方法的电子设备的内部结构示意图。FIG. 3 is a schematic diagram of an internal structure of an electronic device for implementing an intelligent panoramic video synthesis method according to an embodiment of the present invention.

具体实施方式detailed description

应当理解,此处所描述的具体实施例仅仅用以解释本发明,并不用于限定本发明。It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

本发明提供一种智能化的全景视频合成方法。参照图1所示,为本发明一实施例提供的智能化的全景视频合成方法的流程示意图。该方法可以由一个装置执行,该装置可以由软件和/或硬件实现。The invention provides an intelligent panoramic video synthesis method. Referring to FIG. 1 , it is a schematic flowchart of an intelligent panoramic video synthesis method provided by an embodiment of the present invention. The method may be performed by a device, and the device may be implemented by software and/or hardware.

在本实施例中,智能化的全景视频合成方法包括:In the present embodiment, the intelligent panoramic video synthesis method includes:

步骤S1,在全景视频的三维全景图像空间中自适应调整前景视频位置;Step S1, adaptively adjusting the position of the foreground video in the three-dimensional panoramic image space of the panoramic video;

步骤S2,采集全景视频的光照信息,对前景视频的光照信息进行自适应调整,所述光照信息包括亮度和色调;Step S2, collecting lighting information of the panoramic video, and adaptively adjusting the lighting information of the foreground video, the lighting information including brightness and hue;

步骤S3,将自适应调整后的前景视频叠加在所述全景视频中。Step S3, superimposing the adaptively adjusted foreground video on the panoramic video.

在一个而实施例中,步骤S2包括:In one embodiment, step S2 includes:

利用逆色调映射方法提高全景视频光照信息的动态范围,得到HDR图像;Using the inverse tone mapping method to improve the dynamic range of the lighting information of the panoramic video, and obtain the HDR image;

采集逆色调映射得到的HDR图像所包含的光照信息;Collect the lighting information contained in the HDR image obtained by inverse tone mapping;

调整前景视频的光照信息,使得前景视频和HDR图像的光照信息相似。Adjust the lighting information of the foreground video so that the lighting information of the foreground video and the HDR image are similar.

可选地,还包括:Optionally, also include:

采集得到的光照信息在三维模型中生成环境光。The collected lighting information generates ambient light in the 3D model.

上述智能化的全景视频合成方法基于HDR照度采集和三维空间下位置一致性校准等技术提出了一种高效的真实感提升方法,从而为全景视频背景叠加系统提供更具真实性的前景/背景合成方案。The above-mentioned intelligent panoramic video synthesis method proposes an efficient realism enhancement method based on technologies such as HDR illumination acquisition and position consistency calibration in 3D space, so as to provide more realistic foreground/background synthesis for the panoramic video background overlay system Program.

在一个实施例中,所述利用逆色调映射方法提高全景视频光照信息的动态范围的步骤包括:In one embodiment, the step of improving the dynamic range of the lighting information of the panoramic video by using the inverse tone mapping method includes:

根据下式将普通动态范围的全景视频映射至高动态范围The panoramic video of normal dynamic range is mapped to high dynamic range according to the following formula

Figure BDA0002896760540000051
Figure BDA0002896760540000051

其中,

Figure BDA0002896760540000052
R为扩展阈值,大于R的亮度值都会被扩展,Iw,max为扩展后的最大亮度,α为控制色调曲线伸展的衰减指数,Ld(x)为原亮度,Lw(x)为扩展后的亮度。in,
Figure BDA0002896760540000052
R is the expansion threshold, the brightness value greater than R will be expanded, I w,max is the maximum brightness after expansion, α is the attenuation index that controls the extension of the tone curve, L d (x) is the original brightness, L w (x) is Extended brightness.

在一个实施例中,所述采集逆色调映射得到的HDR图像所包含的光照信息的步骤包括:In one embodiment, the step of collecting the illumination information contained in the HDR image obtained by inverse tone mapping includes:

在UE4中利用天空光照部件对逆色调映射得到的HDR图像所包含的光照信息进行采集。In UE4, the sky lighting component is used to collect the lighting information contained in the HDR image obtained by inverse tone mapping.

在一个实施例中,所述采集得到的光照信息在三维模型中生成环境光的步骤包括:In one embodiment, the step of generating ambient light in the three-dimensional model from the collected illumination information includes:

利用天空光照部件采集得到的光照信息在三维模型中生成环境光。Use the lighting information collected by the sky lighting component to generate ambient light in the 3D model.

在一个实施例中,所述调整前景视频的光照信息,使得前景视频和HDR图像的光照信息相似的步骤包括:In one embodiment, the steps of adjusting the illumination information of the foreground video so that the illumination information of the foreground video and the HDR image are similar include:

根据下式调整前景视频的光照信息Adjust the lighting information of the foreground video according to the following formula

Figure BDA0002896760540000061
Figure BDA0002896760540000061

其中,C为防止分母趋于0时产生数值错误的常数,

Figure BDA0002896760540000062
分别表示HDR图像红绿蓝三通道的加权平均值,其计算方法为
Figure BDA0002896760540000063
式中x表示红/绿/蓝通道的加权平均值,xi表示红/绿/蓝通道中第i个归一化像素值,di为第i个背景像素同前景的距离;
Figure BDA0002896760540000064
分别表示前景视频的红绿蓝三通道的平均值,
Figure BDA0002896760540000065
Figure BDA0002896760540000066
Figure BDA0002896760540000067
Figure BDA0002896760540000068
Among them, C is a constant to prevent numerical errors when the denominator tends to 0,
Figure BDA0002896760540000062
Represent the weighted average of the red, green and blue channels of the HDR image, and the calculation method is
Figure BDA0002896760540000063
In the formula, x represents the weighted average value of the red/green/blue channel, x i represents the i-th normalized pixel value in the red/green/blue channel, d i is the distance between the i-th background pixel and the foreground;
Figure BDA0002896760540000064
respectively represent the average value of the red, green and blue channels of the foreground video,
Figure BDA0002896760540000065
for
Figure BDA0002896760540000066
for
Figure BDA0002896760540000067
for
Figure BDA0002896760540000068

在一个实施例中,步骤S1包括:In one embodiment, step S1 includes:

获得三维极坐标系中的全景视频信号;Obtain the panoramic video signal in the three-dimensional polar coordinate system;

获得前景视频在全景视频中的前景附着点,构成附着点集;Obtain the foreground attachment point of the foreground video in the panoramic video to form an attachment point set;

通过坐标变换映射将前景视频信号及其掩膜信号变换至三维极坐标上,获得掩膜信号中所有前景视频信号组成的前景信号点集;transforming the foreground video signal and its mask signal to three-dimensional polar coordinates through coordinate transformation mapping, and obtaining a foreground signal point set composed of all foreground video signals in the mask signal;

通过阈值判断方法获得前景信号点集中的底部前景信号点集;Obtaining the bottom foreground signal point set in the foreground signal point set by a threshold judgment method;

根据底部前景信号点集获得前景视频信号的定位点;Obtain the anchor point of the foreground video signal according to the bottom foreground signal point set;

随机选取附着点集中的前景附着点,根据前景附着点和定位点的距离,将前景视频信号在三维极坐标中进行旋转,使得旋转后的前景视频信号的定位点同前景附着点重合;Randomly select the foreground attachment point in the attachment point set, and rotate the foreground video signal in three-dimensional polar coordinates according to the distance between the foreground attachment point and the anchor point, so that the anchor point of the rotated foreground video signal coincides with the foreground attachment point;

根据三维极坐标系中拍摄装置的高度和前景附着点获得映射半径;Obtain the mapping radius according to the height of the shooting device and the attachment point of the foreground in the three-dimensional polar coordinate system;

将旋转后的三维极坐标下的前景视频信号缩放至以映射半径为半径的三维球体内表面上。Scale the rotated foreground video signal in the three-dimensional polar coordinates to the inner surface of the three-dimensional sphere whose radius is the mapping radius.

在一个实施例中,所述通过坐标变换映射将前景视频信号及其掩膜信号变换至三维极坐标上的步骤包括:In one embodiment, the step of transforming the foreground video signal and its mask signal to three-dimensional polar coordinates through coordinate transformation mapping includes:

将前景视频信号及其掩膜信号的二维图像平面坐标转换至经纬度坐标;Converting the two-dimensional image plane coordinates of the foreground video signal and its mask signal to longitude and latitude coordinates;

将经纬度坐标以全景视频信号分布平面的半径叠加至三维极坐标中。The latitude and longitude coordinates are superimposed into the three-dimensional polar coordinates with the radius of the panoramic video signal distribution plane.

在一个实施例中,步骤S3包括:In one embodiment, step S3 includes:

在三维球体内合成位置自适应调整和光照自适应调整后的掩膜信号、前景视频信号和全景视频信号。The mask signal, the foreground video signal and the panoramic video signal after position adaptive adjustment and illumination adaptive adjustment are synthesized in the three-dimensional sphere.

在一个实施例中,智能化的全景视频合成方法基于HDR照度采集和UE4实现,包含光照一致性和位置一致性调整两部分,具体如下:In one embodiment, the intelligent panoramic video synthesis method is based on HDR illumination collection and UE4 implementation, including two parts of illumination consistency and position consistency adjustment, as follows:

步骤S10,前景/背景图像位置一致性校准,也就是说前景视频在全景视频中的合成位置进行自适应调整,详细地:Step S10, foreground/background image position consistency calibration, that is to say, the composite position of the foreground video in the panoramic video is adaptively adjusted, in detail:

现有全景视频叠加系统主要在二维图像空间中进行叠加操作,即直接从全景视频中选择部分区域进行鱼眼变换后所得到的二维图像作为背景视频,再与二维前景图像进行叠加。然而,上述叠加方法无法有效记录并保持前景物体同背景视频的透视关系,观看视角变化显著时前景视频与背景视频会有明显的位置及映射关系不一致。考虑以上问题,本发明直接在三维全景图像空间中进行视频叠加,从而在合成前景物体背景信号的同时有效保持其透视关系、并维持其位置一致性,具体地,包括:The existing panoramic video superimposition system mainly performs the superimposition operation in the two-dimensional image space, that is, the two-dimensional image obtained by directly selecting a part of the panoramic video for fisheye transformation is used as the background video, and then superimposed with the two-dimensional foreground image. However, the above superimposition method cannot effectively record and maintain the perspective relationship between the foreground object and the background video. When the viewing angle changes significantly, the foreground video and the background video will have obvious position and mapping inconsistencies. Considering the above problems, the present invention directly performs video superimposition in the three-dimensional panoramic image space, thereby effectively maintaining the perspective relationship and maintaining the consistency of its position while synthesizing the background signal of the foreground object, specifically, including:

在三维极坐标系中的全景视频信号为Bs,全景视频信号分布在半径为r0的平面上,即

Figure BDA0002896760540000071
有效(因为全景信号分布在半径为r0的平面上,所以只用当r=r0时才有值,r为其他值时都为0,有效可以指有值)当且仅当r=r0。其中,坐标r是空间中一点M到原点O的距离;
Figure BDA0002896760540000072
是通过z轴和点M的半平面与坐标面zOx所构成的角;θ是线段OM与z轴正方向的夹角。在全景视频中通过用户兴趣选择若干(K个)标记点作为前景视频附着点集
Figure BDA0002896760540000073
Figure BDA0002896760540000074
可以通过用户自己选择附着点,也可以通过用户客户端存储的图片获得用户喜欢的图像的风格,选择符合所述风格的附着点。其中,pi为点集中的第i个点,每个点的半径
Figure BDA0002896760540000075
均为r0;The panoramic video signal in the three-dimensional polar coordinate system is B s , and the panoramic video signal is distributed on a plane with a radius r 0 , namely
Figure BDA0002896760540000071
Effective (because the panoramic signal is distributed on a plane with a radius of r 0 , it only has a value when r=r 0 , and it is 0 when r is other values, valid can refer to a value) if and only if r=r 0 . Among them, the coordinate r is the distance from a point M to the origin O in space;
Figure BDA0002896760540000072
is the angle formed by the half plane passing through the z-axis and point M and the coordinate plane zOx; θ is the angle between the line segment OM and the positive direction of the z-axis. Select several (K) marker points as foreground video attachment point set according to user interest in panoramic video
Figure BDA0002896760540000073
Figure BDA0002896760540000074
The attachment point can be selected by the user, or the style of the image that the user likes can be obtained from the picture stored in the user client, and the attachment point that conforms to the style can be selected. Among them, p i is the i-th point in the point set, and the radius of each point
Figure BDA0002896760540000075
Both are r 0 ;

二维前景信号为F,并记前景信号对应的掩膜信号为M,通过坐标变换映射g(·)分别将F与M变换至三维极坐标上,Fs、Ms分别是三维极坐标中的前景信号和掩膜信号:The two-dimensional foreground signal is F, and the mask signal corresponding to the foreground signal is M, respectively transform F and M to three-dimensional polar coordinates through the coordinate transformation map g(·), F s and M s are the three-dimensional polar coordinates The foreground signal and mask signal of:

Fs=g(F),Ms=g(M)F s =g(F),M s =g(M)

其中,坐标变换映射g(·)的步骤包括:先将二维图像平面坐标转换至经纬度坐标,最后将经纬度坐标以半径r=r0贴图至三维极坐标中。Wherein, the step of coordinate transformation mapping g(·) includes: first converting the two-dimensional image plane coordinates to latitude and longitude coordinates, and finally mapping the latitude and longitude coordinates to three-dimensional polar coordinates with radius r= r0 .

对于三维极坐标下掩膜信号中所有前景信号点集

Figure BDA0002896760540000076
其垂直角坐标θ的下确界记为
Figure BDA0002896760540000077
Figure BDA0002896760540000078
inf是infimum的简称,表示集合最大的下界,位于底部的前景信号点集
Figure BDA0002896760540000079
可通过
Figure BDA00028967605400000710
且|θ-θmin|<TH)}获取,其中TH表示底部前景信号点的判断阈值,即所有水平角度同θminmin为垂直角坐标θ的下确界)差距小于TH的前景信号点均判断为底部前景信号,TH的根据下式获得:
Figure BDA00028967605400000711
sup是supremum的简称,表示集合最小的上界;For all foreground signal point sets in the mask signal in 3D polar coordinates
Figure BDA0002896760540000076
The infimum of its vertical angular coordinate θ is denoted as
Figure BDA0002896760540000077
Figure BDA0002896760540000078
inf is the abbreviation of infimum, indicating the largest lower bound of the set, the foreground signal point set at the bottom
Figure BDA0002896760540000079
accessible
Figure BDA00028967605400000710
And |θ-θ min |<TH)} is obtained, where TH represents the judgment threshold of the bottom foreground signal point, that is, the difference between all horizontal angles and θ minmin is the infimum of the vertical angle coordinate θ) is less than TH All points are judged as the bottom foreground signal, and TH is obtained according to the following formula:
Figure BDA00028967605400000711
sup is the abbreviation of supremum, which means the smallest upper bound of the set;

根据前景信号点集

Figure BDA00028967605400000712
获得前景信号定位点
Figure BDA00028967605400000713
Figure BDA00028967605400000714
其中
Figure BDA00028967605400000715
Figure BDA00028967605400000716
According to the foreground signal point set
Figure BDA00028967605400000712
Get the foreground signal anchor point
Figure BDA00028967605400000713
Figure BDA00028967605400000714
in
Figure BDA00028967605400000715
Figure BDA00028967605400000716

随机选取前景视频的附着点集PB中一点作为前景附着点pB,即pB∈PB,根据pB和前景视频信号定位点pB的距离

Figure BDA00028967605400000717
Figure BDA00028967605400000718
将前景信号在三维极坐标中进行旋转,使得旋转后前景信号定位点pF同背景附着点pB重合。其中dθ,
Figure BDA0002896760540000081
分别为pB和pF两点在θ,
Figure BDA0002896760540000082
坐标上的距离。
Figure BDA0002896760540000083
为点pB在θ,
Figure BDA0002896760540000084
坐标上的值。Randomly select a point in the attachment point set P B of the foreground video as the foreground attachment point p B , that is, p B ∈ P B , and locate the point p B according to the distance between p B and the foreground video signal
Figure BDA00028967605400000717
Figure BDA00028967605400000718
The foreground signal is rotated in three-dimensional polar coordinates, so that the foreground signal positioning point p F coincides with the background attachment point p B after rotation. where d θ ,
Figure BDA0002896760540000081
Respectively p B and p F two points in θ,
Figure BDA0002896760540000082
distance in coordinates.
Figure BDA0002896760540000083
For point p B at θ,
Figure BDA0002896760540000084
The value on the coordinate.

记三维坐标系中摄像机的高度为h,附着点pB位于r=r0的球面上,即

Figure BDA0002896760540000085
根据附着点的垂直角坐标
Figure BDA0002896760540000086
判断其映射半径
Figure BDA0002896760540000087
Figure BDA0002896760540000088
然后将三维极坐标下前景信号Fs放缩至以rm为半径的三维球体内表面上。Note that the height of the camera in the three-dimensional coordinate system is h, and the attachment point p B is located on the spherical surface of r=r 0 , namely
Figure BDA0002896760540000085
According to the vertical corner coordinates of the attachment point
Figure BDA0002896760540000086
Determine its mapping radius
Figure BDA0002896760540000087
Figure BDA0002896760540000088
Then the foreground signal F s in the three-dimensional polar coordinates is scaled to the inner surface of the three-dimensional sphere with r m as the radius.

步骤S20,HDR图像照度采集和场景照明,也就是说,对前景视频的光照信息根据全景视频的光照信息进行自适应调整,详细地:Step S20, HDR image illumination acquisition and scene illumination, that is to say, adaptively adjust the illumination information of the foreground video according to the illumination information of the panoramic video, in detail:

前景视频和全景视频在不同时间、空间下拍摄,光照差别较大,如果直接合成会影响画面真实感。本发明通过采集全景视频的光照信息,对前景视频的色调进行自适应调整,具体包括:The foreground video and panoramic video are shot at different times and spaces, and the lighting is quite different. If they are directly synthesized, the realism of the picture will be affected. The present invention adaptively adjusts the color tone of the foreground video by collecting the illumination information of the panoramic video, specifically including:

利用逆色调映射方法将拍摄得到的普通动态范围(1000:1)全景视频映射至高动态范围(100000:1),Use the inverse tone mapping method to map the captured general dynamic range (1000:1) panoramic video to high dynamic range (100000:1),

Figure BDA0002896760540000089
Figure BDA0002896760540000089

其中

Figure BDA00028967605400000810
in
Figure BDA00028967605400000810

在UE4中利用天空光照部件对逆色调映射得到的HDR图像所包含的光照信息进行采集,主要包含其整体亮度和色调等,利用天空光照部件采集得到的光照信息在三维模型中生成环境光。In UE4, the sky lighting component is used to collect the lighting information contained in the HDR image obtained by inverse tone mapping, mainly including its overall brightness and hue, etc., and the lighting information collected by the sky lighting component is used to generate ambient light in the 3D model.

设置前景视频显示部件的材质为非金属材质,粗糙度为1,使其可以反射光照信息,基于物理渲染后前景视频即可反射含有全景HDR图像照度信息的光线,和背景视频具有相似的亮度和色调前景和背景的相似度S计算方法为:

Figure BDA00028967605400000811
Set the material of the foreground video display part to non-metallic material, with a roughness of 1, so that it can reflect light information. Based on physical rendering, the foreground video can reflect the light containing the illuminance information of the panoramic HDR image, and the background video has similar brightness and The calculation method of the similarity S of the hue foreground and background is:
Figure BDA00028967605400000811

步骤S30,根据三维坐标系下掩膜信号Ms,将前景信号Fs和背景信号Bs进行合成:Step S30, synthesize the foreground signal F s and the background signal B s according to the mask signal M s in the three-dimensional coordinate system:

Figure BDA00028967605400000812
Figure BDA00028967605400000812

其中,CS为合成图像,

Figure BDA00028967605400000813
为放缩至以rm为半径的三维球体内表面上的前景图像。Among them, C S is the synthetic image,
Figure BDA00028967605400000813
is the foreground image scaled to the inner surface of the three-dimensional sphere with r m as the radius.

由于前景和作为背景的全景视频是在不同的环境中拍摄得到的,因此在色调分布、明暗程度等方面会存在一些差异,本发明通过搭建基于HDR图像的光照模型并进行三维空间下位置一致性校准可保证前景画面和全景视频的光照一致性和位置一致性,增加合成画面的真实感,提升用户的观看使用体验。Since the foreground and the panoramic video as the background are shot in different environments, there will be some differences in the distribution of hues, brightness, etc., the present invention builds an illumination model based on HDR images and performs position consistency in three-dimensional space Calibration can ensure the consistency of illumination and position of the foreground image and the panoramic video, increase the realism of the composite image, and improve the user's viewing experience.

如图2所示,是本发明智能化的全景视频合成系统的功能模块图。As shown in FIG. 2 , it is a functional block diagram of the intelligent panoramic video synthesis system of the present invention.

本发明所述智能化的全景视频合成系统100可以安装于电子设备中。根据实现的功能,在一个实施例中所述智能化的全景视频合成系统可以包括位置自适应调整模块10、采集模块20、光照自适应调整模块30和三维合成模块40。本发所述模块也可以称之为单元,是指一种能够被电子设备处理器所执行,并且能够完成固定功能的一系列计算机程序段,其存储在电子设备的存储器中。The intelligent panoramic video synthesis system 100 of the present invention can be installed in electronic equipment. According to the realized functions, the intelligent panoramic video synthesis system in one embodiment may include a position adaptive adjustment module 10 , an acquisition module 20 , an illumination adaptive adjustment module 30 and a three-dimensional synthesis module 40 . The module described in the present invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by the processor of the electronic device and can complete fixed functions, and are stored in the memory of the electronic device.

在本实施例中,关于各模块/单元的功能如下:In this embodiment, the functions of each module/unit are as follows:

智能化的全景视频合成系统100包括:The intelligent panoramic video synthesis system 100 includes:

位置自适应调整模块10,在全景视频的三维全景图像空间中自适应调整前景视频位置;Position adaptive adjustment module 10, adaptively adjusts the foreground video position in the three-dimensional panoramic image space of the panoramic video;

采集模块20,采集位置自适应调整模块前景视频自适应调整后的全景视频的光照信息,所述光照信息包括亮度和色调;Acquisition module 20, collecting the lighting information of the panoramic video after the foreground video adaptive adjustment of the position adaptive adjustment module, and the lighting information includes brightness and hue;

光照自适应调整模块30,对前景视频的光照信息根据采集模块采集的全景视频的光照信息进行自适应调整;The lighting adaptive adjustment module 30 is adapted to adjust the lighting information of the foreground video according to the lighting information of the panoramic video collected by the acquisition module;

三维合成模块40,在全景视频的三维全景图像空间中叠加经过光照自适应调整模块调整后的前景视频。The 3D synthesis module 40 superimposes the foreground video adjusted by the illumination adaptive adjustment module in the 3D panoramic image space of the panoramic video.

在一个实施例中,所述采集模块20包括:In one embodiment, the collection module 20 includes:

映射单元,利用逆色调映射方法提高全景视频光照信息的动态范围,得到HDR图像;A mapping unit that utilizes an inverse tone mapping method to improve the dynamic range of the panoramic video lighting information to obtain an HDR image;

信息采集单元,采集逆色调映射得到的HDR图像所包含的光照信息;an information collection unit, for collecting illumination information contained in the HDR image obtained by inverse tone mapping;

其中,所述自适应调整模块调整前景视频的光照信息,使得前景视频和HDR图像的光照信息相似。Wherein, the adaptive adjustment module adjusts the illumination information of the foreground video, so that the illumination information of the foreground video and the HDR image are similar.

在一个实施例中,所述位置自适应调整模块10包括:In one embodiment, the position adaptive adjustment module 10 includes:

第一采集单元,获得三维极坐标系中的全景视频信号;The first acquisition unit obtains the panoramic video signal in the three-dimensional polar coordinate system;

附着点集构建单元,获得前景视频在全景视频中的前景附着点,构成附着点集;The attachment point set construction unit is used to obtain the foreground attachment point of the foreground video in the panoramic video to form the attachment point set;

前景信号点集构建单元,通过坐标变换映射将前景视频信号及其掩膜信号变换至三维极坐标上,获得掩膜信号中所有前景视频信号组成的前景信号点集;The foreground signal point set construction unit transforms the foreground video signal and its mask signal to three-dimensional polar coordinates through coordinate transformation mapping, and obtains a foreground signal point set composed of all foreground video signals in the mask signal;

底部前景信号点集构建单元,通过阈值判断方法获得前景信号点集中的底部前景信号点集;The bottom foreground signal point set construction unit obtains the bottom foreground signal point set in the foreground signal point set by a threshold judgment method;

定位点获得单元,根据底部前景信号点集获得前景视频信号的定位点;The positioning point obtaining unit obtains the positioning point of the foreground video signal according to the bottom foreground signal point set;

旋转单元,随机选取附着点集中的前景附着点,根据前景附着点和定位点的距离,将前景视频信号在三维极坐标中进行旋转,使得旋转后的前景视频信号的定位点同前景附着点重合;The rotation unit randomly selects the foreground attachment point in the attachment point set, and rotates the foreground video signal in three-dimensional polar coordinates according to the distance between the foreground attachment point and the anchor point, so that the anchor point of the rotated foreground video signal coincides with the foreground attachment point ;

映射半径获得单元,根据三维极坐标系中拍摄装置的高度和前景附着点获得映射半径;The mapping radius obtaining unit obtains the mapping radius according to the height of the shooting device in the three-dimensional polar coordinate system and the attachment point of the foreground;

缩放单元,将旋转后的三维极坐标下的前景视频信号缩放至以映射半径为半径的三维球体内表面上;A scaling unit, which scales the rotated foreground video signal in three-dimensional polar coordinates to the inner surface of a three-dimensional sphere with the mapping radius as the radius;

其中,三维合成模块40三维球体内合成掩膜信号、前景视频信号和全景视频信号。Wherein, the three-dimensional synthesis module 40 synthesizes the mask signal, the foreground video signal and the panoramic video signal within the three-dimensional sphere.

本发明智能化的全景视频合成系统通过面向三维全景图像空间的前景/背景合成方法优化了合成图像的位置一致性;使用照度采集和场景照明实现了前景和背景的光照一致性;。The intelligent panoramic video synthesis system of the present invention optimizes the positional consistency of the synthesized image through the foreground/background synthesis method oriented to the three-dimensional panoramic image space; realizes the illumination consistency of the foreground and the background by using illuminance collection and scene lighting;

如图3所示,是本发明实现智能化的全景视频合成方法的电子设备的结构示意图。As shown in FIG. 3 , it is a schematic structural diagram of an electronic device implementing an intelligent panoramic video synthesis method according to the present invention.

所述电子设备200可以包括存储器210、处理器220和总线,还可以包括存储在所述存储器210中并可在所述处理器220上运行的计算机程序,如全景视频合成程序211。The electronic device 200 may include a memory 210 , a processor 220 and a bus, and may also include a computer program stored in the memory 210 and operable on the processor 220 , such as a panoramic video synthesis program 211 .

其中,所述存储器210至少包括一种类型的可读存储介质,所述可读存储介质包括闪存、移动硬盘、多媒体卡、卡型存储器(例如:SD或DX存储器等)、磁性存储器、磁盘、光盘等。所述存储器210在一些实施例中可以是电子设备200的内部存储单元,例如该电子设备200的移动硬盘。所述存储器210在另一些实施例中也可以是电子设备200的外部存储设备,例如电子设备200上配备的插接式移动硬盘、智能存储卡(Smart Media Card,SMC)、安全数字(Secure Digital,SD)卡、闪存卡(Flash Card)等。进一步地,所述存储器210还可以既包括电子设备200的内部存储单元也包括外部存储设备。所述存储器210不仅可以用于存储安装于电子设备200的应用软件及各类数据,例如全景视频合成程序的代码等,还可以用于暂时地存储已经输出或者将要输出的数据。Wherein, the memory 210 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, magnetic disk, CD etc. The storage 210 may be an internal storage unit of the electronic device 200 in some embodiments, such as a mobile hard disk of the electronic device 200 . The memory 210 may also be an external storage device of the electronic device 200 in other embodiments, such as a plug-in mobile hard disk equipped on the electronic device 200, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital , SD) card, flash memory card (Flash Card), etc. Further, the memory 210 may also include both an internal storage unit of the electronic device 200 and an external storage device. The memory 210 can not only be used to store application software and various data installed in the electronic device 200 , such as the code of the panoramic video synthesis program, but also can be used to temporarily store outputted or to-be-outputted data.

所述处理器220在一些实施例中可以由集成电路组成,例如可以由单个封装的集成电路所组成,也可以是由多个相同功能或不同功能封装的集成电路所组成,包括一个或者多个中央处理器(Central Processing unit,CPU)、微处理器、数字处理芯片、图形处理器及各种控制芯片的组合等。所述处理器220是所述电子设备的控制核心(Control Unit),利用各种接口和线路连接整个电子设备的各个部件,通过运行或执行存储在所述存储器210内的程序或者模块(例如全景视频合成程序等),以及调用存储在所述存储器210内的数据,以执行电子设备200的各种功能和处理数据。In some embodiments, the processor 220 may be composed of an integrated circuit, for example, may be composed of a single packaged integrated circuit, or may be composed of multiple integrated circuits with the same function or different functions, including one or more Combination of central processing unit (Central Processing unit, CPU), microprocessor, digital processing chip, graphics processor and various control chips, etc. The processor 220 is the control core (Control Unit) of the electronic device, which uses various interfaces and lines to connect the various components of the entire electronic device, and runs or executes programs or modules stored in the memory 210 (for example, video synthesis program, etc.), and call the data stored in the memory 210 to execute various functions of the electronic device 200 and process data.

所述总线可以是外设部件互连标准(peripheral component interconnect,简称PCI)总线或扩展工业标准结构(extended industry standard architecture,简称EISA)总线等。该总线可以分为地址总线、数据总线、控制总线等。所述总线被设置为实现所述存储器210以及至少一个处理器220等之间的连接通信。The bus may be a peripheral component interconnect (PCI for short) bus or an extended industry standard architecture (EISA for short) bus or the like. The bus can be divided into address bus, data bus, control bus and so on. The bus is configured to implement communication between the memory 210 and at least one processor 220 and the like.

图3仅示出了具有部件的电子设备,本领域技术人员可以理解的是,图3示出的结构并不构成对所述电子设备200的限定,可以包括比图示更少或者更多的部件,或者组合某些部件,或者不同的部件布置。FIG. 3 only shows an electronic device with components. Those skilled in the art can understand that the structure shown in FIG. 3 does not constitute a limitation to the electronic device 200, and may include fewer or more components, or combinations of certain components, or different arrangements of components.

例如,尽管未示出,所述电子设备200还可以包括给各个部件供电的电源(比如电池),优选地,电源可以通过电源管理装置与所述至少一个处理器220逻辑相连,从而通过电源管理装置实现充电管理、放电管理、以及功耗管理等功能。电源还可以包括一个或一个以上的直流或交流电源、再充电装置、电源故障检测电路、电源转换器或者逆变器、电源状态指示器等任意组件。所述电子设备200还可以包括多种传感器、蓝牙模块、Wi-Fi模块等,在此不再赘述。For example, although not shown, the electronic device 200 may also include a power supply (such as a battery) for supplying power to each component. Preferably, the power supply may be logically connected to the at least one processor 220 through a power management device, so that through power management The device implements functions such as charge management, discharge management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components. The electronic device 200 may also include various sensors, a Bluetooth module, a Wi-Fi module, etc., which will not be repeated here.

进一步地,所述电子设备200还可以包括网络接口,可选地,所述网络接口可以包括有线接口和/或无线接口(如WI-FI接口、蓝牙接口等),通常用于在该电子设备200与其他电子设备之间建立通信连接。Further, the electronic device 200 may also include a network interface. Optionally, the network interface may include a wired interface and/or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which are usually used in the electronic device 200 establishes a communication connection with other electronic devices.

可选地,该电子设备200还可以包括用户接口,用户接口可以是显示器(Display)、输入单元(比如键盘(Keyboard)),可选地,用户接口还可以是标准的有线接口、无线接口。可选地,在一些实施例中,显示器可以是LED显示器、液晶显示器、触控式液晶显示器以及OLED(Organic Light-Emitting Diode,有机发光二极管)触摸器等。其中,显示器也可以适当的称为显示屏或显示单元,用于显示在电子设备200中处理的信息以及用于显示可视化的用户界面。Optionally, the electronic device 200 may further include a user interface. The user interface may be a display (Display) or an input unit (such as a keyboard (Keyboard)). Optionally, the user interface may also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode, Organic Light-Emitting Diode) touch panel, and the like. Wherein, the display may also be appropriately referred to as a display screen or a display unit, and is used for displaying information processed in the electronic device 200 and for displaying a visualized user interface.

应该了解,所述实施例仅为说明之用,在专利申请范围上并不受此结构的限制。It should be understood that the embodiments are only for illustration, and are not limited by the structure in terms of the scope of the patent application.

所述电子设备200中的所述存储器210存储的基于图像和类别的注意力的目标检测程序211是多个指令的组合,在所述处理器220中运行时,可以实现:The target detection program 211 based on image and category attention stored in the memory 210 of the electronic device 200 is a combination of multiple instructions. When running in the processor 220, it can realize:

在全景视频的三维全景图像空间中自适应调整前景视频位置;Adaptively adjust the position of the foreground video in the three-dimensional panoramic image space of the panoramic video;

采集全景视频的光照信息,对前景视频的光照信息进行自适应调整,所述光照信息包括亮度和色调;Collecting the lighting information of the panoramic video, and adaptively adjusting the lighting information of the foreground video, the lighting information including brightness and hue;

将自适应调整后的前景视频叠加在所述全景视频中。The adaptively adjusted foreground video is superimposed on the panoramic video.

具体地,所述处理器220对上述指令的具体实现方法可参考图1对应实施例中相关步骤的描述,在此不赘述。Specifically, for a specific implementation method of the above instructions by the processor 220, reference may be made to the description of relevant steps in the embodiment corresponding to FIG. 1 , and details are not repeated here.

进一步地,所述电子设备200集成的模块/单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时,可以存储在一个计算机可读取存储介质中。所述计算机可读介质可以包括:能够携带所述计算机程序代码的任何实体或装置、记录介质、U盘、移动硬盘、磁碟、光盘、计算机存储器、只读存储器(ROM,Read-Only Memory)。Further, if the integrated modules/units of the electronic device 200 are realized in the form of software function units and sold or used as independent products, they may be stored in a computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM, Read-Only Memory) .

此外,本申请实施例还提出一种计算机可读存储介质,所述计算机可读存储介质可以是非易失性,也可以是易失性,计算机可读存储介质中包括计算机程序,该计算机程序被处理器执行时实现如下操作:In addition, the embodiment of the present application also proposes a computer-readable storage medium. The computer-readable storage medium may be non-volatile or volatile. The computer-readable storage medium includes a computer program, and the computer program is When the processor is executed, the following operations are implemented:

在全景视频的三维全景图像空间中自适应调整前景视频位置;Adaptively adjust the position of the foreground video in the three-dimensional panoramic image space of the panoramic video;

采集全景视频的光照信息,对前景视频的光照信息进行自适应调整,所述光照信息包括亮度和色调;Collecting the lighting information of the panoramic video, and adaptively adjusting the lighting information of the foreground video, the lighting information including brightness and hue;

将自适应调整后的前景视频叠加在所述全景视频中。The adaptively adjusted foreground video is superimposed on the panoramic video.

本发明之计算机可读存储介质的具体实施方式与上述智能化的全景视频合成方法、装置、电子设备的具体实施方式大致相同,在此不再赘述。The specific implementations of the computer-readable storage medium of the present invention are substantially the same as the specific implementations of the above-mentioned intelligent panoramic video synthesis method, device, and electronic equipment, and will not be repeated here.

本发明智能化的全景视频合成方法、装置、电子设备利用逆色调映射方法提升全景视频的动态范围,并对其进行照度采集后作为场景照明,使前景视频可以反射背景光照,从而实现了前景、背景的光照一致性;为保持合成过程中前景/背景视频的透视关系,通过面向三维全景图像空间的前景/背景合成方法在整个合成过程中维持并记录前景/背景透视关系。The intelligent panoramic video synthesis method, device, and electronic equipment of the present invention use the inverse tone mapping method to improve the dynamic range of the panoramic video, and collect the illuminance and use it as scene lighting, so that the foreground video can reflect the background light, thereby realizing the foreground, The illumination consistency of the background; in order to maintain the perspective relationship of the foreground/background video during the synthesis process, the foreground/background perspective relationship is maintained and recorded throughout the synthesis process through the foreground/background synthesis method oriented to the three-dimensional panoramic image space.

在上述各实施例中,给出先对背景视频进行位置自适应调整然后再进行光照自适应调整的实施例,但是本发明并不限于此,可以先进行光照自适应调整,再进行位置自适应调整,先进行位置自适应调整相对于先进行光照自适应调整的优点在于,可以避免因为位置不同对光照的影响。In each of the above-mentioned embodiments, an embodiment is given in which the position adaptive adjustment is performed on the background video first, and then the illumination adaptive adjustment is performed, but the present invention is not limited thereto, and the illumination adaptive adjustment can be performed first, and then the position adaptive adjustment is performed , the advantage of performing position adaptive adjustment first compared to first performing light adaptive adjustment is that it can avoid the influence of different positions on the light.

在本发明所提供的几个实施例中,应该理解到,所揭露的设备,装置和方法,可以通过其它的方式实现。例如,以上所描述的装置实施例仅仅是示意性的,例如,所述模块的划分,仅仅为一种逻辑功能划分,实际实现时可以有另外的划分方式。In the several embodiments provided by the present invention, it should be understood that the disclosed devices, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

所述作为分离部件说明的模块可以是或者也可以不是物理上分开的,作为模块显示的部件可以是或者也可以不是物理单元,即可以位于一个地方,或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案的目的。The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

另外,在本发明各个实施例中的各功能模块可以集成在一个处理单元中,也可以是各个单元单独物理存在,也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现,也可以采用硬件加软件功能模块的形式实现。In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, or each unit may physically exist separately, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware, or in the form of hardware plus software function modules.

对于本领域技术人员而言,显然本发明不限于上述示范性实施例的细节,而且在不背离本发明的精神或基本特征的情况下,能够以其他的具体形式实现本发明。It will be apparent to those skilled in the art that the invention is not limited to the details of the above-described exemplary embodiments, but that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention.

因此,无论从哪一点来看,均应将实施例看作是示范性的,而且是非限制性的,本发明的范围由所附权利要求而不是上述说明限定,因此旨在将落在权利要求的等同要件的含义和范围内的所有变化涵括在本发明内。不应将权利要求中的任何附关联图标记视为限制所涉及的权利要求。Accordingly, the embodiments should be regarded in all points of view as exemplary and not restrictive, the scope of the invention being defined by the appended claims rather than the foregoing description, and it is therefore intended that the scope of the invention be defined by the appended claims rather than by the foregoing description. All changes within the meaning and range of equivalents of the elements are embraced in the present invention. Any reference sign in a claim should not be construed as limiting the claim concerned.

此外,显然“包括”一词不排除其他单元或步骤,单数不排除复数。系统权利要求中陈述的多个单元或装置也可以由一个单元或装置通过软件或者硬件来实现。第二等词语用来表示名称,而并不表示任何特定的顺序。In addition, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or devices stated in the system claims may also be realized by one unit or device through software or hardware. Secondary terms are used to denote names without implying any particular order.

最后应说明的是,以上实施例仅用以说明本发明的技术方案而非限制,尽管参照较佳实施例对本发明进行了详细说明,本领域的普通技术人员应当理解,可以对本发明的技术方案进行修改或等同替换,而不脱离本发明技术方案的精神和范围。Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention without limitation. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be Modifications or equivalent replacements can be made without departing from the spirit and scope of the technical solutions of the present invention.

Claims (6)

1.一种智能化的全景视频合成方法,其特征在于,包括:1. an intelligent panoramic video synthesis method, characterized in that, comprising: 在全景视频的三维全景图像空间中自适应调整前景视频位置;Adaptively adjust the position of the foreground video in the three-dimensional panoramic image space of the panoramic video; 采集全景视频的光照信息,对前景视频的光照信息进行自适应调整,所述光照信息包括亮度和色调;Collecting the lighting information of the panoramic video, and adaptively adjusting the lighting information of the foreground video, the lighting information including brightness and hue; 将自适应调整后的前景视频叠加在所述全景视频中,superimposing the adaptively adjusted foreground video on the panoramic video, 所述采集全景视频的光照信息,对前景视频的色调进行自适应调整的步骤包括:The step of collecting the lighting information of the panoramic video and adaptively adjusting the hue of the foreground video includes: 利用逆色调映射方法提高全景视频光照信息的动态范围,得到HDR图像;Using the inverse tone mapping method to improve the dynamic range of the lighting information of the panoramic video, and obtain the HDR image; 采集逆色调映射得到的HDR图像所包含的光照信息;Collect the lighting information contained in the HDR image obtained by inverse tone mapping; 调整前景视频的光照信息,使得前景视频和HDR图像的光照信息相似,Adjust the lighting information of the foreground video so that the lighting information of the foreground video and the HDR image are similar, 所述在全景视频的三维全景图像空间中自适应调整前景视频位置的步骤包括:The step of adaptively adjusting the position of the foreground video in the three-dimensional panoramic image space of the panoramic video includes: 获得三维极坐标系中的全景视频信号;Obtain the panoramic video signal in the three-dimensional polar coordinate system; 获得前景视频在全景视频中的前景附着点,构成附着点集;Obtain the foreground attachment point of the foreground video in the panoramic video to form an attachment point set; 通过坐标变换映射将前景视频信号及其掩膜信号变换至三维极坐标上,获得掩膜信号中所有前景视频信号组成的前景信号点集;transforming the foreground video signal and its mask signal to three-dimensional polar coordinates through coordinate transformation mapping, and obtaining a foreground signal point set composed of all foreground video signals in the mask signal; 通过阈值判断方法获得前景信号点集中的底部前景信号点集;Obtaining the bottom foreground signal point set in the foreground signal point set by a threshold judgment method; 根据底部前景信号点集获得前景视频信号的定位点;Obtain the anchor point of the foreground video signal according to the bottom foreground signal point set; 随机选取附着点集中的前景附着点,根据前景附着点和定位点的距离,将前景视频信号在三维极坐标中进行旋转,使得旋转后的前景视频信号的定位点同前景附着点重合;Randomly select the foreground attachment point in the attachment point set, and rotate the foreground video signal in three-dimensional polar coordinates according to the distance between the foreground attachment point and the anchor point, so that the anchor point of the rotated foreground video signal coincides with the foreground attachment point; 根据三维极坐标系中拍摄装置的高度和前景附着点获得映射半径;Obtain the mapping radius according to the height of the shooting device and the attachment point of the foreground in the three-dimensional polar coordinate system; 将旋转后的三维极坐标下的前景视频信号缩放至以映射半径为半径的三维球体内表面上;Scaling the rotated foreground video signal in three-dimensional polar coordinates to the inner surface of a three-dimensional sphere whose radius is the mapping radius; 其中,所述将自适应调整后的前景视频叠加在所述全景视频中的步骤包括:Wherein, the step of superimposing the adaptively adjusted foreground video on the panoramic video includes: 在三维球体内合成掩膜信号、前景视频信号和全景视频信号。A mask signal, a foreground video signal and a panoramic video signal are synthesized within a three-dimensional sphere. 2.根据权利要求1所述的智能化的全景视频合成方法,其特征在于,所述采集全景视频的光照信息,对前景视频的色调进行自适应调整的步骤还包括:2. the intelligent panoramic video composition method according to claim 1, is characterized in that, the illumination information of described collection panoramic video, the step that the color tone of foreground video is carried out self-adaptive adjustment also comprises: 采集得到的光照信息在三维模型中生成环境光。The collected lighting information generates ambient light in the 3D model. 3.根据权利要求1所述的智能化的全景视频合成方法,其特征在于,所述利用逆色调映射方法提高全景视频光照信息的动态范围的步骤包括:3. the intelligent panorama video composition method according to claim 1, is characterized in that, the described step that utilizes inverse tone mapping method to improve the dynamic range of panorama video illumination information comprises: 根据下式将普通动态范围的全景视频映射至高动态范围The panoramic video of normal dynamic range is mapped to high dynamic range according to the following formula
Figure FDA0003813208400000011
Figure FDA0003813208400000011
其中,
Figure FDA0003813208400000021
R为扩展阈值,大于R的亮度值都会被扩展,Iw,max为扩展后的最大亮度,α为控制色调曲线伸展的衰减指数,Ld(x)为原亮度,Lw(x)为扩展后的亮度。
in,
Figure FDA0003813208400000021
R is the expansion threshold, the brightness value greater than R will be expanded, I w,max is the maximum brightness after expansion, α is the attenuation index that controls the extension of the tone curve, L d (x) is the original brightness, L w (x) is Extended brightness.
4.一种智能化的全景视频合成系统,其特征在于,包括:4. an intelligent panoramic video synthesis system, is characterized in that, comprising: 位置自适应调整模块,在全景视频的三维全景图像空间中自适应调整前景视频位置;A position adaptive adjustment module, adaptively adjusts the position of the foreground video in the three-dimensional panoramic image space of the panoramic video; 采集模块,采集位置自适应调整模块前景视频自适应调整后的全景视频的光照信息,所述光照信息包括亮度和色调;The acquisition module collects the lighting information of the panoramic video after the position adaptive adjustment module foreground video adaptive adjustment, and the lighting information includes brightness and hue; 光照自适应调整模块,对前景视频的光照信息根据采集模块采集的全景视频的光照信息进行自适应调整;The illumination adaptive adjustment module is adapted to adjust the illumination information of the foreground video according to the illumination information of the panoramic video collected by the acquisition module; 三维合成模块,在全景视频的三维全景图像空间中叠加经过光照自适应调整模块调整后的前景视频,所述采集模块包括:The three-dimensional synthesis module superimposes the foreground video adjusted by the illumination adaptive adjustment module in the three-dimensional panoramic image space of the panoramic video, and the acquisition module includes: 映射单元,利用逆色调映射方法提高全景视频光照信息的动态范围,得到HDR图像;A mapping unit that utilizes an inverse tone mapping method to improve the dynamic range of the panoramic video lighting information to obtain an HDR image; 信息采集单元,采集逆色调映射得到的HDR图像所包含的光照信息;an information collection unit, for collecting illumination information contained in the HDR image obtained by inverse tone mapping; 其中,所述光照自适应调整模块调整前景视频的光照信息,使得前景视频和HDR图像的光照信息相似,所述位置自适应调整模块包括:Wherein, the illumination adaptive adjustment module adjusts the illumination information of the foreground video, so that the illumination information of the foreground video and the HDR image are similar, and the position adaptive adjustment module includes: 第一采集单元,获得三维极坐标系中的全景视频信号;The first acquisition unit obtains the panoramic video signal in the three-dimensional polar coordinate system; 附着点集构建单元,获得前景视频在全景视频中的前景附着点,构成附着点集;The attachment point set construction unit is used to obtain the foreground attachment point of the foreground video in the panoramic video to form the attachment point set; 前景信号点集构建单元,通过坐标变换映射将前景视频信号及其掩膜信号变换至三维极坐标上,获得掩膜信号中所有前景视频信号组成的前景信号点集;The foreground signal point set construction unit transforms the foreground video signal and its mask signal to three-dimensional polar coordinates through coordinate transformation mapping, and obtains a foreground signal point set composed of all foreground video signals in the mask signal; 底部前景信号点集构建单元,通过阈值判断方法获得前景信号点集中的底部前景信号点集;The bottom foreground signal point set construction unit obtains the bottom foreground signal point set in the foreground signal point set by a threshold judgment method; 定位点获得单元,根据底部前景信号点集获得前景视频信号的定位点;The positioning point obtaining unit obtains the positioning point of the foreground video signal according to the bottom foreground signal point set; 旋转单元,随机选取附着点集中的前景附着点,根据前景附着点和定位点的距离,将前景视频信号在三维极坐标中进行旋转,使得旋转后的前景视频信号的定位点同前景附着点重合;The rotation unit randomly selects the foreground attachment point in the attachment point set, and rotates the foreground video signal in three-dimensional polar coordinates according to the distance between the foreground attachment point and the anchor point, so that the anchor point of the rotated foreground video signal coincides with the foreground attachment point ; 映射半径获得单元,根据三维极坐标系中拍摄装置的高度和前景附着点获得映射半径;The mapping radius obtaining unit obtains the mapping radius according to the height of the shooting device in the three-dimensional polar coordinate system and the attachment point of the foreground; 缩放单元,将旋转后的三维极坐标下的前景视频信号缩放至以映射半径为半径的三维球体内表面上;A scaling unit, which scales the rotated foreground video signal in three-dimensional polar coordinates to the inner surface of a three-dimensional sphere with the mapping radius as the radius; 其中,三维合成模块在三维球体内合成掩膜信号、前景视频信号和全景视频信号。Wherein, the three-dimensional synthesis module synthesizes the mask signal, the foreground video signal and the panoramic video signal in the three-dimensional sphere. 5.一种电子设备,其特征在于,所述电子设备包括:5. An electronic device, characterized in that the electronic device comprises: 至少一个处理器;以及,at least one processor; and, 与所述至少一个处理器通信连接的存储器;其中,a memory communicatively coupled to the at least one processor; wherein, 所述存储器存储有可被所述至少一个处理器执行的指令,所述指令被所述至少一个处理器执行,以使所述至少一个处理器能够执行如权利要求1至3中任一所述的智能化的全景视频合成方法。The memory is stored with instructions executable by the at least one processor, the instructions are executed by the at least one processor, so that the at least one processor can perform any one of claims 1 to 3 Intelligent panoramic video synthesis method. 6.一种计算机可读存储介质,存储有计算机程序,其特征在于,所述计算机程序被处理器执行时实现如权利要求1至3中任一所述的智能化的全景视频合成方法。6. A computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, the intelligent panoramic video synthesis method according to any one of claims 1 to 3 is realized.
CN202110043474.4A 2021-01-13 2021-01-13 Intelligent panoramic video synthesis method and system, electronic device and medium Active CN112866507B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110043474.4A CN112866507B (en) 2021-01-13 2021-01-13 Intelligent panoramic video synthesis method and system, electronic device and medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110043474.4A CN112866507B (en) 2021-01-13 2021-01-13 Intelligent panoramic video synthesis method and system, electronic device and medium

Publications (2)

Publication Number Publication Date
CN112866507A CN112866507A (en) 2021-05-28
CN112866507B true CN112866507B (en) 2023-01-10

Family

ID=76003456

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110043474.4A Active CN112866507B (en) 2021-01-13 2021-01-13 Intelligent panoramic video synthesis method and system, electronic device and medium

Country Status (1)

Country Link
CN (1) CN112866507B (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114003331A (en) * 2021-11-10 2022-02-01 浙江博采传媒有限公司 LED circular screen virtual reality synthesis method and device, storage medium and electronic equipment
CN119135850A (en) * 2024-11-12 2024-12-13 航天极创物联网研究院(南京)有限公司 Image quality optimization method and system for panoramic video fusion video

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007140834A (en) * 2005-11-17 2007-06-07 System Keikaku Kenkyusho:Kk Program, information storage medium, photograph printer and photograph printing method
CN108027961A (en) * 2015-08-31 2018-05-11 汤姆逊许可公司 Method and apparatus for inverse tone mapping (ITM)

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9883155B2 (en) * 2016-06-14 2018-01-30 Personify, Inc. Methods and systems for combining foreground video and background video using chromatic matching

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007140834A (en) * 2005-11-17 2007-06-07 System Keikaku Kenkyusho:Kk Program, information storage medium, photograph printer and photograph printing method
CN108027961A (en) * 2015-08-31 2018-05-11 汤姆逊许可公司 Method and apparatus for inverse tone mapping (ITM)

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
马尔科夫随机场化的光照一致图像合成方法;黄美玉;《计算机辅助设计与图形学学报》;20150430;全文 *

Also Published As

Publication number Publication date
CN112866507A (en) 2021-05-28

Similar Documents

Publication Publication Date Title
US10977764B2 (en) Viewport independent image coding and rendering
JP7007348B2 (en) Image processing equipment
JP2006053694A (en) Space simulator, space simulation method, space simulation program, recording medium
US20230186550A1 (en) Optimizing generation of a virtual scene for use in a virtual display environment
JP2022188059A (en) Method and device for compositing image
CN114125310B (en) Photographing method, terminal device and cloud server
CN101422035A (en) Image high-resolution upgrading device, image high-resolution upgrading method, image high-resolution upgrading program and image high-resolution upgrading system
CN112262413B (en) Real-time compositing in mixed reality
US10719920B2 (en) Environment map generation and hole filling
CN110992484B (en) Display method of traffic dynamic video in real scene three-dimensional platform
WO2023207452A1 (en) Virtual reality-based video generation method and apparatus, device, and medium
US11276150B2 (en) Environment map generation and hole filling
CN112866507B (en) Intelligent panoramic video synthesis method and system, electronic device and medium
EP3057316B1 (en) Generation of three-dimensional imagery to supplement existing content
CN110870304B (en) Method and apparatus for providing information to a user for viewing multi-view content
CN109427089B (en) Mixed reality object rendering based on ambient lighting conditions
US20210125399A1 (en) Three-dimensional video processing
WO2022022260A1 (en) Image style transfer method and apparatus therefor
US10902669B2 (en) Method for estimating light for augmented reality and electronic device thereof
JP7447403B2 (en) Information processing device, information processing system, information processing method and program
CN117541478B (en) Image processing method and related device
WO2024042893A1 (en) Information processing device, information processing method, and program
US20240406338A1 (en) Information processing device, video processing method, and program
CN116245741B (en) Image processing method and related device
CN119342155A (en) Virtual shooting method, device, equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant