WO2024101874A1

WO2024101874A1 - System and method for supporting service for converting real-time data into 3d object in 3d virtual reality space and selectively fusing same

Info

Publication number: WO2024101874A1
Application number: PCT/KR2023/017834
Authority: WO
Inventors: 박승하; 최은규; 홍충기
Original assignee: 주식회사 안지온
Priority date: 2022-11-08
Filing date: 2023-11-08
Publication date: 2024-05-16

Abstract

A method for supporting a service for converting real-time data into a 3D object in a 3D virtual reality space and selectively fusing same according to an embodiment of the present invention comprises the steps of: receiving image data from a CCTV camera and preprocessing same; extracting the position of a real object in the image data; generating a digital twin space reflecting geographic information of a point where the CCTV camera is located; converting the real object into a 3D object; matching real (absolute) coordinates in the image data with spatial coordinates in the digital twin space; and synchronizing and reflecting the position change of the 3D object in the digital twin space in real time.

Description

A system and method for supporting services for selectively fusion processing by converting real-time data into 3D objects in 3D virtual reality space

The present invention relates to a system and method that supports services for converting real-time data into 3D objects in a 3D virtual reality space and selectively converging them.

Existing augmented reality (AR) technology has been focused on research centered on entertainment. Efforts are continuing to collect and analyze a variety of large-scale data generated in cities to study urban spatial issues and build smart cities.

One example of such an attempt is to collect information and set up the Internet of Things (IoT) in a wide range of cities to transform various urban phenomena into useful data.

As a way to visualize the information collected in this way, data visualization using augmented reality will be provided for various city services.

Meanwhile, despite efforts to disseminate smart services, the development of augmented reality visualization tools for city managers is still insufficient. The reason it is difficult for smart city researchers and city service designers to utilize existing IoT data is because they do not have a system in place to collect and transmit data in real time, or even if an IoT system is in place, it is difficult to obtain permission to access IoT data from the management of other organizations. am.

In addition, due to the increasing demand for smart cities, IoT sensors are not available at the time of producing city digital twins, resulting in unnecessary time spent on development. Accordingly, there is a need to research IoT event generation systems that will serve as virtual signal generation, data transmission, and storage instead of hardware IoT sensors.

In particular, a simulation method using virtual IoT is required to minimize the waste of time and material costs that may arise depending on the utilization of IoT when building an actual IoT system in the digital twin development stage for initial smart city service design. .

[Prior art literature]

[Non-patent literature]

Nguyen D., Meixner G.: Comparison User Engagement of Gamified and Non-gamified Augmented Reality Assembly Training Advances in Agile and User-Centred Software Engineering. pp. 142-152. Springer International Publishing (2020)

The purpose of the present invention is to provide a system and method that supports services for selectively fusion processing by converting real-time data into 3D objects in a 3D virtual reality space that can solve conventional problems.

A system that supports a service for converting real-time data into 3D objects in a 3D virtual reality space and selectively converging them according to an embodiment of the present invention to solve the above problem is a pre-processing system that receives video data from CCTV cameras and pre-processes them. wealth; an object extraction unit that extracts a live-action object in the image data and the location coordinates of the live-action object; A digital twin space creation unit that generates a digital twin space reflecting geographical information of the point where the CCTV camera is located; a 3D object conversion unit that converts the photorealistic object into a 3D object; A coordinate matching unit that matches real (absolute) coordinates in the image data with spatial coordinates of the digital twin space; and a data registration unit that projects the 3D object by synchronizing the positional change of the 3D object in real time within the digital twin space.

In one embodiment, the apparatus further includes an object characteristic value granting unit for assigning object characteristic values including at least one of the type, shape, size, color, movement coordinate, and departure coordinate of the photorealistic object to the 3D object.

In one embodiment, the device further includes a tracking unit that tracks the movement line of a 3D object moving within the digital twin space, and the 3D object is a moving object.

A method of supporting a service for converting real-time data into 3D objects in a 3D virtual reality space and selectively converging them according to an embodiment of the present invention to solve the above problem includes the steps of receiving video data from a CCTV camera and pre-processing it. ; Extracting a live-action object and location coordinates of the live-action object from the image data; Creating a digital twin space reflecting geographical information of the point where the CCTV camera is located; converting the photorealistic object into a 3D object; Matching absolute coordinates of real space within the image data with spatial coordinates of the digital twin space; and projecting the 3D object by synchronizing the positional change of the 3D object in real time within the digital twin space.

In one embodiment, the step of assigning object characteristic values including at least one of the type, shape, size, color, movement coordinate, and departure coordinate of the real-life object to the 3D object in an object characteristic value granting unit. It is characterized by

In one embodiment, the method further includes tracking the movement line of the 3D object moving within the digital twin space in a tracking unit.

According to a system and method for supporting a service for converting real-time data into 3D objects in a 3D virtual reality space according to an embodiment of the present invention and selectively fusion processing, the basic state of the video for privacy infringement issues of CCTV is anonymized. It provides the advantage of realizing the generalization of three-dimensional tracking technology combined with 3D models by changing the status to that of anonymity and selectively releasing anonymity for exceptional situations such as criminal vehicles, missing persons, wanted criminals, traffic accidents, and emergency situations.

Figure 1 is a network configuration diagram of a system supporting a service for selectively fusion processing by converting real-time data into 3D objects in a 3D virtual reality space according to an embodiment of the present invention.

Figure 2 is a hierarchical diagram structuring GIS data and BIM data.

Figure 3 is an example of synchronization between a live object in a CCTV image and a 3D object in a CCTV digital twin image.

Figure 4 is a flowchart illustrating a method of supporting a service for selectively fusion processing by converting real-time data into 3D objects in a 3D virtual reality space according to an embodiment of the present invention.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and will be implemented in various different forms. The present embodiments only serve to ensure that the disclosure of the present invention is complete and are within the scope of common knowledge in the technical field to which the present invention pertains. It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims.

The terminology used herein is for describing embodiments and is not intended to limit the invention. As used herein, singular forms also include plural forms, unless specifically stated otherwise in the context. As used herein, “comprises” and/or “comprising” refers to the presence of one or more other components, steps, operations and/or elements. or does not rule out addition.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings that can be commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless clearly specifically defined.

Preferred embodiments of the present invention will be described in more detail. The same reference numerals are used for the same components in the drawings, and duplicate descriptions for the same components are omitted.

Hereinafter, based on the attached drawings, a system and method for supporting a service for selectively fusion processing by converting real-time data into 3D objects in a 3D virtual reality space according to an embodiment of the present invention will be described in more detail. .

First, the service for selectively converging and converting real-time data into 3D objects in a 3D virtual reality space according to an embodiment of the present invention is a service that provides information about the real world when implementing a virtual reality (digital twin) space running on a smart device. As a means of matching the direction between the direction and the virtual world, geographic points (markers) are used to match the azimuth of the real world and the azimuth of the virtual world.

After matching the azimuth of real space and the azimuth of virtual space, convert real-time moving objects in real space into 3D objects and then project them into virtual space, that is, overlapping 3D objects for each video frame in real space, but retaining individual information. This is a service for tracking virtual and real real-time movements while selectively solving privacy issues through convergence.

For reference, the service for selectively converging and converting real-time data into 3D objects in the 3D virtual reality space presented in the present invention can be applied to smart cities, and smart cities solve urban problems and improve the quality of life at the same time. As an urban management system that can be improved, cases of smart city introduction are increasing around the world in various fields such as energy, transportation, and healthcare. In addition to the continued interest in smart cities, the development of corresponding technologies is required to solve various urban problems, and based on the vast amount of data collected, digital twin technology is used for efficient operation of smart cities. This is in the spotlight.

In order to build a digital twin at the city level for general-purpose smart city services, a GIS (Geographic Information System) containing geographic information from a macroscopic perspective and buildings from a microscopic perspective are used. /BIM (Building Information Modeling), which contains detailed information of urban facilities, is needed, and therefore, great efforts are being made to integrate them and utilize them as an urban management system.

Therefore, through the service proposed in the present invention, the basic state of the video regarding CCTV privacy infringement issues is changed to an anonymized state, and selectively for exceptional situations such as stolen vehicles, missing persons, wanted criminals, traffic accidents, and emergency situations. The key point of the present invention, which will be described later, is to commercialize three-dimensional tracking technology combined with 3D models by removing anonymity.

Figure 1 is a network configuration diagram of a system supporting a service for selectively fusion processing by converting real-time data into 3D objects in a 3D virtual reality space according to an embodiment of the present invention, and Figure 2 is a network configuration diagram of GIS data and BIM data It is a structured hierarchy, and Figure 3 is an example of synchronization between a live object in a CCTV image and a 3D object in a CCTV digital twin image.

Referring to FIGS. 1 to 3, a system 100 that supports a service for converting real-time data into 3D objects and selectively fusion processing in a 3D virtual reality space according to an embodiment of the present invention includes an image pre-processing unit 110, It includes an object extraction unit 120, a digital twin space creation unit 130, a 3D object conversion unit 140, a spatial coordinate matching unit 150, and an image data matching unit 160.

The image pre-processing unit 110 may be configured to receive CCTV image data captured by a CCTV camera 10, which is an intelligent camera, and pre-process it into a perspective view with a specific viewing angle.

In addition, the image preprocessing unit 110 can preprocess CCTV image data to support functions beyond simple streaming image data, and the present invention can create a 6 degrees of freedom (6 DoF) virtual environment using multiple 360 image data. .

The object extraction unit 120 may be configured to extract a live object in CCTV image data and the location coordinates (absolute coordinates) of the live object.

The object extraction unit 120 extracts pixel coordinates for the location of the object in the image data using an instance segmentation algorithm, and is constructed based on a street view and a 3D model. Image depth can be estimated using camera posture and ground information in the virtual space of the digital twin. Afterwards, the initial position, which is the 3D position of the object, can be estimated using the pixel coordinates of the object and the depth value of the corresponding pixel.

The digital twin space creation unit 130 creates a digital twin space reflecting the geographical information of the point where the CCTV camera 10 is located.

Here, the digital twin space generator 130 creates a virtual space based on three-dimensional information about a specific space by replicating the corresponding space using a digital twin technique.

For example, three-dimensional information about a specific space may be any one of a blueprint for the space, 3D scanned data, a floor plan, or data generated by actually measuring the specific space.

The digital twin space generator 130 can classify GIS (Geographic Information System) data and BIM (Building Information Modeling) data, and GIS data provided at the national level contains various information according to LOD (Level of Detail). It contains them. All GIS data contains various attribute information such as shape information from a macroscopic perspective, building site address, use, name, height, number of floors, and area.

The digital twin space creation unit 130 uses the digital twin technique to create a virtual space with a view angle taken from the point where the CCTV camera 10 is located.

In addition, the digital twin space generator 130 calculates the difference with the object in the image data using a 3D model (3D model reconstruction) placed in the digital twin based on the initial position, and minimizes the difference. Through optimization, geographic data including the location and angle of an object can be estimated and the estimated geographic data can be implemented as a digital twin space.

In addition, the PSO (Particle Swarm Optimization) algorithm is used to calculate the difference between the object outline of the estimated data generated by projecting the 3D object of the 3D model to the initial position and the object outline in the image data. , optimization can be performed to minimize errors.

In addition, the optimization process is performed through iterations that change the position and rotation of the 3D object (Position/Rotation Refinement) using the PSO algorithm, resulting in 6DoF including the position and angle of the object in the image data. The pose (6DoF pose) can be estimated (final position/rotation) and implemented in a digital twin space.

The digital twin referred to in the present invention is used in various systems and industries such as manufacturing, medicine, transportation, aerospace, and construction, and uses standardized data to generate IoT data, 3D objects, and 3D maps. It may be formed based on (3DGeographic Map)

For reference, Digital Twin technology uses a digital twin model that maintains the identity and consistency of an actual physical model or an actual sensor information model that can be dynamically analyzed in real time, so that future phenomena such as an actual physical model or an actual sensor information model can be used. It is known as a technology that can predict and control the actual physical model or actual sensor information model.

The digital twin space creation unit 130 can provide digital representations of architectural and environmental objects using building information modeling (BIM) and geographic information systems (GIS).

For example, BIM deals with building information and focuses on the microscopic representation of the building itself. GIS deals with spatial information and provides a macroscopic representation of the external environment of a building.

Spatial information for 3D modeling can be largely divided into indoor spatial information and outdoor spatial information. In general, indoor spatial information can be expressed as BIM data, and outdoor spatial information can be expressed as GIS data.

Specifically, BIM data, like GIS data, has shape information and attribute information, and contains detailed and diverse information for each building element from a microscopic perspective. Each IFC file of a building is a text-based file, and in order to visualize it, numerous string processing operations are performed at the parsing stage, requiring a long loading time.

Therefore, in order to implement a digital twin at the city level, the digital twin space creation unit 130 divides the second attribute information into categories such as building, storey, space, and element in advance. Data that can be classified and pre-classified is stored in a database and can be selectively extracted when necessary.

For reference, IFC (Industry Foundation Classes) is a standard format used in the architecture and construction industry to provide interoperability between different software applications.

GIS stores, extracts, manages, and analyzes information by linking topographic information that spatially expresses location and non-geometric attribute information that explains and supplements its form and function with graphics and database management functions.

In this case, geographic data occupying a spatial location and attribute data related to it are integrated and processed.

BIM data used in this specification may refer to all information about buildings used when implementing BIM, such as roads, bridges, and structures, and GIS data also includes street lights, crosswalks, etc., in addition to typical geographical information such as rivers, parks, etc. It can collectively refer to information including parking facilities, IOT (internet of things) facilities for measuring air pollution, CCTV, etc.

Referring to Figure 2, looking at the hierarchy that structures GIS and BIM data, it is a hierarchy that structures data from a macroscopic and microscopic aspect, and is somewhat different from the CityGML and IFC standard hierarchy. There may be. GIS and BIM information contain various overlapping information such as building name, longitude and latitude, and address. This information can be used to link GIS and BIM information, and can be selectively extracted and used as needed.

In addition, there is a variety of data in the Digital Twin, such as real-time data collected from sensors and legacy data for each service, and tables can be added to the database depending on the use case.

The 3D object conversion unit 140 is a component that converts a real-life object into a 3D object, and converts the real-life object into a 3D object of a similar shape to prevent privacy infringement on the real-life object.

Next, the spatial coordinate matching unit 150 may be configured to match real (absolute) coordinates in the image data with spatial coordinates of the digital twin space.

Next, the image data registration unit 160 may be configured to project the 3D object by synchronizing the positional change of the 3D object in real time within the digital twin space.

In addition, the system 100, which supports a service for converting real-time data into 3D objects in a 3D virtual reality space according to an embodiment of the present invention and selectively fusion processing, includes the type, shape, size, color, and It may further include an object characteristic value granting unit 170 that assigns an object characteristic value including at least one of movement coordinates and starting coordinates to the 3D object.

In addition, the system 100 that supports a service for converting real-time data into 3D objects in a 3D virtual reality space and selectively fusion processing according to an embodiment of the present invention is a 3D moving within the digital twin space. It may further include a tracking unit 180 that tracks the moving line of the object.

The tracking unit 180 may be configured to track a 3D object including object characteristic values including at least one of the type, shape, size, color, movement coordinate, and departure coordinate of the real-life object in a time-series manner.

Referring to FIG. 4, the method (S700) of supporting a service for selectively fusion processing by converting real-time data into 3D objects in a 3D virtual reality space according to an embodiment of the present invention is first, in the image pre-processing unit 110. Image data is received from the CCTV camera 10 and the CCTV image is pre-processed (S710).

Thereafter, the object extraction unit 120 extracts the real-life object and the location coordinates of the real-life object from the preprocessed image data (S720).

The object extraction unit 120 is a component that extracts the location coordinates (absolute coordinates) of the real-life object and the real-life object in the CCTV image data, and uses an instance segmentation algorithm to extract the pixel coordinates of the location of the object in the image data. can be extracted, and image depth can be estimated using camera posture and ground information in the virtual space of a digital twin built based on street view and 3D model. . Afterwards, the initial position, which is the 3D position of the object, can be estimated using the pixel coordinates of the object and the depth value of the corresponding pixel.

When the S720 process is completed, the digital twin space creation unit 130 creates a digital twin space reflecting the geographical information of the point where the CCTV camera 10 is located (S730). The digital twin space generator 130 can classify GIS (Geographic Information System) data and BIM (Building Information Modeling) data, and GIS data provided at the national level contains various information according to LOD (Level of Detail). It contains them. All GIS data contains various attribute information such as shape information from a macroscopic perspective, building site address, use, name, height, number of floors, and area.

Next, when the S730 process is completed, the 3D object conversion unit 140 converts the real-life object into a 3D object (S740), and the coordinate matching unit 150 converts the real (absolute) coordinates in the image data and the digital twin ( Match the spatial coordinates of the Digital Twin space (S750).

At this time, the coordinate matching unit 150 sets a specific or invariant geographical point in real space as absolute coordinates, and based on the absolute coordinates, the spatial coordinates of real space and the digital twin space, for example, virtual space, are set as absolute coordinates. Match spatial coordinates.

Thereafter, the image data registration unit 160 projects or registers the 3D object by real-time synchronizing the change in position of the 3D object within the digital twin space (S760).

Meanwhile, a method (S700) of supporting a service for converting real-time data into a 3D object in a 3D virtual reality space according to an embodiment of the present invention for selective fusion processing includes: the type of the photo-realistic object in an object characteristic value granting unit; The method may further include assigning object characteristic values including at least one of shape, size, color, movement coordinates, and departure coordinates to the 3D object.

In addition, a method (S700) of supporting a service for converting real-time data into 3D objects in a 3D virtual reality space and selectively fusion processing according to an embodiment of the present invention is to use the digital twin in the tracking unit 180. ) It may further include tracking the movement line of the 3D object moving in space.

Therefore, according to the system and method for supporting a service for selectively fusion processing by converting real-time data into 3D objects in a 3D virtual reality space according to an embodiment of the present invention, the basic state of the image regarding the privacy infringement issue of CCTV It provides the advantage of realizing the generalization of three-dimensional tracking technology combined with 3D models by changing it to an anonymized state and selectively de-anonymizing it in exceptional situations such as criminal vehicles, missing persons, wanted criminals, traffic accidents, and emergency situations. do.

The system or device described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, and a field programmable gate array (FPGA). , may be implemented using one or more general-purpose or special-purpose computers, such as a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

*Explanation of symbols*

100: A system that supports services for selective fusion processing by converting real-time data into 3D objects in 3D virtual reality space

110: Image pre-processing unit

120: Object extraction unit

130: 3D object conversion unit

140: Digital twin space creation unit

150: Coordinate matching unit

160: Image transparent part

170: Object characteristic value granting unit

180: tracking unit

Claims

A pre-processing unit that receives video data from CCTV cameras and pre-processes it;

an object extraction unit that extracts a live-action object in the image data and location coordinates of the live-action object;

A digital twin space creation unit that generates a digital twin space reflecting geographical information of the point where the CCTV camera is located;

a 3D object conversion unit that converts the photorealistic object into a 3D object;

A coordinate matching unit that matches real (absolute) coordinates in the image data with spatial coordinates of the digital twin space; and

Comprising a data matching unit that projects the 3D object by synchronizing the positional change of the 3D object in real time within the digital twin space,

A system that supports services for selectively fusion processing by converting real-time data into 3D objects in 3D virtual reality space.
According to paragraph 1,

Further comprising an object characteristic value granting unit for assigning an object characteristic value including at least one of the type, shape, size, color, movement coordinate, and departure coordinate of the photorealistic object to the 3D object,

A system that supports services for selectively fusion processing by converting real-time data into 3D objects in 3D virtual reality space.
According to paragraph 2,

It further includes a tracking unit that tracks the movement line of the 3D object moving within the digital twin space,

A system that supports a service for selectively fusion processing by converting real-time data into a 3D object in a 3D virtual reality space, wherein the 3D object is a moving object.
Receiving and pre-processing video data from a CCTV camera;

Extracting a live-action object and location coordinates of the live-action object from the image data;

Creating a digital twin space reflecting geographical information of the point where the CCTV camera is located;

converting the photorealistic object into a 3D object;

Matching absolute coordinates of real space within the image data with spatial coordinates of the digital twin space; and

Comprising the step of projecting the 3D object by synchronizing the positional change of the 3D object in real time within the digital twin space.

A method of supporting services for selectively fusion processing by converting real-time data into 3D objects in 3D virtual reality space.
According to clause 4,

Further comprising the step of assigning, by an object characteristic value granting unit, an object characteristic value including at least one of the type, shape, size, color, movement coordinate, and departure coordinate of the real-life object to the 3D object.

A method of converting real-time data into 3D objects in a 3D virtual reality space and supporting services for selective fusion processing.
According to clause 5,

Further comprising tracking the movement line of the 3D object moving within the digital twin space in a tracking unit,

A method of supporting services for selectively fusion processing by converting real-time data into 3D objects in 3D virtual reality space.