BG67302B1 - Interactive system of computer assisted contact-free trying on of clothes and accessories by adapting the augmented reality toolbox and using depth camera system - Google Patents

Abstract

The main application of the proposed interactive computer assisted contact-free trying on of clothing and accessories by adapting the augmented reality toolbox and using a depth camera system is fundamental, but not limited to fashion, commerce, advertising and entertainment. The proposed system allows advertising of targeted products, including by way of customized advertising that allows the prospective customer to see what they would look like in the advertised item offered, as well as in a combination of one or more advertised and / or available items, as based on artificial intelligence the system offers the customer the best size, clothing model and colour corresponding to the specifics of their figure and desire. The proposed system also allows the purchase of the item liked by the customer , as well as its payment and delivery, including without intervention of an employee of the retailer and regardless of the time of selection and purchase, as well as the manufacturer or supplier.

Description

Field of technology

Multimedia interactive device

BACKGROUND OF THE INVENTION

Attempts to create computer-based systems to support customer choice when buying clothes and accessories marked its development with the massification of computer technology, and especially after the advent of color CRT monitors and graphics operating systems. Through the process of supporting the customer's choice, retailers ensure an increase in sales volumes and a reduction in returned goods, and customers receive a higher degree of satisfaction with their purchase.

The main approaches for the realization of computer-assisted choice of customers when buying clothes and accessories, and in particular in the part of visualization of the selected product and how it would fit the specific customer, are mainly aimed at:

1. Visualization of a photo of a garment (with or without a mannequin) (Sources. Http://www2.hm.com/bg bg / index.html, http://www.explosi.bg/)

2. Visualization of a photograph of a garment superimposed as a still position and combined with a video stream from a camera or a photograph of the face, in which the client "adjusts" on the photograph to see what it will look like (Source: https://www.youtube. com / watch? v = 5TZmQPdhpak, http://zugara.com/virtual-dressing-room- technology / virtual-dressing-room-kiosk, https://www.youtube.com/watch?v=L9EwtGUTNzO)

3. Upload a custom photo, where the system morphs body parts to replace the head and visible parts of an existing photo with a mannequin

4. Visualization of a clothing model with an option for viewing / rotating by the client (Source: https://www.magictoolbox.com/magic360/)

5. Building a typical computer-generated avatar, which "dresses" with computer-generated clothes - entirely in a computer environment (Source: https://www.yotube.com/watch7v—ZDJsZla9SYQ, https://www.voutube.com/ watch? v = ^: mWO InW-Zvs)

6. Building a customized (based on body type) computer-generated avatar, which "dresses in computer-generated clothes - entirely in a computer environment (source: http://www.trimirror.com/en/about/)

7. Building a customized (with actually measured or entered customer data) computer-generated avatar, which "dresses" with computer-generated clothes - entirely in a computer environment (Source: https://www.youtube.com/watch?v= DMladTwgHoI)

8. Generating and superimposing a computer-generated garment on a video stream from a camera with an image of the client based on the silhouette of the client's body (in a controlled environment) (Source: https://www.youtube.com/watch?v=BD4TCfECxsU&feature ^:: = voutu.be)

BG 67302 Bl

9. Generating and superimposing a computer-generated garment on a video stream from a camera with an image of the client based on a virtual skeleton of the client, obtained from depth cameras (in a controlled environment) (Source: https://www.youtube.com/watch?v= libvnklT4vQ)

10. Generating and superimposing a computer-generated garment on a video stream from a camera with an image of the client based on the silhouette of the client's body (in an uncontrolled environment) (Source: https://www.youtube.com/watch?v=u a-jFzQFgQ )

11. Generating and superimposing a computer-generated garment on a video stream from a client's image based on a client's virtual skeleton obtained from depth cameras (in an uncontrolled environment) (source: https://www.youtube.com/watch?v= gz5ynrfJSDM, https: //www.youtube.com/watch? v = IB2iutoT4Eo, https://www.voutube.com/watch7v—h3vP3hdwf6I).

And although based on these approaches there are a number of commercial solutions, at the moment none of them solves the problem with the realism of the depicted garment, so that the client has the feeling that he sees himself with a real garment on himself.

The main problems of the existing solutions are related to the fact that the client cannot be seen from all sides (solutions related to approach 1, 2 and 3), the simulation leaves a feeling for a computer game and not for a real try on clothes, solutions related to approach 4, 5, 6 and 7), or the presence of a significant discrepancy between the customer's movement and the computer-generated garment (solutions related to approaches 8,9,10 and 11), as well as the lack of realism of the garment's behavior, according to the specifics of the specific figure of the client (solutions related to approach 1, 2, 3, 4, 5, 8, 9, 10 and 11).

Another key aspect of the issue is related to the reporting of the actual physical data of the client, in view of the correct visualization of the computer-generated garment or accessory, incl. and to determine the appropriate size garment or accessory. The main approaches for measuring the physical measurement data of the client are related to the use of:

1. Physical means for assisted or independent contact measurement

2. Means for calculating estimated dimensions based on a two-dimensional photographic image (photo)

3. Means for non-contact measurement of body dimensions by creating a digital model

Despite the advancement of the technology for contactless measurements, the most accurate remain the so-called contact measurements, as the alternative methods are characterized by a certain level of mathematical approximation, extrapolation and modeling, as well as by the presence of measurement errors due to the specific technology used.

Regarding the measurement by TA modeling, the main approaches are related to the use of:

1. Photogrammetric methods based on high quality photographic images

2. Creating a TA depth map using one or more depth sensors (type ΚΟΒΩ, ToF or other applicable technology).

In order to achieve high accuracy of the created ZD model, these approaches require the use of multiple sensors or cameras, performing a series of multiple images processed with appropriate algorithms to create relatively accurate ZD models. This approach usually requires a certain period of time and does not allow the creation and tracking of the human health model in real time.

BG 67302 Bl

Technical essence of the invention

The presented interactive system for computer-assisted contactless testing of clothes and accessories by adapting the tools of augmented reality and using a system of depth cameras solves a number of technical problems facing long-term workers in the field and helps to meet long-standing needs. approaches and solutions related to the construction of computer-assisted systems for contactless testing of clothes and accessories.

Although in its complexity there is no similar solution to the offered interactive system for computer-assisted contactless testing of clothes and accessories by adapting the tools of augmented reality and using a system of depth cameras, the closest solutions and components related to individual aspects of the system are as follows:

1. Regarding the approach for visualization of the selected product and how it would stand for the specific customer, as part of the systems for realization of computer-assisted selection of customers when buying clothes and accessories, the closest solutions are related to generating and superimposing a computer-generated garment on a video stream from a camera with an image of the client based on a virtual skeleton of the client obtained from depth cameras (in an uncontrolled environment). The main problem with existing solutions is that they mainly use stand-alone depth sensors (such as MS Kinect or similar), which are designed and optimized for the purpose of computer game consoles (eg MS Xbox) and are therefore characterized by a specific range of action (0.80 m - 4.00 m. Source: https://blogs.msdn.microsoft.com/ kinectforwindows / 2012/01/20 / peagmode-what-it-is-and-isnt /), limited angle of capturing objects (43 ° vertically and 57 ° horizontally source: https://msdn.microsoft.com/en-us/librarv/iil31033.aspx), which reflects extremely high values of measurement error in the Z coordinate with increasing distance (source. Maimone, A., Fuchs, F. Encumbrance-free telepresence system with real-time 3D capture and display using commodity depth cameras. https://www.researchgate.net/publication/221221196 Encumbrance-free tele-presence system_with realtime 3D_capture_and_display_using_commodity_depth_cameras Fig. 5). It should be noted that although in practice there are solutions that use depth cameras located above the floor (ceiling), these systems are again characterized by shooting in a single plane, and their use is aimed at security activities in closed spaces (source: Gasparrini, S., Cippitelli, E., Spinsante, S., Gambi, E. A Depth-Based Fall Detection System Using a Kinect® Sensor. https://www.ncbi.nlm.nih.gov/ pmc / articles / PMC3958279 /).

In this regard, the proposed interactive system for computer-assisted contactless testing of clothing and accessories by adapting the tools of augmented reality and using a system of depth cameras, introduces a new approach combining a system of depth cameras, at least one of which is located above the floor (on the ceiling), which allows shooting of at least two planes, one of which is necessarily a top view and serves as a compensatory for the Z coordinate of the frontally located depth chamber. The adoption of such an approach allows the application of a specific compensatory

BG 67302 Bl approach, which allows through a system of specific parameters to compensate for errors in the measurement of depth cameras in one of the dimensions (mainly Z), which in turn allows the construction of an accurate ZD model and a virtual skeleton of the subject.

2. Regarding the non-contact measurement of the body, the closest solution to the offered interactive system for computer-assisted contactless testing of clothes and accessories by adapting the tools of augmented reality and using a system of depth cameras is the use of means for non-contact measurement of the body. body dimensions by creating a TA digital model through the methods of TA scanning. This approach is usually characterized by the use of a large number of optical (source: http://www.aniwaa.com/3d-bodv-scanning/) and depth cameras (source: https://www.artec3d.com/3d: scanner / shapifybooth), laser measurements (source: https://ww.youtube.com/watch?v= AzBlzxUgrg) or using a combination of sensors and a rotating platform (source: http://www.styku.com/). In addition, the proposed approach requires time (min. 10 s) to create the model and perform the measurements (source: https://www.artec3d.com/3d-scanner/shapifvbooth).

In this regard, the proposed interactive system for computer-assisted contactless testing of clothing and accessories by adapting the tools of augmented reality and using a system of depth cameras, introduces a new approach, building on the approach adopted by Microsoft to create a virtual skeleton (source: https: //msdn.microsoft.com/en-us/library/iil31025.aspx) by using a system of depth cameras, at least one of which is located above the floor (ceiling) eliminates errors in depth measurement and subsequent generation of the so-called "Virtual skeleton". Unlike similar solutions for non-contact measurement, the construction of a virtual skeleton using a system of depth cameras, at least one of which is located above the floor (ceiling) allows its generation and tracking in real time (up to 30 frames in second - source: https://i.stack.imgur.com/OQiI4.pnghttps://stackoverflow.com/questions/12978415/all-frames-from-kinect-at-30fps), which in turn allows the implementation of anthropometric measurement of established anthropometric ratios between key points in the skeletal structure and key points of the physical body, taking into account the contour measurements of the depth chambers along the X and Y axes.

The technical essence of the offered interactive system for computer-assisted contactless testing of clothes and accessories by adapting the tools of augmented reality and using a system of depth cameras, is locked in the visualization of computer-generated virtual clothing superimposed on incoming video stream as high resolution layer of digital information is created using adapted tools of augmented reality and based on a customized (with non-contact measured customer data) computer-generated invisible avatar, which "dresses" with computer-generated clothing, taking into account the specifics of matter and its behavior, such as fitting of computer-generated layers, incl. and augmented reality is based on a unified virtual skeleton obtained from multiple depth chambers, at least one of which is located above the floor (the virtual skeleton also serves as a marker for augmented reality).

The interactive system for computer-assisted contactless testing of clothes and accessories by adapting the tools of augmented reality and using a system of depth cameras for

BG 67302 Bl computer-assisted contactless testing of clothing and accessories by adapting augmented reality tools and using a system of depth cameras, includes at least: a high-quality video camera 3 placed at the level of human vision and capturing space in front of the interactive installation connected through a computer system 5 with a device for visualization 6 of the captured space in front of the interactive installation, while maintaining the real ratio between the components of the captured objects, a system of at least two depth cameras, the first camera 2 being located near the camera for high-quality video recording 3 of the space in front of the interactive installation and is directed in the same shooting direction as it, and the second camera 1 is positioned so as to allow capturing a target area from above, as the system of depth cameras through specialized hardware and / or software module 4 is connected with a computer system 5 for processing the incoming streams of information from the cameras, rendering the TT model of selected or selected by the customer items, combining them into a single video stream for visualization of the connected visualization device 5, in real time or not more than 0.033 s delay . The system of at least two depth cameras, at least one of them (element 2) is located near the camera for high-quality video camera (element 3) of the space in front of the interactive installation and is directed in the same shooting direction as it, and at least one of them (element 1) is located so as to allow the capture of a target area with a view from above, as the incoming information from the depth cameras (elements 1 and 2) enters a specialized hardware and / or software module (element 4), in which by applying a specially developed algorithm, the real positions of key points of the human anatomy are calculated by eliminating the errors in the measurement with the depth chambers, and on this basis the module (element 4) allows non-contact measurement of various visually visible parts of the human anatomy, including for determining customer measurement data. These key points in human anatomy are also used as specialized markers to coordinate and visualize computer-generated objects, including, but not limited to, augmented, virtual, mixed, and other types of realities created using local, distributed devices. or remote computing power, regardless of the type of computing device (CPU, GPU or other applicable), the technology of use (microprocessor, nanotechnology or other applicable) and the architecture of the computing device itself (RISC, CISC or other applicable).

At the same time, the offered interactive system for computer-assisted contactless testing of clothes and accessories by adapting the tools of augmented reality and using a system of depth cameras, allows a combination of multilayer video streams and computer-generated images, incl. and transparent ones, allowing the visualization of the realistic behavior of selected clothes and accessories on the real figure of the particular user, which is realized by creating a customized transparent computer avatar, by adapting a basic avatar based on statistical anthropometric information, by contactless form reading and key dimensions of the real human body, incl. and key measurement data, using the component of the interactive installation presented in claim 2.

BG 67302 Bl

Explanation of the attached figure

Figure 1 presents the relationship between the individual elements of the interactive system for computer-assisted contactless testing of clothing and accessories by adapting the augmented reality tools and using a system of depth cameras.

Examples of the invention

The offered interactive system for computer-assisted contactless testing of clothes and accessories by adapting the tools of augmented reality and using a system of depth cameras is a complex component system that includes at least: a camera for high quality video recording 3, placed on the level of human vision and capturing the space in front of the interactive installation connected via a computer system 5 to a device for visualizing 6 the captured space in front of the interactive installation, while maintaining the actual ratio between the components of the captured objects, a system of at least two depth cameras 2 is located close to the high-quality video camera 3 of the space in front of the interactive installation and is directed in the same shooting direction as it, and the second camera 1 is located so as to allow shooting of a target area from above, as the system of depth and cameras through a specialized hardware and / or software module 4 is connected to a computer system 5 for processing the incoming streams of information from the cameras, rendering the TD model of selected or customer-selected items, combining them into a single video stream for visualization of the connected visualization device 5, in real time or with a delay of not more than 0.033 s.

Application of the invention

The offered interactive system for computer-assisted contactless testing of clothes and accessories by adapting the tools of augmented reality and using a system of depth cameras can be built mainly, but not only in the form of a stand-alone installation, and its individual components can be built both in the main body of the installation and in the room in which it is located. The use of standard components with general purpose and / or specialized application, which are manufactured by more than one manufacturer and are offered for free retail, allow the smooth construction and operation in an uncontrolled environment of the offered interactive system for computer-assisted contactless testing of clothing and accessories by adapting the tools of augmented reality and using a system of depth cameras.

The main application of the offered interactive system for computer-assisted contactless testing of clothes and accessories by adapting the tools of augmented reality and using a system of depth cameras is basic, but not only in the field of fashion, trade, advertising and entertainment. The offered system can be located as part of a store network or in the form of a free-standing installation outside a physical store, as its main purpose is to advertise certain target products, incl. and by the customized advertising method, allowing the potential customer to see what it would look like in the respective advertised item, as well as in a combination of one or more advertised and / or available items, the visualization being performed by a real-time photorealistic image. Based on the available artificial intelligence, the system offers the customer the best size of clothing, corresponding to the specifics of his figure, and also allows visualization of how he would look when choosing a different size from the items offered. The offered system also allows the purchase of an item liked by the consumer, as well as its payment and delivery, incl. and without the intervention of an employee of the commercial site and regardless of the time of the selection and purchase.

Claims (1)

Interactive system for computer-assisted contactless testing of clothing and accessories by adapting the tools of augmented reality and using a system of depth cameras, characterized in that it includes at least: a camera for high-quality video recording (3), placed at the level of human vision and capturing the space in front of the interactive installation connected through a computer system (5) with a device for visualization (6) of the captured space in front of the interactive installation, while maintaining the real ratio between the components of the captured objects cameras, the first camera (2) being located close to the high-quality video camera (3) of the space in front of the interactive installation and directed in the same shooting direction as it, and the second camera (1) being positioned to allow shooting a target area with a view from above, as the system of depth cameras through the spa A hardware and / or software module (4) is connected to a computer system (5) for processing the incoming information flows from the cameras, rendering the 3D model of selected or customer-selected items, combining them into a single video stream for visualization of the connected device. visualization (5), in real time or with a delay of not more than 0.033 s