JP2016511850A - Method and apparatus for annotating plenoptic light fields - Google Patents

Abstract

The present invention solves and reduces problems related to existing augmented reality systems. Retrieving (100) data representing a light field using a plenoptic capture device (4); executing program code to match the retrieved data with corresponding reference data (100). 101) and executing (102) program code to retrieve at least one annotation (61, 63, 64) in a plenoptic format associated with an element of the reference data; And (103) executing program code to generate annotated data in a plenoptic format from the data and the annotation. [Selection] Figure 8

Description

  The present invention relates to an augmented reality method and apparatus, and more particularly to a method and various apparatus for annotating data corresponding to a scene.

The rapid development of handheld portable devices such as smartphones, palmtop computers, portable media players, and personal digital assistant (PDA) devices has led to the inclusion of new features and applications involving image processing. . For example, augmented reality applications are known in which when a user points a mobile device at a scene, such as a landscape, a building, a poster, or a picture in a museum, the display shows an image with superimposed information about the scene. Such information may include commercial information such as, for example, names for mountains and settlements, names of persons, historical information on buildings, and advertisements such as, for example, restaurant menus. An example of such a system is described in EP 1246080 and EP 2207113.

  It is known to provide annotation information to portable devices by a server in a wireless communication network. Annotation systems and annotation methods including communication networks with servers and portable devices are also known.

  Many annotation methods involve comparing an image, such as a 2D image or a computer generated image generated by a standard pinhole camera with a standard CCD or CMOS sensor, to a set of reference images stored in a database. including. Since the actual viewing angle and lighting conditions can be different for images stored in the database, the aim of the comparison algorithm is to remove the effects of these parameters.

  For example, International Publication No. 2008/134901 describes a method for taking a first image using a digital camera associated with a communication terminal. Query data relating to the first image is communicated over the communication network to the remote recognition server, where a matching reference image is identified. By replacing a part of the first image with at least a part of the annotated image, an extended image is generated and displayed on the communication terminal. Augmentation of the first image taken with the camera occurs in the plane space and processes only two-dimensional images and objects.

  Light ray information, such as the direction of light within each point in space, is discarded in conventional image annotation systems. Annotation without ray information makes the realistic view of the annotated scene more difficult. For example, ray information is required to capture or display a texture on the surface of an object. Each object has a different texture on its surface, but it is not possible to add texture information within the current annotation system. As a result, annotations that are not actually incorporated into the scene are attached.

  In addition, the rapid growth of augmented reality applications can cause annotation flooding in the future. For example, some scenes in cities contain many elements that are tied to different annotations, resulting in an annotated image with a large number of annotations covering a large portion of the background image. In many situations, the user is only interested in a limited number of these annotations and other annotations are simply distracting. Therefore, in many cases, it is desirable to provide a method for limiting the number of annotations and selecting the annotations to be displayed.

  In addition, computational time is a significant problem for viewing annotated scenes. It is thought that reduction of calculation time is required.

  Accordingly, it is an object of the present invention to solve or at least reduce the above-mentioned problems associated with existing augmented reality systems.

According to the present invention, these objectives are:
-Retrieving data representing a light field using a plenoptic capture device;
-Executing program code to match the captured data with the corresponding reference data;
-Executing program code to retrieve plenoptic format annotations associated with an element of the reference data;
-Executing program code to generate annotated data in a plenoptic format from the captured data and the annotations;
Achieved by a method comprising:

The present invention is also an apparatus for capturing and annotating data corresponding to a scene,
A plenoptic capture device for capturing data representing a light field;
-With a processor;
-A display;
For causing the processor to retrieve at least one annotation in a plenoptic format associated with an element of data captured using the plenoptic capture device when the program code is executed; And the program code for rendering on the display a view generated from the captured data and including the at least one annotation;
Achieved by a device comprising:

The present invention is also an apparatus for determining annotations:
-With a processor;
-A storage device;
-When the program code is executed, the processor receives data representing a light field, matches the data with one reference data, and from the storage device in the plenoptic format associated with the reference data The program code for determining an annotation and causing the remote device to transmit either the annotation in plenoptic format or the data corresponding to the annotated image in plenoptic format;
There is also provided an apparatus comprising:

  The addition of billable plenoptic format annotations allows for more realistic embedding of annotations in images in the plenoptic format. Annotations appear to be one element of the captured scene, not just text superimposed on a single image. Plenoptic format annotation (also referred to in this application as “Plenoptic Annotation”) includes a more complete description of the light field than conventional annotation, including information on how rays are modified. It is out.

  Similarly, by providing annotations in a plenoptic format, depending on the viewpoint and / or focus distance that the user selects during image rendering or is automatically selected based on, for example, user interests, An annotation to be displayed can be selected.

  Since the annotation is in the same space as the captured data (ie, plenoptic space), the computation time for the annotation process is reduced.

  Specifically, the computation time for rendering plenoptic data in a human readable format is reduced. In practice, the rendering process is similar for both because the plenoptic format image and plenoptic annotation are in the same space. In one embodiment, a single rendering process can be used to render images and associated annotations. In this case, the projection parameters selected for the plenoptic rendering process (e.g. selection of focus, depth, viewpoint change ...) also apply to plenoptic annotation. For example, when changing the focus or viewpoint of a plenoptic image, the same transformation can be used to display plenoptic annotation at various distances. In another embodiment, the annotation effect is also applied to the captured plenoptic image to render the modified plenoptic image.

  Thus, plenoptic annotation, or plenoptic format annotation, provides a realistic annotation display method, allows for more types of annotation including textured annotation, and increases computational efficiency.

  Unlike conventional annotation, plenoptic annotation may contain information about as many rays as the image captured by the plenoptic capture device. Therefore, it is possible to synthesize annotation directly in the captured light field without loss of ray information caused by projection onto the 2D image. For example, annotation can preserve light reflection properties on the surface of an annotated object, which is not possible with conventional annotation systems. In this sense, the annotated view appears to be more realistic.

  Direct modification of rays can facilitate calculations such as the simultaneous generation of annotated scenes from multiple viewpoints. In the annotated scene generation example, instead of attaching annotations on the 2D images generated for each viewpoint and applying additional processing, the annotation processing and other additional processing on the scene, such as blurring or sharpening, are performed. Applied directly once, in plenoptic format. Thus, synthesizing plenoptic images and plenoptic annotations in a direct plenoptic format can result in reduced computation time.

The present invention is also a method for attaching annotations to a reference image in a plenoptic format,
-Presenting the reference image in plenoptic format using a viewer;
-Selecting an annotation;
Using the viewer to select a position for the annotation and to select one or more directions in which the annotation can be viewed;
Linking the annotation and the plenoptic format reference image with the position and the direction in memory;
It also relates to a method comprising:

  The method may be implemented using a suitable authoring system such as a suitable software application or website.

  The invention will be better understood with the aid of the description of an embodiment provided as an example and shown in the figures.

1 schematically illustrates a plenoptic capture device for capturing data representing a light field of a scene with one object at a first distance. FIG. 2 schematically illustrates a plenoptic capture device for capturing data representing a light field of a scene with one object at a second distance. Fig. 6 schematically shows a plenoptic capture device for capturing data representing a light field of a scene with one object at a third distance. 1 schematically illustrates a system that includes various devices that together implement the present invention. Shows an annotated view rendered from the same plenoptic data, where the viewpoint selected by the user during rendering changes between the two views, so that the same annotation is rendered differently. Shows an annotated view rendered from the same plenoptic data, where the viewpoint selected by the user during rendering changes between the two views, so that the same annotation is rendered differently. Shows an annotated view rendered from the same plenoptic data, where the viewpoint selected by the user during rendering changes between the two views, so that the first annotation is on the first view. The second annotation is visible on the second view. Shows an annotated view rendered from the same plenoptic data, where the viewpoint selected by the user during rendering changes between the two views, so that the first annotation is on the first view. The second annotation is visible on the second view. Shows an annotated view rendered from the same plenoptic data, where the focus distance selected by the user during rendering varies between the two views, so that the first annotation on the first view And the second annotation is visible on the second view. Shows an annotated view rendered from the same plenoptic data, where the focus distance selected by the user during rendering varies between the two views, so that the first annotation on the first view And the second annotation is visible on the second view. FIG. 3 is a block diagram of a method for generating and rendering a view with plenoptic format annotation. FIG. 6 is a block diagram of a method for modifying the rendering of annotations when the viewer selects different viewing directions and / or different focus distances on the view. FIG. 6 is a block diagram of a method for associating plenoptic format annotations with reference data. FIG. 3 is a block diagram of a method for continuous annotation of a series of plenoptic images, eg, video plenoptic images or plenoptic images captured by a moving user.

  Conventional cameras capture a 2D projection of a scene on a sensor and generate data representing the light intensity on each colored or colorless pixel. On the other hand, a plenoptic capture device known as such is a matrix that shows data representing the light field, ie more complete information about the light field including not only the light intensity but also the direction of the light. To capture.

  A complete light field can have up to seven parameters to describe each ray (or to describe a ray at a given position): three for position, two for direction, one for wavelength, and (video In the case of), it may contain one parameter for time. Some of the current plenoptic cameras send out plenoptic data containing two parameters for position, two for direction and one for wavelength. The sensor generates plenoptic data representing a so-called plenoptic light field, i.e. a matrix indicating at least the position and direction of the rays. This means that the plenoptic data generated by the plenoptic capture device contains more information about the light field than the conventional 2D image data generated by the conventional 2D camera. I mean.

  As of today, at least two companies, Lytro and Raytrix, have proposed plenoptic sensors capable of recording such plenoptic light fields. Although the two cameras are slightly different in design, the main idea is to resolve the different directions of light that are supposed to hit a single photosite (or pixel) in a standard camera sensor. is there. To that end, as shown in FIG. 1, an array of microlenses 20 is placed behind the main lens 1 in place of conventional camera sensors.

  In this way, the microlens 20 changes the direction of the light according to the incident angle, and the changed light reaches the different pixels 210 of the sensor 21. The amount of light measured by each of the N × M pixels 210 that make up the sub-image depends on the direction of the light beam that hits the microlens 20 in front of this sub-image.

  1-3 show a simple one-dimensional sensor containing n = 9 sub-images, each sub-image having one row of N × M pixels (or photosites) 210, In this example, N is equal to 3 and M is equal to 1. Many plenoptic sensors have more subimages and more pixels for each subimage, eg, 9 × 9 pixels, and N × M = 81 different light orientations on the microlens 20 Makes it possible to distinguish between them. Assuming that all objects in the scene are in focus, each sub-image thus contains a patch of lightness values representing the amount of light coming from different directions on that sub-image.

  In this structure, the array of microlenses 20 is positioned on the image plane formed by the main lens 1 of the plenoptic capture device, and the sensor 21 is positioned at a distance f from the microlens, where And f is the focal length of the microlens. This design allows high angular resolution, but relatively low spatial resolution (the number of effective pixels per rendered image is equal to the number of microlenses). This problem is addressed by other plenoptic capture devices where the microlens focuses on the image plane of the main lens, thus creating a gap between the microlens and the image plane. The price to pay in such a design is that the angular resolution is lower.

  As seen in FIGS. 1-3, the plenoptic light field corresponding to the scene with a single point 3 in this embodiment depends on the distance from the point 3 to the main lens 1. In FIG. 1, all light beams originating from this object reach the same microlens 20, so that all pixels in the sub-image corresponding to this microlens record a first positive light intensity while This results in a plenoptic light field where all other pixels corresponding to other lenses record different null light intensities. In FIG. 2, where the object 3 is closer to the lens 1, some of the light beams from point 3 are associated with other sub-images, ie sub-images associated with the two micro-lenses adjacent to the micro-lenses hit above. Reach the pixel. In FIG. 3 where the object 3 is at a greater distance from the lens 1, some of the light beams originating from the point 3 will be connected to different microlenses associated with the two microlenses adjacent to the microlenses hit above. Reach the pixel. Therefore, the digital data 22 transmitted by the sensor 21 depends on the distance to the object 3.

  The plenoptic sensor 21 thus sets for each subimage corresponding to the microlens 20 a set of (N × M) values indicating the amount of light originating from different directions on the lens above this subimage. The plenoptic data 22 including is sent out. For a given focused object point, each pixel in the sub-image corresponds to a measure of light intensity that hits the sensor at a constant angle of incidence phi (in the plane of the page) and theta (perpendicular to the page plane). To do.

  FIG. 4 schematically shows a block diagram of an annotation system embodying the present invention. The system includes a user device 4 such as a handheld device, a smartphone, a tablet, a camera, glasses, goggles, contact lenses, and the like. Device 4 is suitable with a plenoptic capture device 41, eg, a plenoptic capture device such as the camera shown in FIGS. 1-3 for capturing data representing a light field on scene 3. A communication module 401 such as a WIFI and / or a cellular interface for connecting the device 4 to a remote server 5 such as a cloud server via a network such as the Internet 6 and a processor such as a microprocessor 400 with program code. Including. Server 5 is accompanied by images such as a database, such as an SQL database, a set of XML documents, a set of plenoptic formats, etc., of one or more global models and / or reference plenoptic data representing the images. It includes a storage 50 for storing the collection and a processor 51 including a microprocessor with computer code for causing the microprocessor to perform the operations required in the annotation method. Annotations and corresponding positions may also be stored in storage 50 along with reference plenoptic data.

  The program code executed by the user device 4 can include, for example, application software or app that can be downloaded and installed by the user in the user device 4. The program code can also include part of the operating code of the user device 4. Program code can also include code that is embedded in a web page or executed in a browser, including, for example, Java, Javascript, HTML5 code, and the like. The program code may be stored as a computer program product in a tangible device readable medium, such as flash memory, hard disk, or any type of permanent or semi-permanent memory.

  The program code is executed by the microprocessor 400 in the user device 4 so that the microprocessor can transfer the captured data sets corresponding to the light field or at least some of the features of these data sets to the remote server 5. To send. The program code is configured to transmit data in a “plenoptic format”, ie without losing information about the direction of the light beam. The program code also causes the microprocessor 400 to receive from the server 5 annotated data in the plenoptic format, or annotated images, or annotations related to previously transmitted plenoptic data, A view corresponding to the accompanying captured data can also be rendered.

  The plenoptic annotation method may include two parts, an offline process and an online process. In general, the main purpose of the offline process is to tie annotations with plenoptic format reference images or other 2D, stereoscopic or 3D reference images.

Offline Phase For reference images in plenoptic format, the offline process may include the following steps, for example:
1. Receiving plenoptic format reference data from device 4 and representing a light field;
2. Presenting a rendered view of the plenoptic reference image, eg, using a plenoptic viewer;
3. Selecting plenoptic annotation;
4). Selecting a position and orientation for the annotation in the rendered view;
5. Selecting one or more light field parameters of the annotation;
6). (Optional) Attributing one action to the annotation;
7). Associating a reference image ray in memory with an annotation ray based on its position and orientation.

  This offline process can be implemented on the server 5, in the user device 4, or in another device, such as a personal computer, a tablet, etc. Typically, this offline process is performed only once for each annotation associated with the reference image. If the selected annotation is not initially available in the plenoptic format, it may be converted to a plenoptic format.

The main purpose of the online process is to add plenoptic annotation to plenoptic images. The online process may include two stages. The first stage is performed by program code executed by a microprocessor in the server 5 which may include an executable program or other code for causing the server 5 to perform at least some of the following tasks: May be:
1. Receiving plenoptic format data from device 4 and representing the light field;
2. Retrieving a previously stored model (reference image) and / or a plurality of reference data from the database 50;
3. Matching data received from a user device with a portion of a reference image or one of a plurality of reference images, respectively;
4). Determining the annotation associated with the matched reference image,
5. Sending plenoptic format annotations or plenoptic format annotated images to device 4.

  In various embodiments, instead of sending capture data to the remote server 5 to match a reference image in the server, this match is stored locally in a locally stored reference image set or device. And can be done locally within the user's device. In this embodiment, the server 5 is mounted on the user's device 4. The online process can be executed several times upon user request.

The second stage of the online process may be performed by program code executed by a microprocessor in device 4 that may include an executable program, or other to cause device 4 to perform at least some of the following tasks: May be implemented by the following code:
1. Receiving annotation data in plenoptic format from the server 5 with possibly associated actions;
2. Applying the received annotation data to the captured plenoptic light field;
3. Rendering an annotated light field into a user-visible view;
4). Interpret user interaction and perform associated annotation actions.

  In various embodiments, instead of applying the received annotation to the plenoptic light field captured on the device 4, this step can be performed on the server 5 side. In this case, either the final rendered view or the entire annotated light field is transmitted back to the device 4.

  Thus, the user can associate an annotation with a particular position and orientation in relation to the rendered view of the plenoptic reference image, and one or more annotations that should be used within this particular view. The light field parameters can be indicated. The same annotation may be rendered differently depending on the viewpoint selected by the viewer during view rendering. Since the light field parameter of the annotation may change, when the viewer selects a different viewpoint, the first annotation may be replaced with the second annotation at the same location.

  An example of an offline process flowchart is shown in FIG. This flow chart allows the user to select the annotation to be associated with the reference image and the location, orientation and light field parameters that accompany this annotation, so that this annotation will be captured prenoop matching this plenoptic reference image. It shows how to be applied to tick images.

  This method may use an annotation authoring system that may be executed locally within the user's device 4. The annotation authoring system may also be hosted on the server 5, where the web platform presents several tools to manage the annotations and relate them to the plenoptic reference images. Services such as augmented reality usage statistics are also available from the web platform. The annotation authoring system may also be run on different servers or devices, including the user's personal computer, tablet, etc.

  In step 150, the user selects a reference image, eg, an image in a plenoptic format. Images are uploaded onto the plenoptic authoring system and serve as support images for annotation.

  As part of the plenoptic authoring system, the viewer renders the uploaded data to the user in such a way that the user can visualize it. If the data is in a plenoptic format that humans cannot easily understand as such, this includes a plenoptic model for rendering the plenoptic model in a space understandable by the user. A step of using a optic rendering module may be included. The viewer constitutes a tool for manipulating plenoptic data and placing annotations at the desired position and orientation in relation to a given view, but all processing and combinations using plenoptic annotation Is performed directly in the plenoptic space.

  In one embodiment, the plenoptic model may be rendered as a 2D view, thus allowing the user to visualize it from one viewpoint at a time and one focus distance at a time, thus allowing the user to You will be able to understand and edit ptick models. In order to navigate from one 2D view to the other 2D view, a control mechanism is available so that another 2D view can be displayed upon request.

  In another embodiment, the plenoptic model may be rendered as a partial 3D scene that can visualize different directions of rays. The main difference from a standard full 3D scene is that the search for 3D scenes is limited when rendered from a plenoptic model. For example, the view direction as well as the view position are limited to those that have been captured by the plenoptic capture device.

  In step 151, the user selects a plenoptic annotation that he wishes to associate with a particular element or location of the plenoptic model. As already mentioned, plenoptic annotation is defined in plenoptic space and is therefore described by rays. These rays can describe, for example, text, images, video or other elements that act directly on the plenoptic image rays. Plenoptic annotation may be retrieved from a plenoptic annotation library in a database or in a file explorer or the like. Plenoptic annotation can also record for example by capturing it with a plenoptic capture device, by entering text using a text editor, by drawing an image, and / or recording audio or video To make it on the fly.

  In one embodiment, plenoptic annotation can be presented as a preview in a list or library on the authoring system. The plenoptic annotation preview corresponds to the rendering of the annotation for the default view. This default view can be taken randomly or, in the preferred embodiment, can be taken to correspond to the center view in terms of position and orientation plenoptic annotation ranges. The preview allows the user to quickly and clearly recognize what the plenoptic annotation corresponds to. For a general type of annotation that does not affect the wavelength of the model, ie, an annotation that cannot be visualized as it is, the preview shows the annotation applied to the center of the current model view rendered by the authoring system. Thus, if this type of annotation only has the effect of rotating all model rays by 10 °, the preview consists of the center portion of the current model rendered view where each ray is rotated by 10 °. Will be.

  At step 152, the user selects a position in the rendered view coordinate system of the selected reference model to which he / she wants to add plenoptic annotation using the plenoptic annotation authoring system. This can be done, for example, by dragging the annotation from the annotation preview list at the top of the displayed view at the desired location, and possibly translating, rotating, resizing, cropping and / or others. It can be done by editing in the form of Alternatively, the user may also enter coordinates as numerical values in the control panel.

  In step 152 ', the user can adjust the parameters of the annotation ray to generate another view of the annotation. As the user changes the annotation parameters, for example using a computer mouse pointer to change the orientation of the annotation, the annotation rays are combined with the plenoptic model rays for each new position or new orientation. A new 2D view is generated in the viewer. This is possible because the user's mouse pointer and its movement are projected onto the plenoptic space. The pointer movement is then applied to the annotation in a plane parallel to the virtual plane corresponding to the 2D rendered view.

  Once the plenoptic model and annotation rays are combined, the annotation effect is applied to the reference image rays. The process of superimposing plenoptic annotation can be thought of as a process of modifying rays. The captured plenoptic data can include information about the ray direction, wavelength (ie, color) for each ray, and thus annotation can be considered a modification of these parameters. For example, attaching text on the surface of an object can be thought of as a modification of the wavelength of a ray in a particular area on the surface.

  The type of effect generated by the annotation is determined by the annotation itself. In one embodiment, plenoptic annotation consists only of opaque text, for example. In this case, the wavelength of the model ray is completely replaced by the annotation ray wavelength for the mapped ray. For other annotations, by taking into account annotations that change the texture of the model, the ray of the model reflects the new texture, so its direction may be changed by the annotation. In yet another embodiment, the position of the model ray may be changed by annotation.

  Plenoptic annotation can be thought of as a filter that corrects rays. This provides more possibilities for displaying annotated scenes. One further example of this process is to modify the direction of the rays. In one embodiment, a glow effect can be applied to rays incident from a particular object in the captured plenoptic image by adding randomness to the direction of the rays. . Annotated objects can be given reflectivity. Another example is modification of surface properties such as modification of texture information. Plenoptic annotation allows modification of ray variables such as direction and wavelength, so combining the modification of variables modifies the surface of the object as if a texture were added on top of it Is possible. For example, plenoptic annotation allows a red flat surface to be changed to a yellow lumpy surface by modifying the direction and wavelength.

  Information describing the effect of the annotation on the model ray may be stored in the plenoptic annotation array as described in step 154.

  In step 153, the user selects one or more annotation light field parameters. This can be the wavelength of the annotation, for example for the purpose of changing its color. The user may similarly define different appearances for the same annotation as seen from different directions, as well as different annotations associated with the same element as seen from different directions.

  Alternatively, once successfully adjusted on the rendered plenoptic model, the user can choose to navigate to another view of the plenoptic viewer. Plenoptic annotation is automatically reported on the new view of the plenoptic model. The user can then edit the annotation and decide to change its light field parameters or appearance for this particular view. The user can proceed in the same way for all available views of the plenoptic model.

  An interpolation process is performed between the first and second views of the plenoptic annotation so that the user does not have to navigate through all the views of the plenoptic model. Also good. These two views of plenoptic annotation need not be contiguous. The user must specify the appearance of the annotation in the two views, and the plenoptic authoring system automatically generates an intermediate view of the plenoptic annotation. Other views of the plenoptic model that are not associated with annotations do not display it, and as a result, may not render annotations for specific viewpoints or focal planes of the scene .

  A plenoptic annotation may include data that corresponds to a ray and is described with a set of parameters. When rendering plenoptic annotation for the first specific view, the viewer sets some parameters, while others allow the user to modify. Navigating from this view to the second view, the user can change the parameters that the viewer must define, while other parameters can be modified. The interpolation process automatically calculates the ray parameters of the plenoptic annotation between these two views.

  In one embodiment, the parameters of each plenoptic annotation may be as follows: three (or possibly two) parameters for ray position in space, two parameters for its direction, One parameter for that wavelength, and possibly one parameter for time. For a particular view rendered by the plenoptic viewer, the viewer may set parameters such as position, direction and time, for example. The user can then change parameters not defined by the viewer, which in this example corresponds to the wavelength of the light beam. Now assume that the user sets it to the first value v1. Now suppose that for another view of the annotation, ie for different values of position, direction and time parameters, the user changes the wavelength value for the second view and sets it to eg v2. The interpolation process aims to calculate an annotation value between v1 and v2 for the view in the middle of the position, direction and time parameters associated with the first and second views. In other embodiments, the interpolation may also consider calculating values for other parameters of the plenoptic data including position, direction, wavelength, and / or time.

  Specific examples of interpolation include, for example, changing the color of a plenoptic annotation that transitions from orange to a more reddish color, for example, while an annotation is visible for a particular view, Includes changing the visibility of annotations when they are invisible.

  Different interpolation methods are possible, including, for example, linear interpolation, quadratic interpolation or higher order interpolation between the two views of the annotation. Similarly, more advanced interpolation methods may take into account other characteristics of the scene or the annotation itself to generate a new ray of annotation.

  In step 153 ', when an annotation is displayed on the captured image, one action may be associated with all or part of the annotation. These actions can be triggered by the user, or can be performed automatically using, for example, a timer. Actions include launching a web browser at a specific URL, moving an annotation, making it appear or disappear, animating the annotation, playing a video, and a menu that presents further possible actions. Activating, activating a slide show, or playing an audio file. Actions that allow a view of the plenoptic data provided to the user to be modified are also possible, for example, actions that allow the view of the plenoptic data to be focused at a given focal length.

  In step 154, the plenoptic annotation is stored in memory, eg, database 51 or user device, and associated with the corresponding location, orientation, and selected reference plenoptic model. Since the required annotations are known, it is possible to store the annotations attached to each reference plenoptic model in a plenoptic format. Each annotation is stored as a separate plenoptic file.

  The plenoptic annotated reference data is generated from the plenoptic reference data and the corresponding one or more plenoptic annotations. This augmented reality model takes the form of a file that contains all the information needed to render the plenoptic model back with its accompanying annotations. It therefore describes the relationship between plenoptic reference data and its annotations. Plenoptic annotated reference data can be rendered directly on the plenoptic annotation authoring system to pre-visualize the results, as well as rendered on the client side for some plenoptic extensions You can also render reality.

  Information describing the effect of the annotation on the model ray is stored in the plenoptic annotation data. Modifications defined by annotations affect the model ray parameters. As a result, the annotation can describe, for example, a modification of the direction, position, time or wavelength of the model ray. In other words, this information describes a function of the model rays.

  At the time of annotation creation, a unique identifier is assigned to each ray of the annotation. When applying annotations on the authoring system, the unique identifiers of the annotation rays are matched to their corresponding rays in the model. As a result, each ray of the model is assigned an annotation ray identifier, which is then used when the system must apply annotations on the model for each ray, as is the case, for example, mainly in the online phase. Used by.

  Annotation information can be stored in a two-dimensional array, where each ray contains information about its effect on the model for each parameter. The unique identifier of the annotation ray is then used to define the corresponding ray effect in the array for each parameter. In other words, the first dimension of the array corresponds to the ray referenced by its identifier, and the second dimension corresponds to those parameters, ie the light field parameters. This format can be used to fully represent any annotation, since any modification of the model ray for any parameter can be represented in the array.

  In one embodiment, the annotation can modify all model ray directions, for example, 10 ° for one angle. As shown in Table 1 below, the two-dimensional array now includes 10 ° in a parameter column corresponding to the directional angle. Since all rays are assumed to act in the same way, the column is marked 10 ° for all rays. If you want to apply the annotation effect on the corresponding model ray, the system first identifies the annotation and model ray pair, extracts the unique identifier corresponding to the annotation ray, examines the annotation table in detail, See what effect the ray has and thus finally apply this change to the model ray. In this example, the angles of all model rays affected by the annotation are rotated by 10 °.

As an example of an offline phase, a user may wish to add text annotation to a scene that includes a building. In addition, the color of the text annotation needs to vary between one viewpoint and another viewpoint. The user then takes the following steps:
1. A plenoptic capture of the building is uploaded to the plenoptic annotation authoring system.
2. A 2D view is rendered from the captured plenoptic image and presented to the user.
3. The user selects a text annotation type from the annotation type list, enters his text, and drags the text annotation onto the rendered 2D view.
4). The user can move the position and orientation of the rendered 2D view viewpoint or annotation so that the annotation looks exactly what the user wants.
5. The user sets the text color for the current rendered viewpoint.
6). The user moves the viewpoint of the rendered plenoptic image to another position.
7). The user sets the text color to a different value for this other viewpoint.
8). The plenoptic annotation model is saved and ready for use for the online phase of the annotation process.

The plenoptic annotation authoring system performs the following tasks to generate a proper annotation model based on the previously described user action steps for text annotation.
1. A 2D view is rendered to the user based on the viewpoint settings initially set to default values.
2. By tracing the ray from the text object to the virtual viewpoint, a plenoptic version of the text annotation is generated. This creates a set of rays, each described by a unique identifier. This ray set describes the text. These rays are represented in the memory by an array corresponding to the modifications that must be applied to the reference plenoptic image. In this case, the array contains the wavelength values that the ray matched with the annotation ray must take.
3. The annotation is initially in a predefined default position within the authoring tool. Annotation rays are combined with reference plenoptic image rays. These relationships between the reference image rays and the annotations are stored for future use by using the ray unique identifier.
4). As the user moves / changes the orientation of the annotation using, for example, a computer mouse pointer, the different rays of the annotation are combined with the other rays of the captured plenoptic image and new for each position or orientation modification. A 2D view is generated. This is possible because the user's mouse pointer is projected into the plenoptic space. At this time, pointer movement is applied to the annotation in a plane parallel to the virtual plane corresponding to the 2D rendered view. As the annotation is moved, the ray relationship between the reference image and the annotation is changed and updated according to changes in the annotation position or orientation.
5. When the user selects a text color for the current viewpoint, the wavelength value of the annotation array is changed to match the selected color.
6). When a new viewpoint is selected and a new text color is selected, the wavelength values of the annotation array corresponding to the rays used to generate this new rendered view are changed. Wavelength values intermediate between the first viewpoint and the second viewpoint are interpolated using standard or special interpolation methods.
7). When the user saves the model, the plenoptic annotation array is saved with the uploaded plenoptic reference model and can be used in the online phase.

Online Phase As explained above, the online phase of the overall annotation process occurs when a user capturing a plenoptic image wants the image to be annotated.

  The online phase of the annotation process is applied to the input plenoptic image to obtain the final plenoptic annotated image. It matches the input image with several reference models, retrieves the annotations of the matched reference model, combines the annotation with the input plenoptic image, annotated to the user in an understandable form Rendering the view and possibly processing the user's interaction to generate different defined actions for the annotation.

  Since the annotation content composed of rays is in the plenoptic format and the captured image is also in the plenoptic format, these two data sets exist in the same space. Thus, the annotation can be applied directly to the plenoptic image without the need for further projection. The modified plenoptic space to which the annotation has been applied can then be projected, for example, into a 2D view. This also means that the projection parameters selected for the plenoptic rendering process (e.g. selection of focus, depth, viewpoint change, etc.) implicitly apply to plenoptic annotation. For example, when changing the focus or viewpoint of a rendering process, annotations have an effect applied to them.

  The online plenoptic annotation process illustrated in FIG. 8 includes a first step 100 in which data representing a plenoptic format light field (plenoptic data) is retrieved. The plenoptic data is retrieved by a device 4 that captures data using a plenoptic capture device, or by an apparatus such as a server 5 that receives plenoptic data from the device 4 over a communication link. May be retrieved.

  In step 101, the retrieved data is matched with reference data. This step may be performed in the device 4 and / or the server 5. This step includes determining the feature set in the capture data, finding matching reference data representing a reference image with matching features, and registering the capture data and the reference data, for example, as described in US13645762. Registering may be involved. The reference data may represent plenoptic format images or other images and may be stored in a memory 51 such as a database accessible from multiple devices. Identification of matching reference data may be based on an indication provided by the user's location, time, time, signals received from scene elements, user and / or image similarity. The purpose of the registration process is to estimate the transformation between the captured plenoptic image ray and the ray from the matched plenoptic reference image so that the user's position and reference can be estimated. The goal is to discover geometric relationships between data.

  In step 102, the plenoptic annotation associated with the matching reference data is retrieved from the memory 51, for example. This annotation is a plenoptic format. That is, it is described by light rays. These annotation rays may represent, for example, text, still images, video images, logos, and / or other elements that act directly on the plenoptic image rays.

  Annotations include sound in the plenoptic space, for example sound attached to a specific group of rays in the plenoptic reference image, so that the selected rays are visible in the plenoptic image and / or Alternatively, the sound may be played back only in a part of the directions in focus.

  In step 103, the retrieved annotations in the plenoptic format are combined with the captured plenoptic data to generate annotated data representing the annotated image in the plenoptic format. This combination may take place in the server 5 or in the device 4. In the latter case, the server 5 may send the annotated data to the device 4, which then performs the combination. This annotation combination is possible because the transformation that projects the rays of the reference image onto the captured plenoptic image is known from the matching step (step 101). Thus, annotations can also be applied to captured plenoptic images.

Plenoptic annotation can be applied to a captured plenoptic image using the following method:
1. Find a transformation to project the reference plenoptic image ray onto the online plenoptic image ray retrieved in step 100 of FIG. 8;
2. For each retrieved annotation of the reference plenoptic image defined within the offline phase:
1. By reading the annotation array defined within the offline phase, it identifies and selects which rays of the reference plenoptic image should be modified according to the annotation.
2. Project the ray identified in item (1) onto the online plenoptic image. This creates a correspondence between the selected ray of the reference plenoptic image and the ray derived from the capture plenoptic image.
3. For each ray of the captured plenoptic image selected in item (2), a transformation is applied to the ray as defined in the plenoptic annotation array. The array is used as a lookup table, where the ray and transformation parameters (eg, wavelength, direction, ...) that are identifiable by the selection process of steps (1) and (2) are used as lookup keys. .

  As an example, if the annotation ray represents text, the annotation array includes a single non-null light field parameter that is the wavelength corresponding to the color of the text. Thus, the captured plenoptic image ray is modified by increasing or decreasing the ray wavelength by a factor stored in the annotation array. This coefficient is looked up in the array by using the conversion between rays calculated during the registration process.

  In step 104, the view is rendered from annotation data, such as a 2D or stereoscopic view, and displayed on the display 40 or using another device and presented to the user / viewer. This view rendering process is described in further detail below in conjunction with FIG.

  In step 105, interaction with the annotation is enabled. The system can react to different events to perform the specific actions defined earlier in the offline part of the annotation process. Such an event can be a user interaction with the annotation. Using a touch screen, hand tracking sensor or any other input device, the user can point to and interact with a given annotation. This interaction will generate an interaction event that can trigger a specific action defined in the offline phase of the annotation process.

  Another possible type of event is an event that is triggered when a specific change in the scene is detected. As will be described later in this section, occlusion due to objects of the reference model in the captured plenoptic image may be detected. This occlusion event may trigger an action previously defined during the offline phase of the annotation process. As another example of a possible event that triggers an annotation action, a speech recognition module can be used to trigger a certain action based on a certain type of detected speech.

  FIG. 9 illustrates the view rendering and various possibilities for the viewer to subsequently modify the rendering. As pointed out above, the augmented reality view is rendered in step 104 from the annotated data generated from the captured view, and from the plenoptic format annotation data as previously described with reference to FIG. Is done. The rendered view can be a standard 2D view, as generated by a pinhole camera, a stereoscopic view, a video, a holographic projection of plenoptic data, or preferably for refocusing and / or changing the viewpoint. It may be a dynamic image module that presents an image using several commands. The video module can be an HTML5 / Javascript web page that can render a plenoptic image as a function of command values or as a flash object, or other technology that allows for the dynamic presentation of multiple images. Examples of views that may be rendered during step 104 are shown in FIGS. 5A, 6A and 7A. The views on FIGS. 5A and 6A include an object 60 with annotation 61. In the view of FIG. 7A, additional objects 62 are also seen at different depths and thus out of focus. The refocusing or viewpoint change can be triggered manually by the user (eg by selecting an object or position on or around the image) or automatically (eg when the user moves).

  In step 105, the user enters a command to modify the viewpoint for the purpose of generating during step 107 a new view from the same plenoptic data corresponding to the same scene viewed from a different viewpoint. Algorithms for generating various 2D images of a scene viewed from different viewpoints or viewing directions from plenoptic data are known as such and are described, for example, in US Pat. No. 6,222,937. . An example of a modified 2D image generated by this command and executed by the viewpoint selection module 403 is shown in FIG. 5B. As can be seen here, not only the appearance of the object 60 (perspective) but also the appearance of the annotation 61 is similarly modified by this command. In fact, the annotation is applied directly on the plenoptic space represented by the input plenoptic data, so when the view is generated from the plenoptic space, the annotation is a plenoptic image. Appears in a converted state in the same way as. This creates a more realistic annotation.

  Some annotations may only be visible from the first viewing direction set and not from other directions. Thus, as shown in FIG. 6B, the change of viewpoint during step 105 results in a new view in which one annotation 61 is hidden but a new annotation 64 associated with the same object is revealed. There is a possibility to bring. Multiple annotations may be associated with a single location of the reference image, but in this case the viewing direction is different. The annotation itself can also look different when rendered from the first viewing direction compared to the second different viewing direction due to different annotation light field parameters set during the offline phase of the annotation process. There is sex. The change in appearance can be defined by the annotation itself, but can also be a function of the input plenoptic image.

  In step 106 of FIG. 9, the user enters a command to refocus the image and generate a new image focused at a different distance from the data in the plenoptic format. This command may be executed by the refocus module 402. As can be seen from FIGS. 7A and 7B, the first annotation 61 visible at the first focus distance thus disappears or becomes less blurred at the second focus distance shown in FIG. In this case, only the second annotation 63 may appear at the second focus distance.

  The different commands used in steps 105 and 106 to change the rendered view can also be issued automatically in the context of user movement. In one embodiment, user movement is tracked using an inertial measurement unit (IMU) embedded in the plenoptic capture device. By using this module, the rendered view is automatically updated as the user moves. For example, if the user moves to the left, the viewing direction is moved slightly to the left. The same principle applies when the user moves forward, where the focus range is moved forward as well, making the object clearer in the background plane compared to the previously rendered view. And create softer objects in the foreground plane. The present invention is not limited to the use of an IMU to track user movement. Other means such as using the content of the plenoptic image directly to track user movement can also be used.

  In another embodiment, an online plenoptic annotation process is continuously applied to a plenoptic image stream generated by a moving user's plenoptic capture device. This continuous processing allows the user to move continuously or move his / her plenoptic capture device to update the plenoptic annotation in real time. The plenoptic image stream must be processed in real time, similar to the view rendering (step 104 in FIG. 8), so that the user perceives the annotation as if it were part of the scene. become. In this embodiment, the number of plenoptic images that need to be processed from the stream is much less due to the fact that the viewing direction can be modified later without having to have another plenoptic capture It is possible to achieve the same effect in the state. In fact, assuming that a single plenoptic capture would allow the rendering of a view within a certain viewing direction range, and if the user did not move outside this range, any plenoptics from the stream would be Ptic images need not be processed, only step 104 of FIG. 8 need be performed again. This opens new possibilities for more computationally efficient real-time tracking by asynchronously processing new plenoptic image frames as the user approaches the viewing range boundary, thus allowing new frames to be captured. Users no longer perceive latency when it has to be processed.

  An example of a method for annotating a plenoptic video is shown in FIG.

  Steps 200, 201, 202, and 203 in FIG. 11 are similar to or equivalent to steps 100, 101, 102, and 103 in FIG.

  In step 204, viewing direction parameters are calculated as a result of the registration process of step 201.

  In step 205, the view is rendered based on the viewing direction calculated in the previous step.

  In step 206, an inertial measurement unit (IMU) is used to determine the user's movement in relation to the time that step 200 was calculated. At this time, a decision is made to either return to step 200 to process the new plenoptic image or go directly to step 204 to update the viewing direction parameter based on the IMU motion estimation. The amount of movement is used to determine whether the previously captured plenoptic data can be used to create a new view. This typically depends on the field of view of the plenoptic capture device.

  Possible blockages may be taken into account when rendering plenoptic annotation. If the target element to be annotated is obscured from the view of the capture device by another object present in the input plenoptic image, the plenoptic annotation may be occluded.

  In one embodiment, the rendering module utilizes the plenoptic format of the captured data to visually hide the annotation behind unrelated objects. The rendering module recognizes from the plenoptic reference data the characteristics of the captured ray that would have come from each element of the captured plenoptic image. If the captured ray has a different property than the element's expected ray, it means that the occlusion object is in front of the element, and therefore the annotation need not be displayed for this element. May mean.

  Similarly, if a ray corresponding to an element in the captured image has a different direction than the corresponding direction in the reference image, this may mean that the element is at a different depth. The rendering module can use this information to detect occlusions. In addition, the light color information can be used to determine whether the capture element is occluded. However, the color information is not sufficient because the object to be blocked may have the same color as the target element.

Fields of Use The process of providing annotations in the plenoptic format and annotating plenoptic images in the same space as the annotations provides a new field of use for augmented reality.

  An example of a first field of use is the use of plenoptic annotation systems within a social context. In fact, the plenoptic image of the object / scene can be captured by the user using its plenoptic capture device. At this time, the captured plenoptic image can be annotated by the user using all types of annotation, including the plenoptic image previously captured and used as an annotation. The annotated scene can then be shared with the user's friends using a social network so that these friends can experience the annotated scene while capturing it using their plenoptic capture device It is like that. The advantage of using the plenoptic annotation process in this case is enhanced by the fact that the annotation already exists in the plenoptic space because it is a plenoptic image. Thus, performing the annotation process in the same plenoptic space produces a more computationally efficient and more realistic annotated scene.

  A second field of application utilizing different information in the plenoptic space is the use of specially designed plenoptic annotation in the field of architectural design. As described in the preceding part of the present invention, plenoptic annotation consists of rays combined with plenoptic image rays within the online phase. The way in which the rays are combined is defined in the offline part of the annotation process. This combination may be such that the rays from the plenoptic image are not replaced by other rays from the annotation, but only their direction, for example, is changed. By defining annotations that modify not only the wavelength of light in the plenoptic image, but also their direction, for example, it is possible to simulate changes in the texture or material of the captured scene. For this architectural design, it is advantageous to use plenoptic annotation to simulate how a particular room or a particular building will look when different materials are applied to the walls, for example. Can do. In another embodiment, a simulation of weather conditions can be applied to the captured plenoptic image. Annotations that simulate rain can be applied to a scene. This produces an annotated image with the rain effect applied, so that the user can visually see how the scene will look in case of rain or other different climatic conditions, where Different light reflections and refractions are appropriately processed and calculated in a realistic way thanks to plenoptic information.

  As another example, a treasure hunt is a well-known application in conventional 2D augmented reality solutions. It consists of attaching annotations to physical objects and having them look for these annotations (called treasures) by giving hints to friends or others. In other words, when someone approaches a hidden object, he scans surrounding objects using his plenoptic capture device to determine whether they are associated with a single annotation. be able to. By using plenoptic annotation, the treasure hunt becomes more exciting because we can limit annotation visibility to some viewing directions or focus distances. For example, the user decides to attach an annotation to a plastic image and make this annotation visible only if a future hunter stands in front of the plastic image and is looking at the plastic image from that angle. be able to. Similarly, the refocusing property of the plenoptic space can be used to ensure that the hunter is focused on the plastic image itself and therefore only displays the annotation in this case. As a result, the treasure hunt becomes even more intriguing because the user is prevented from discovering treasures while randomly scanning the surroundings, and is forced to resolve the navel truly.

  Another area of use relates to city guides in urban environments. For example, consider a user who is in a visiting city and is looking for tourist attractions such as historical monuments, tourist points, statues, museums, and local restaurants. Users, of course, do not want all information to appear on their screen at once using their augmented reality system. You will only be confused by this whole visual overlap on the screen. Instead, plenoptic annotation can be created according to the user's perspective and focus. For example, elements of an image captured by a user at a particular viewing angle (or within a particular viewing angle range) may be displayed with a lower importance than the elements that the user faces. In one embodiment, low-importance annotations may only be displayed on the screen as titles or points (this can be expanded when the user clicks on them), while the more important interests The points that either present further details or have a larger size or enhancement on the image.

  The ability to select a viewing direction in which annotations are not visible can be distracted by annotations attached to elements such as advertisements, stores, etc. that are not related to traffic, but want to obtain augmented reality images on the navigator display. It is attractive for drivers of unwanted vehicles. In this case, these distracting annotations may be associated with a single orientation range that is selected not to be displayed on the image captured from the road.

Terms and Definitions The various operations of the methods described above may be performed by any suitable means capable of performing the operations, for example, various hardware and / or software component (s), circuits, and / or module (s). . In general, any of the operations described in this application may be performed by corresponding functional means capable of performing the operations. Various means, logic blocks and modules may be used in a circuit, application specific integrated circuit (ASIC) or general purpose processor, digital signal processor (DSP), application specific integrated circuit (ASIC), field programmable gate array signal (FPGA) or others. Including, but not limited to, programmable logic devices (PLDs), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein May include various hardware and / or software component (s) and / or module (s). A general purpose processor may be a microprocessor, but in the alternative, the processor may be any commercially available processor, controller, microcontroller or state machine. The processor may also be implemented as a combination of computing devices, such as a DSP and microprocessor, multiple microprocessors, one or more microprocessors in combination with a DSP core, or any other combination of such configurations. . The server may be implemented as a single machine, as a set of machines, as a virtual server, or as a cloud server.

  As used herein, the expression “plenoptic data” is generated using a plenoptic capture device or calculated from other types of data and is a light field image of a scene, ie, light brightness. And any data describing an image in which this direction of light as well as color is stored. A 2D or stereoscopic projection rendered from such plenoptic data is not considered a plenoptic image because this light direction is lost.

  As used herein, the expression “plenoptic space” may represent a multi-dimensional space that can describe a light field, a function that describes the amount of light in all directions in space. The plenoptic space may be described by at least two parameters for the position of light, two for its orientation, one for its wavelength, and possibly one for time (for video).

  As used herein, the term “annotation” can be superimposed in a plenoptic space represented by, for example, text, still images, video images, logos, audio and / or plenoptic data. Or it encompasses a variety of possible elements, including other elements that can be merged in other ways. More generally, the term annotation encompasses various methods for modifying various parameters of a plenoptic spatial ray represented by plenoptic data. Annotations are dynamic and may change their position and / or appearance over time. Furthermore, annotations are user-interactive and may be responsive to user operations (eg, moved or transformed at the time of user interaction).

  As used herein, the term “pixel” may refer to a single black and white photosite or a plurality of adjacent photosites for detecting light in different colors. For example, three adjacent photosites for detecting red, green and blue light can form a single pixel.

  As used herein, the term “determining” encompasses a variety of actions. For example, “determining” includes calculating (calculating, computing), processing, deriving, investigating, looking up (eg, in a table, database or another data structure). ), Ascertaining, estimating, and the like. Similarly, “determining” may include receiving (eg, receiving information), accessing (eg, accessing data in a memory), and the like. Further, “determining” may include determining (resolving), selecting (selecting, choosing), establishing (establishing), and the like.

  Capturing an image of a scene involves using a digital pinhole camera to measure the brightness of light reaching the camera's image sensor. Capturing plenoptic data may involve using a plenoptic capture device or generating light field data or other descriptions of scenes and light sources from a virtual 3D model May be. Retrieving an image may involve capturing the image or retrieving the image over a communication link from a different device.

  For example, the expression “render a view”, such as “render a 2D view from plenoptic data”, refers to calculating or generating an image, eg, a 2D image from information contained in the plenoptic data or Includes the action of computing a holographic image.

  The method or algorithm steps described in connection with this disclosure may be implemented directly in hardware, in a software module executed by a processor, or a combination of the two. A software module may reside in any form of storage medium that is known in the art. Some examples of usable storage media include random access memory (RAM), read only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, etc. . A software module may contain a single instruction, or many instructions, and may be distributed over different code segments, between different programs, and across multiple storage media. Software modules are executable programs, single parts or routines or libraries used within a complete program, multiple interconnected programs, “apps” executed by multiple smartphones, tablets or computers, widgets, It may be composed of a Flash application, a part of an HTML code, or the like. A storage medium may be coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The database may be implemented as an SQL database, an XML document set, a semantic database, or any structured data collection that includes a set of information available on an IP network, or any other suitable structure.

  Thus, some aspects may include a computer program product for performing the operations presented herein. For example, such a computer program product may include a computer readable medium having instructions stored thereon (and / or encoded) with instructions for performing the operations described herein. It can be executed by one or more processors. For some aspects, the computer program product may include packaging material.

  It is to be understood that the claims are not limited to the precise configuration and components illustrated above. Various modifications, changes and variations may be made in the arrangement, operation and details of the methods and apparatus described above without departing from the scope of the claims.

1 Main Lens 4 User Device 5 Server 6 Internet 20 Micro Lens 21 Sensor 40 Display 41 Plenoptic Capture Device 50 Storage 51 Processor 60, 62 Object 61, 63, 64 Annotation 210 Pixel 400 Microprocessor 401 Communication Module 402 Refocusing module

EP 1246080 specification European Patent Application No. 2207113 International Publication No. 2008/134901

Claims (28)

-Retrieving (100) data representing a light field using a plenoptic capture device (4);
-Executing (101) program code to match the corresponding reference data with the retrieved data;
-Executing (102) program code to retrieve at least one annotation (61, 63, 64) in a plenoptic format associated with an element of said reference data;
-Executing (103) program code to generate annotated data in a plenoptic format from the retrieved data and the annotation;
Including methods. -Selecting a viewing direction (105);
Rendering a view corresponding to the annotated data from the viewing direction (107);
The method of claim 1, further comprising: wherein the representation of the annotation (61) depends on the viewing direction. Rendering a first view corresponding to the annotated data from a first viewing direction (104);
-Selecting a second viewing direction (105);
-Rendering (107) a second view corresponding to the annotated data from the second viewing direction;
The method of claim 1, further comprising: wherein the representation of the annotation (61, 61 ') is changed between the first view and the second view. Linking the first annotation (61) with the first location and the first direction;
-Linking a second annotation (64) with the first location and the second direction;
-Rendering a view corresponding to the annotated data;
-Making a selection between the first or second viewing direction (105);
-If a first viewing direction is selected, rendering a view that contains the first annotation but not the second annotation, or if a second viewing direction is selected, Rendering a view that includes a second annotation but not the first annotation;
The method according to claim 1, further comprising: -Rendering a plenoptic format reference data and a first view corresponding to the first viewing direction;
-Binding annotations to one element in the first view;
Rendering the reference data in a plenoptic format and a second view corresponding to a second viewing direction;
-Binding annotations to the elements in the second view;
Interpolating annotations of the element in an intermediate view between the first and second viewing directions;
The method according to any one of claims 1 to 4, further comprising:   6. The method of claim 5, further comprising calculating a plenoptic format annotation from the first, second and intermediate views. Rendering (104) a first view corresponding to the annotated data and a first focus distance;
-Modifying the focus distance (106);
Rendering (107) a second view corresponding to the annotated data and the modified focus distance;
The method according to any one of the preceding claims, further comprising: wherein the representation of the annotation (61) is changed between the first view and the second view. Linking the first annotation (61) with the first location and the first depth;
-Associating a second annotation (63) with said first location and a second depth;
Rendering (104) a first view corresponding to the annotated data;
-Selecting between the first or second focus distances (106);
If a first focus distance is selected, render a second view that includes the first annotation (61) but not the second annotation (63), or a second focus distance Rendering a view that includes the second annotation but does not include the first annotation (107);
The method of claim 7, further comprising:   The method according to any one of claims 1 to 8, wherein at least one of the annotations is a voice attached to coordinates and bound in a specific direction.   The method according to claim 1, wherein at least one of the annotations is a video.   11. A method according to any one of the preceding claims, wherein at least one of the annotations acts as a filter for altering the direction of the light ray at a specific location in the plenoptic space.   The method of claim 11, wherein one of the annotations modifies the direction of a ray.   The method of claim 12, wherein the one annotation modifies a surface or texture property of the object.   14. A method according to any one of the preceding claims, wherein the annotation is defined by a ray direction or an array defining ray direction modifications at different points in the plenoptic space.   Rendering determines if the annotation is occluded by an element of the retrieved light field or if the annotation occludes an element of the retrieved light field from the direction of the ray corresponding to the element 15. A method according to any one of claims 2 to 14, comprising determining as a function of the depth of the element to be performed.   Rendering includes retrieving an annotation in a plenoptic format and applying the annotation to a plurality of consecutive retrieved light fields in a stream of retrieved light fields. Item 16. The method according to any one of Items 2 to 15.   17. A method according to any one of claims 2 to 16, wherein rendering includes merging rays corresponding to retrieved data and annotation rays. A device for capturing and annotating data corresponding to a scene,
A plenoptic capture device (41) for capturing data representing a light field;
-With a processor (400);
-A display (40);
At least one annotation (61, 63, 64) in a plenoptic format associated with an element of data captured using the plenoptic capture device (41) when the program code is executed The program code for rendering a view on the display (40) generated from capture data and including the at least one annotation;
(4) comprising:   The program code further includes a refocus module (402) for allowing a user to refocus the view and changing the presentation of the annotation in response to a selected focus distance. The device described in 1.   The program code further comprises a viewpoint selection module (403) for allowing a user to change the viewpoint used for the rendering and to change the presentation of the annotation in response to the selected viewpoint. The apparatus according to 18 or 19. An apparatus (5) for determining annotations:
A processor (51);
A storage device (50);
-When the program code is executed, the processor receives data representing a light field, matches the data with one reference data, and a plenoptic format associated with the reference data from the storage device. The program for determining an annotation (61, 63, 64) and causing the remote device (4) to transmit either the annotation in plenoptic format or the annotated image in plenoptic format With code;
Including the device.   The apparatus of claim 21, wherein the program code further includes a module (510) for adding plenoptic format annotations and associating them with positions and viewing angles in the reference data.   23. The apparatus of claim 22, further comprising a memory that stores annotations as ray directions or an array of ray direction corrections at different points in the plenoptic space. A method for attaching annotations to a reference image in a plenoptic format,
Presenting the reference image in plenoptic format using a viewer (150);
-Selecting an annotation (151);
Using the viewer to select a position for the annotation (152) and to select one or more directions in which the annotation can be viewed (153);
Linking in step 154 the annotation and the reference image in plenoptic format with the position and direction;
Including methods.   25. The method of claim 24, comprising associating a plurality of annotations with a single location but having a plurality of different directions. -Rendering a plenoptic format reference data and a first view corresponding to the first viewing direction;
Linking a first annotation to an element in the first view;
Rendering the reference data in a plenoptic format and a second view corresponding to a second viewing direction;
-Linking a second annotation different from the first annotation to the element in the second view;
26. The method of claim 24 or 25, further comprising: -Rendering a plenoptic format reference data and a first view corresponding to the first viewing direction;
-Binding an annotation to an element in the first view;
Rendering the reference data in a plenoptic format and a second view corresponding to a second viewing direction;
-Attaching annotations to the elements in the second view;
Interpolating annotations of the element in an intermediate view between the first viewing direction and the second viewing direction;
27. The method of any one of claims 24-26, further comprising: An apparatus for attaching annotations to a reference image in a plenoptic format,
-With a processor;
-Let the processor present the reference image in plenoptic format using a viewer (150); allow the user to select an annotation (151) and allow the user to select a position for the annotation; (152) program code for allowing (153) to select one or more directions in which the annotation can be seen;
-A memory for storing the annotation, the position and the direction;
Including the device.

Images