Nothing Special   »   [go: up one dir, main page]

WO2012004622A1 - An augmented reality method, and a corresponding system and software - Google Patents

An augmented reality method, and a corresponding system and software Download PDF

Info

Publication number
WO2012004622A1
WO2012004622A1 PCT/IB2010/002032 IB2010002032W WO2012004622A1 WO 2012004622 A1 WO2012004622 A1 WO 2012004622A1 IB 2010002032 W IB2010002032 W IB 2010002032W WO 2012004622 A1 WO2012004622 A1 WO 2012004622A1
Authority
WO
WIPO (PCT)
Prior art keywords
terminal
image
real
real objects
information representative
Prior art date
Application number
PCT/IB2010/002032
Other languages
French (fr)
Inventor
Vincent Daniel Piraud
Original Assignee
Vincent Daniel Piraud
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Vincent Daniel Piraud filed Critical Vincent Daniel Piraud
Priority to PCT/IB2010/002032 priority Critical patent/WO2012004622A1/en
Publication of WO2012004622A1 publication Critical patent/WO2012004622A1/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/05—Geographic models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality

Definitions

  • the present invention relates to the general technical field of digital imaging, and more precisely to augmented reality techniques enabling virtual elements to be combined with an image of a real scene.
  • the present invention relates more particularly to an augmented reality method, to a computer program including computer program code means suitable for executing the steps of such a method, to a medium capable of being read by a computer and having such a computer program recorded thereon, and to an augmented reality system.
  • Augmented reality methods and systems are already known. They enable a user to improve the perception of real surroundings by making it possible for the user to view said real surroundings while also perceiving additional information elements that are absent from the real scene.
  • the concept of augmented reality has become increasingly widespread with the appearance and commercial success of portable electro-optical appliances of the portable digital assistant (PDA) or smart phone type.
  • PDA portable digital assistant
  • a smart phone that, when associated with appropriate control software, enables the user to view the real image of the landscape surrounding the user on the screen of said smart phone together with overlaid information relating to the landscape in question (toponyms, etc.).
  • augmented reality software for a smart phone makes use of the geographical position of the smart phone as determined by the GPS system incorporated therein, and also of topographic data stored in a database with which the smart phone communicates over a radio channel.
  • the advantage of such an augmented reality application lies essentially in that it is intended to enable augmented viewing to take place in real time while the user is on the move.
  • the user must be capable of observing a landscape in real time on the smart phone screen while also seeing virtual elements that are displayed thereon as an overlay and in real time, which virtual elements augment the reality of the observed real scene.
  • This augmented viewing in real time should ideally be continuous as the user varies the position and/or the orientation of the smart phone.
  • Such functions require a large amount of computer resources, which are not necessarily available in all smart phones. Specifically because they are portable, smart phones are necessarily limited in terms of computer power and memory.
  • the objects given to the invention consequently seek to remedy the above-mentioned drawbacks and to propose a novel augmented reality method and system that provide a real time result that is particularly fluid and reliable, including when the method is implemented on an appliance having limited computer resources.
  • Another object of the invention seeks to provide a novel augmented reality method and system that are particularly accurate and reliable.
  • Another object of the invention seeks to provide a novel augmented reality method and system that can be implemented easily and under all circumstances.
  • Another object of the invention seeks to provide a novel augmented reality method and system that consume particularly little in terms of computer resources.
  • Another object of the invention seeks to provide a novel augmented reality method and system that are suitable for being implemented using commercially available means. Another object of the invention seeks to provide a novei augmented reality method and system that are particularly adapted to viewing outdoor scenes.
  • Another object of the invention seeks to provide a novel augmented reality method and system that are particularly inexpensive.
  • Another object of the invention seeks to provide a novel augmented reality method and system that enable particularly fluid augmented reality viewing to take place in real time and while on the move.
  • Another object of the invention is to propose a novel computer program adapted to execute the steps of an augmented reality method, said program consuming particularly little in terms of computer resources and providing a result that is extremely accurate and reliable.
  • an augmented reality method comprising at least:
  • a first rendering step in which the terminal displays said first image and substantially simultaneously issues at least some of the virtual information relating to at least one of the real objects present in said first image, and does so in a manner that is perceptible for a user;
  • a second capture step of capturing a second image of a second real scene using said terminal e) a second capture step of capturing a second image of a second real scene using said terminal; f) a movement measurement step of measuring the movement performed by the terminal between the first capture step and the second capture step; g) a determination step of determining which real objects ought to be assumed to be present in said second image on the basis of said movement performed by the terminal and on the basis of the information representative of the locations of real objects; and h) a second rendering step in which the terminal displays said second image and simultaneously issues at least some of the virtual information relating to at least one of the real objects assumed to be present in said second image, and does so in a manner that is perceptible to the user.
  • the objects given to the invention are also achieved by a computer program comprising computer program code means adapted to execute the steps of the above method.
  • an augmented reality system comprising at least one terminal that is designed to:
  • a data set that is an extract from a data collection stored in a main database, said main database including at least: information representative of the visual appearance of real objects forming part of a real environment, information representative of the locations of said real objects, and virtual information relating to said real objects;
  • FIG. 1 is a diagrammatic view of a terminal of a system in accordance with the invention, said terminal displaying in real time the image of a mountainous landscape with the names of certain mountains (X, Y, and Z) and their respective altitudes superimposed thereon; and
  • FIG. 2 is a diagram showing a system in accordance with the invention comprising a terminal identical to that shown in Figure 1 (but here shown from the side) and a server that communicates with the terminal over a wireless connection.
  • the invention relates to an augmented reality method, i.e. a method designed to superpose contextual information (metadata) on a real image (still or moving) of a real scene from which said information is absent, so as to provide additional information to a user observing said real image.
  • the augmented reality method in accordance with the invention thus enables composite information to be rendered to the user that is the result of combining a real image (of the photograph or video type) together with virtual information, e.g. of a visible and/or audible nature, which information enhances the real image.
  • the augmented reality method in accordance with the invention comprises at least a first step a) of capturing a first image 1 of a first real scene 2 using a terminal 3.
  • the terminal 3 is advantageously fitted with a camera 4, such as a digital camera.
  • the camera 4 thus enables pictures to be taken of a real scene either singly (like a still camera) or in sequence (like a motion picture camera).
  • the camera 4 may be fitted with an array of sensors of the charge-coupled device (CCD) or complementary metal-oxide-on-silicon (CMOS) type for converting light into an electrical signal, together with an analog-to-digital converter for digitizing the electrical signal and for obtaining therefrom a digital image that faithfully reproduces the visual appearance of the first real scene 2.
  • the terminal 3 is fitted with a screen 5 for displaying said first image 1 obtained using the camera 4 for the attention of the user.
  • the screen 5 is in the form of a substantially rectangular slab (generally a liquid crystal display (LCD) slab) and is preferably suitable for producing a color display.
  • the screen 5 is a touch-sensitive screen that thus serves not only to perform its conventional display function, but also performs a pointer function by the user contacting the screen 5 with a finger or with a stylus handled by the user.
  • the screen 5 occupies a mean plane represented by dashed line 5A in Figure 2, while the camera 4 presents an optical axis X-X' that is substantially normal to said mean plane 5A occupied by the screen 5.
  • the user of the terminal 3 can see on the screen 5 the image of the real scene to which the camera 4 is pointing, which is found to be particularly practical and ergonomic for viewing augmented reality images.
  • the terminal 3 is designed to be held and manipulated in the hand by a user.
  • the terminal 3 advantageously constitutes a portable device that is sufficiently small and lightweight to be handled, preferably in one hand.
  • the terminal 3 is constituted by a personal digital assistant (PDA), or in even more preferred manner by a smart phone (preferably of the type sold under the registered trademark "iPhone").
  • PDA personal digital assistant
  • a smart phone preferably of the type sold under the registered trademark "iPhone”
  • the invention naturally also applies to an augmented reality system as such, comprising at least a terminal 3, preferably as described above, said terminal 3 being designed, as mentioned above, to take a first image 1 of a first real scene 2.
  • the augmented reality method in accordance with the invention also includes a step b) of selecting a data set as a function at least of the location of the terminal 3 at the time it captures the first image 1 , which data set is extracted from a data collection stored in a main database.
  • the selection step b) consists in detecting the geographical position of the terminal at the instant it captures the first image 1 and in selecting a data set from a larger data collection, the selection being performed as a function of the location of the terminal 3 as determined at the time the first image 1 is captured.
  • this selection step is preferably performed automatically by means of a computer program.
  • the data collection stored in the main database includes at least:
  • the data collection stored in the main database thus presents a data structure that is organized as at least three fields, namely a first information field representative of the visual appearance of real objects forming part of a real environment, a second field relating to information representative of the locations of said real objects, and a third field relating to virtual information about said real objects.
  • the selection step thus advantageously consists in detecting the geographical position of the terminal 3 substantially at the moment it captures the first image 1 , and then in searching for said instantaneous geographical position of the terminal 3 in the above-mentioned second field of the data collection stored in the main database, and finally in selecting a data set that is extracted from said data collection centered on the geographical position of the terminal 3.
  • the data set as selected in this way thus advantageously itself includes the following data, extracted from the data collection:
  • selection step b) is performed by the terminal 3 itself, which is thus designed, as a function at least of the location of the terminal 3 at the time it captures the first image 1 , to select a data set that is extracted from a data collection stored in a main database, said data collection including at least:
  • said main database is remote from the terminal 3 and is advantageously stored on a server 9 that is independent of and remote from the terminal 3.
  • the augmented reality system in accordance with the invention thus advantageously includes said server 9 hosting the main database.
  • the terminal 3 is designated to communicate via a wireless connection with said server 9, e.g. by radio type communication.
  • This facility for communication that advantageously exists between the terminal 3 and the server 9 enables data to be transferred between them.
  • Such a principle for providing communication between a portable terminal and a remote fixed server is itself known and it is implemented by most portable terminals of the smart phone type.
  • Such an architecture makes it possible to use the large capacity of a fixed server in terms of storage and computation, thus avoiding making excessive use of the more limited resources of the terminal 3.
  • the main database is advantageously remote from the terminal 3, the terminal including a local database of size that is generally smaller than that of the main database.
  • the location of the terminal 3 used in selection step b) preferably includes a step of determining the geodetic coordinates of the terminal 3.
  • determining the geodetic coordinates of the terminal 3 enables the position of the terminal 3 to be known at any point on the terrestrial globe. Once this geographical position of the terminal 3 is known, the selection step enables the small data set relating solely to the geographical zone in which the terminal 3 is located to be extracted from the data collection (advantageously covering the entire terrestrial globe).
  • the location of the terminal 3 used in the selection step b) is determined using a global positioning system (GPS) incorporated in the terminal 3.
  • GPS global positioning system
  • the terminal 3 forming part of the system in accordance with the invention thus advantageously includes a GPS system for locating said terminal 3, preferably in absolute manner on the terrestrial globe.
  • a GPS system for establishing the geographical location of the terminal 3.
  • said information representative of the locations of real objects 2A, 2B, and 2C comprises the geodetic coordinates of said real objects 2A, 2B, and 2C.
  • Selection step b) then advantageously includes an operation of comparing the instantaneous geodetic coordinates of the terminal 3 (at the moment it captures the first image 1 ) with the geodetic coordinates stored in the data collection housed in the main database.
  • selection step b) may consist in selecting the data collection hosted in the main database for the latitude, longitude, and altitude values that correspond to those of the instantaneous position of the terminal 3, and then extracting the data corresponding to said position and also to the neighborhood thereof.
  • the data set selected during the selection step b) is selected as a function not only of the location of the terminal 3 at the moment the first image 1 is captured, but also as a function of one or more other parameters, e.g. as a function of the orientation of the terminal 3 at the moment the first image 1 is captured.
  • the orientation of the terminal 3 is determined by the terminal 3 itself, which terminal is thus designed to select the data set as a function not only of its location at the moment the first image 1 is captured, but as a function of its orientation at the moment said first image 1 is captured.
  • Knowledge of the orientation of the terminal 3 which corresponds to knowledge of the orientation of the optical axis X-X' (and preferably of a direction along said axis), then makes it possible to reduce the size of the data set that is extracted from the data collection during selection step b), so as to retain only the most pertinent data, e.g. corresponding to an angular sector of at least 180 degrees centered on the optical axis X-X'.
  • the orientation of the terminal 3 used in selection step b) is determined using a magnetometer incorporated in the terminal 3. Nevertheless, using a terminal 3 that incorporates a magnetometer for determining the orientation of the terminal 3 is not absolutely essential, and it is entirely possible to envisage having recourse to means other than a magnetometer for determining the orientation of the terminal 3.
  • the augmented reality method in accordance with the invention further comprises a first comparison step c) of comparing the first image 1 of the first real scene 2 with the information representative of the visual appearance of real objects 2A, 2B, and 2C included in said selected data set, in order to determine whether at least one of said real objects is present in said first image 1.
  • said comparison step c) is performed by the terminal 3, which is thus advantageously designed to compare the first image 1 of the first real scene 2 with the information representative of the visual appearance of real objects 2A, 2B, and 2C included in the selected data set, in order to determine whether at least one of said real objects 2A, 2B, and 2C is present in said first image.
  • this method inc!udes, prior to the first comparison step c), a step of storing in the local database (advantageously incorporated in the terminal 3) the data set that was selected in selection step b).
  • the information representative of the visual appearance of real objects 2A, 2B, and 2C on the basis of which the first comparison step c) is to be performed is thus locally stored in the terminal 3, thereby increasing the speed of execution of the first comparison step c).
  • the terminal 3 thus incorporates a local database in which it stores the data set it selected as a function of its instantaneous position at the moment of capturing the first image 1 , prior to comparing the first image 1 of the first real scene 2 with the information representative of the visual appearance of real objects 2k, 2B, and 2C.
  • the first comparison step c) enables the terminal 3 to identify its surroundings accurately, relying not only on the location and orientation information of the terminal 3, but also on recognizing the visual appearance of the surroundings in question.
  • the first comparison step c) first comprises analyzing the first image 1 so as to identify in said first image 1 the real objects 2A, 2B, and 2C from their visual appearance.
  • said information representative of the visual appearance of real objects 2A, 2B, and 2C is representative of the shapes and/or the outlines of said real objects 2A, 2B, and 2C, with the first comparison step c) then including an operation of identifying the real objects 2A, 2B, and 2C present in the first image 1 by shape and/or outline recognition.
  • the augmented reality method in accordance with the invention makes it possible, on the basis of data contained in the data set extracted from the data collection stored in the main database, to recognize one or more real objects 2A, 2B, and 2C that are assumed to be present in the image 1 given the location and the orientation of the terminal 3 at the moment the first image 1 is captured, and do so on the basis of their shapes and/or outlines.
  • the terminal 3 is designed to identify real objects 2A, 2B, and 2C present in the first image 1 by shape and/or outline recognition, i.e.
  • the terminal 3 includes means, preferably software means, capable of analyzing the first image 1 to find the real objects 2A, 2B, and 2C that it is assumed to contain on the basis of knowledge of the information representative of the visual appearances of said real objects 2A, 2B, and 2C as contained in the data set selected during selection step b).
  • said information representative of the visual appearances of real objects 2A, 2B, and 2C is formed by a digital terrain model (DTM), i.e. by a digital data collection representative of the surface relief of the terrestrial globe and from which it is possible automatically to calculate the slopes, the crest lines, contour lines, etc.
  • DTM digital terrain model
  • the first comparison step c) advantageously comprises:
  • the first comparison step c) is performed entirely by the terminal 3 or within it, said terminal 3 thus being designed to:
  • the digital terrain mode! may be constituted by a set of adjacent rectangular cells (e.g. having dimensions of 38 kilometers (km) by 18 km) each formed by a matrix of points with an altitude value being associated with each of said points.
  • Recognizing outlines and/or crest lines present in the image 1 on the basis of the information in the data set selected as a function of the location of the terminal 3 enables the terminal to identify accurately the positions in the image 1 of various real objects 2A, 2B, and 2C (mountains, buildings, bodies of water, or any other element of the surroundings), thereby improving the accuracy, the speed, and the fluidity of the augmented reality rendered to the user.
  • the augmented reality method in accordance with the invention includes a first rendering step d) in which the terminal 3 displays, preferably on its screen 5, the first image 1 and issues substantially simultaneously at least some of the virtual information 6, 7, 8 relating to at least one of the real objects 2A, 2B, and 2C present in said first image 1 , and does so in a manner that is perceptible to a user.
  • the first rendering step d) thus consists in making perceptible to the user simultaneously both the first image 1 and virtual information 6, 7, 8 relating to the real objects 2A, 2B, and 2C that has been detected as being present in the first image 1 during the preceding steps a), b), and c).
  • said virtual information 6, 7, 8 is designed so as to be perceived visually by the user.
  • the virtual information 6, 7, 8 is thus of a visual nature, such that the first rendering step consists in displaying on the screen 5 of the terminal 3 the first image 1 with virtual information 6, 7, 8 superposed thereon relating to the real objects 2A, 2B, and 2C present in the first image 1.
  • the instantaneous overlay of virtual information 6, 7, 8 on the first image 1 the user has access to visual elements that are simulated (i.e. absence from the first real scene 2) and that serve to improve the user's perception of the first real scene 2.
  • the user obverses a real scene 2 constituted by mountain scenery through the terminal 3.
  • a real image (a photographic rendering) is displayed in real time on the screen 5, with this real image being enhanced, as shown in Figure 1 , by simulated visual elements (e.g. of toponymic and topographic type) constituted specifically by the names and the altitudes of the mountains observed by the user through the terminal 3.
  • simulated visual elements e.g. of toponymic and topographic type
  • the virtual information 6, 7, 8 is positioned on the screen 5 in extremely reliable and accurate manner relative to the positions on the same screen 5 of the real objects 2A, 2B, and 2C (here mountains) to which the virtual information 6, 7, 8 relates.
  • the time that elapses between the first capture step a) and the first rendering step d) is substantially imperceptible to the user, i.e. it is short enough to give the user the impression that it takes place instantaneously.
  • This speed with which the method runs between the first capture step c) and the first rendering step d) is made possible in particular by locally storing (in the terminal 3) a portion only of the data collection.
  • the augmented reality method in accordance with the invention further includes a second capture step e) of capturing a second image of a second real scene using the terminal 3.
  • This section capture step is advantageously similar to the first capture step, except that the second real scene differs from the first real scene 2.
  • the difference between the first real scene 2 and the second real scene may possibly result merely from changing the picture-taking angle, with the observed real objects 2A, 2B, and 2C remaining the same.
  • the first image 1 and the second image may be constituted by photographic views obtained as stills, or on the contrary they may form part of a set of views taken continuously with a video device.
  • the terminal 3 takes pictures permanently, as a video, thereby enabling the user to view, on the screen 5, continuously and in real time, any real scene lying within the capture field of the camera 4.
  • the screen 5 with which the terminal 3 is advantageously provided is thus designed also to display the second image, the terminal 3 preferably being designed to capture said second image of the second real scene using the same camera means 4 as described above.
  • the first image 1 and the second image 2 include at least one real element 2A, 2B, or 2C in common, i.e. said first and second images show real scenes that are geographically close.
  • the first and second real scenes thus both form portions of a single geographical zone of an extent that is defined, for example, substantially by the horizon seen through the camera 4.
  • the data set that is extracted from the data collection stored in the main database includes at least: information representative of the visual appearance of all of the real objects 2A, 2B, and 2C included in the geographical zone in question, information representative of the locations of said real objects 2A, 2B, and 2C, and virtual information about said real objects 2A, 2B, and 2C.
  • the augmented reality method in accordance with the invention also includes a movement measurement step f) of measuring the movement performed by the terminal 3 between the first capture step and the second capture step.
  • the terminal 3 is itself designed to measure the movement it performs between the first and second capture steps.
  • the measurement step f) is advantageously performed using an accelerometer incorporated in the terminal 3. Nevertheless, it is entirely possible to envisage the terminal 3 including some device for measuring the movement of the terminal other than an accelerometer as in the above-mentioned variant.
  • the method in accordance with the invention also includes a determination step g) of determining the real objects 2A, 2B, and 2C that should be assumed to be present in said second image on the basis of said movement performed by the terminal 3 and on the basis of the information representative of the locations of the real objects 2A, 2B, and 2C.
  • This step g) is advantageously performed by the terminal 3 itself, which may be designed to respond to the movement it performs between the first capture step and the second capture step and to the information representative of the locations of the real objects 2A, 2B, and 2C to determine which real objects 2A, 2B, and 2C should be assumed to be present in said second image.
  • Knowing the path it follows between the first capture step and the second capture step thus enables the terminal 3 to identify the real objects 2A, 2B, and 2C that ought theoretically to be present in its field of observation, and thus in the second image.
  • the determination step g) thus enables the terminal 3 to select data from the data set stored locally in the terminal 3 (or possibly from the data collection stored in the main database), which data relates to the real objects that ought to be in the second image given the movement to which the terminal 3 has been subjected between the first and second capture step.
  • the determination step g) thus does not make use of an absolute determination of the location of the terminal 3 as in step b), nor does it make use of a simple step of shape recognition as in the first comparison step c), but rather it makes use of a relative location step on the basis of an accurately documented starting situation as determined by the first capture step a) and the selection, comparison, and rendering steps b), c), and d). This serves to optimize the use made of the resources available in the terminal 3, and contributes to making the method fluid and fast.
  • the method in accordance with the invention including a step after the determination step g) but before the second comparison step i) that follows it (described in greater detail below), in which said second image of the second real scene is compared with the information representative of the visual appearance of said real objects 2A, 2B, and 2C in order to verify the actual locations of the real elements 2A, 2B, and 2C that are assumed to be present in the second image.
  • the second comparison step i) is substantially identical, at least in principle, to the first comparison step c) as described above. It serves to verify that the theoretical locations of the real objects that are assumed to be present in the second image are indeed true, and to make any adjustments that might be necessary if the match is not perfect.
  • This second comparison step i) may nevertheless be cut down relative to the first comparison step c), i.e. the processing of the second image in terms of shape recognition may be based on an algorithm that is less accurate, simplified (and therefore consumes less in terms of resources), since the essential purpose of the second comparison step i) is to confirm the content of the second image.
  • This second comparison step i) is advantageously performed by (or in) the terminal 3, preferably using suitable shape and/or outline recognition software.
  • the method in account with the invention includes a second rendering step h) in which the terminal 3 displays said second image, and substantially simultaneously issues at least some of the virtual information 6, 7, 8 relating to at least one of the real objects 2A, 2B, and 2C that are assumed to be present in said second image, and does so in a manner that is perceptible to the user.
  • This second rendering step h) is advantageously performed by the terminal 3, which is thus advantageously designed to display said second image (preferably on its screen 5) and substantially simultaneously to issue at least some of the virtual information 6, 7, 8 relating to at least one of the real objects 2A, 2B, and 2C that are assumed to be present in said second image as a result of the determination step g), and to do so in a manner that is perceptible to the user.
  • the second rendering step h) is advantageously similar to the first rendering step d), i.e. it preferably consists in displaying pertinent virtual information relating to the real objects 2A, 2B, and 2C present in the second image on the screen 5 of the terminal 3 superposed on said second image.
  • the user sees no difference of rendering, even though the means for rendering the second image with its virtual information overlaid thereon results from a process that is different from that implemented to display the first image with its virtual information overlaid thereon.
  • the process implemented for displaying the second image and its virtual information simultaneously and in overlay is thus much less demanding in resources than the process used to achieve superposed display of the first image with the corresponding pertinent virtual information.
  • the method in accordance with the invention thus relies on implementing a main process corresponding to steps a) to d) in which accuracy is optimized, thereby enabling the first image to be displayed with augmented reality, and a secondary process (or tracking process), corresponding to steps e) to h) that relies at least in part on the results of the main process in order to obtain an augmented reality display of a second image.
  • a secondary process or tracking process
  • steps e) to h that relies at least in part on the results of the main process in order to obtain an augmented reality display of a second image.
  • these two processes are particularly advantageous when the first and second images relate to real scenes that are geographically close, or at least to real scenes that do not require the terminal 3 to move through a large amplitude between the first capture step and the second capture step.
  • the method in accordance with the invention is particularly well suited to conventional observation in which the user begins by observing an initial given scene in a fixed and stable manner, and then moves the terminal 3 slowly, essentially with a panning movement, in order to observe the neighborhood of the initial scene.
  • the length of time that elapses between the second capture step e) and the second rendering step h) is substantially imperceptible to the user, in the same manner as for the duration that elapses between the first capture step a) and the first rendering step d), as mentioned above.
  • the augmented reality method is a real-time method, thereby enabling a user to observe various scenes in real time in successive and continuous manner while benefiting continuously and immediately from the display of simulated information elements relating to the real elements observed through the terminal 3.
  • the invention is advantageously implemented by computer means that control the operation of the terminal 3 and of peripherals incorporated therein (camera 4, magnetometer, accelerometer, screen 5, etc.).
  • the invention thus provides, in independent manner, a computer program having computer program code means adapted to execute the steps of a method in accordance with the above description when the program is executed on a computer, including on a pocket computer such as a smart phone or a PDA.
  • the computer program in question is recorded on a computer-readable medium, regardless of the nature of the medium in question, the essential point being that the medium on which the program is recorded can be read by a computer.
  • the invention finds industrial application in the design and implementation of an augmented reality method and system.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Computer Graphics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Geometry (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Remote Sensing (AREA)
  • Processing Or Creating Images (AREA)

Abstract

Augmented reality method and system for mobile terminals involving overlaying location specific virtual information into the real images of the camera of the mobile terminal. The virtual information and also visual information about the environment at the location of the terminal is selected and downloaded from a remote database server using as well the location (via GPS) and the orientation (via magnetometer, compass and accelerometer) of the mobile terminal. This information is continuously updated by measuring the movement of mobile terminal and by predicting the real image content. The outline of the captured scene (i.e. crest lines of mountains in the real camera images) is compared with the outline of a terrain model of the scene at the location of the mobile terminal.

Description

AN AUGMENTED REALITY METHOD, AND A CORRESPONDING SYSTEM
AND SOFTWARE
TECHNICAL FIELD
The present invention relates to the general technical field of digital imaging, and more precisely to augmented reality techniques enabling virtual elements to be combined with an image of a real scene.
The present invention relates more particularly to an augmented reality method, to a computer program including computer program code means suitable for executing the steps of such a method, to a medium capable of being read by a computer and having such a computer program recorded thereon, and to an augmented reality system.
PRIOR ART
Augmented reality methods and systems are already known. They enable a user to improve the perception of real surroundings by making it possible for the user to view said real surroundings while also perceiving additional information elements that are absent from the real scene. The concept of augmented reality has become increasingly widespread with the appearance and commercial success of portable electro-optical appliances of the portable digital assistant (PDA) or smart phone type.
In particular, a smart phone is known that, when associated with appropriate control software, enables the user to view the real image of the landscape surrounding the user on the screen of said smart phone together with overlaid information relating to the landscape in question (toponyms, etc.). Such augmented reality software for a smart phone makes use of the geographical position of the smart phone as determined by the GPS system incorporated therein, and also of topographic data stored in a database with which the smart phone communicates over a radio channel. Although such an augmented reality system generally gives satisfaction, it nevertheless presents certain drawbacks. Thus, the advantage of such an augmented reality application lies essentially in that it is intended to enable augmented viewing to take place in real time while the user is on the move. In other words the user must be capable of observing a landscape in real time on the smart phone screen while also seeing virtual elements that are displayed thereon as an overlay and in real time, which virtual elements augment the reality of the observed real scene. This augmented viewing in real time should ideally be continuous as the user varies the position and/or the orientation of the smart phone. Such functions require a large amount of computer resources, which are not necessarily available in all smart phones. Specifically because they are portable, smart phones are necessarily limited in terms of computer power and memory.
There is therefore a real risk of an augmented reality method of the kind mentioned above operating in a manner that is not sufficiently fluid, or even that interrupts in unwanted manner because of excessive requirements in terms of memory and/or computer power. OBJECTS AND SUMMARY OF THE INVENTION
The objects given to the invention consequently seek to remedy the above-mentioned drawbacks and to propose a novel augmented reality method and system that provide a real time result that is particularly fluid and reliable, including when the method is implemented on an appliance having limited computer resources. Another object of the invention seeks to provide a novel augmented reality method and system that are particularly accurate and reliable.
Another object of the invention seeks to provide a novel augmented reality method and system that can be implemented easily and under all circumstances.
Another object of the invention seeks to provide a novel augmented reality method and system that consume particularly little in terms of computer resources.
Another object of the invention seeks to provide a novel augmented reality method and system that are suitable for being implemented using commercially available means. Another object of the invention seeks to provide a novei augmented reality method and system that are particularly adapted to viewing outdoor scenes.
Another object of the invention seeks to provide a novel augmented reality method and system that are particularly inexpensive.
Another object of the invention seeks to provide a novel augmented reality method and system that enable particularly fluid augmented reality viewing to take place in real time and while on the move.
Another object of the invention is to propose a novel computer program adapted to execute the steps of an augmented reality method, said program consuming particularly little in terms of computer resources and providing a result that is extremely accurate and reliable.
The objects given to the invention are achieved by an augmented reality method comprising at least:
a) a first capture step of capturing a first image of a first real scene using a terminal;
b) a selection step of selecting a data set as a function of at least the location of the terminal at the moment the first image is captured, the data set comprising an extract from a data collection stored in a main database, said data collection including at least: information representative of the visual appearance of real objects forming part of a real environment, information representative of the locations of said real objects, and virtual information relating to said real objects;
c) a first comparison step of comparing said first image of the first real scene with information representative of the visual appearance of real objects included in said selected data set in order to determine whether at least one of said real objects is present in said first image;
d) a first rendering step in which the terminal displays said first image and substantially simultaneously issues at least some of the virtual information relating to at least one of the real objects present in said first image, and does so in a manner that is perceptible for a user;
e) a second capture step of capturing a second image of a second real scene using said terminal; f) a movement measurement step of measuring the movement performed by the terminal between the first capture step and the second capture step; g) a determination step of determining which real objects ought to be assumed to be present in said second image on the basis of said movement performed by the terminal and on the basis of the information representative of the locations of real objects; and h) a second rendering step in which the terminal displays said second image and simultaneously issues at least some of the virtual information relating to at least one of the real objects assumed to be present in said second image, and does so in a manner that is perceptible to the user. The objects given to the invention are also achieved by a computer program comprising computer program code means adapted to execute the steps of the above method.
The objects given to the invention are also achieved by a medium capable of being read by a computer and having recorded thereon the computer program in question.
Finally, the objects given to the invention are achieved by an augmented reality system comprising at least one terminal that is designed to:
- capture a first image of a first real scene;
- as a function of at least the location of the terminal at the moment the first image is captured, select a data set that is an extract from a data collection stored in a main database, said main database including at least: information representative of the visual appearance of real objects forming part of a real environment, information representative of the locations of said real objects, and virtual information relating to said real objects;
- compare said first image of the first real scene with the information representative of the visual appearance of real objects included in said selected data set in order to determine whether at least one of said real objects is present in said first image;
- display said first image and substantially simultaneously issue at least some of the virtual information relating to at least one of the real objects present in said first image, and do so in a manner that is perceptible to a user;
- capture a second image of a second real scene;
- measure the movement performed by the terminal between the first capture step and the second capture step;
- on the basis of said movement performed by the terminal and on the basis of information representative of the locations of the real objects determine which real objects should be assumed to be present in said second image; and
- display said second image and substantially simultaneously issue at least some of the virtual information relating to at least one of the real objects assumed to be present in said second image, and do so in a manner that is perceptible to the user.
BRIEF DESCRIPTION OF THE DRAWING
Other objects and advantages of the invention appear in greater detail on reading the following description with reference to the accompanying drawing, given purely by way of non-limiting illustration, and in which: - Figure 1 is a diagrammatic view of a terminal of a system in accordance with the invention, said terminal displaying in real time the image of a mountainous landscape with the names of certain mountains (X, Y, and Z) and their respective altitudes superimposed thereon; and
- Figure 2 is a diagram showing a system in accordance with the invention comprising a terminal identical to that shown in Figure 1 (but here shown from the side) and a server that communicates with the terminal over a wireless connection. BEST MANNER OF PERFORMING THE INVENTION
The invention relates to an augmented reality method, i.e. a method designed to superpose contextual information (metadata) on a real image (still or moving) of a real scene from which said information is absent, so as to provide additional information to a user observing said real image. The augmented reality method in accordance with the invention thus enables composite information to be rendered to the user that is the result of combining a real image (of the photograph or video type) together with virtual information, e.g. of a visible and/or audible nature, which information enhances the real image. The augmented reality method in accordance with the invention comprises at least a first step a) of capturing a first image 1 of a first real scene 2 using a terminal 3. For this purpose, the terminal 3 is advantageously fitted with a camera 4, such as a digital camera. The camera 4 thus enables pictures to be taken of a real scene either singly (like a still camera) or in sequence (like a motion picture camera). For this purpose, the camera 4 may be fitted with an array of sensors of the charge-coupled device (CCD) or complementary metal-oxide-on-silicon (CMOS) type for converting light into an electrical signal, together with an analog-to-digital converter for digitizing the electrical signal and for obtaining therefrom a digital image that faithfully reproduces the visual appearance of the first real scene 2. Advantageously, the terminal 3 is fitted with a screen 5 for displaying said first image 1 obtained using the camera 4 for the attention of the user. By way of example, the screen 5 is in the form of a substantially rectangular slab (generally a liquid crystal display (LCD) slab) and is preferably suitable for producing a color display. In particularly advantageous manner, the screen 5 is a touch-sensitive screen that thus serves not only to perform its conventional display function, but also performs a pointer function by the user contacting the screen 5 with a finger or with a stylus handled by the user.
Preferably, and as shown more particularly in Figure 2, the screen 5 occupies a mean plane represented by dashed line 5A in Figure 2, while the camera 4 presents an optical axis X-X' that is substantially normal to said mean plane 5A occupied by the screen 5. By means of this characteristic, the user of the terminal 3 can see on the screen 5 the image of the real scene to which the camera 4 is pointing, which is found to be particularly practical and ergonomic for viewing augmented reality images.
Advantageously, the terminal 3 is designed to be held and manipulated in the hand by a user. In other words, the terminal 3 advantageously constitutes a portable device that is sufficiently small and lightweight to be handled, preferably in one hand. In particularly advantageous manner, the terminal 3 is constituted by a personal digital assistant (PDA), or in even more preferred manner by a smart phone (preferably of the type sold under the registered trademark "iPhone"). The invention naturally also applies to an augmented reality system as such, comprising at least a terminal 3, preferably as described above, said terminal 3 being designed, as mentioned above, to take a first image 1 of a first real scene 2.
The augmented reality method in accordance with the invention also includes a step b) of selecting a data set as a function at least of the location of the terminal 3 at the time it captures the first image 1 , which data set is extracted from a data collection stored in a main database. In other words, the selection step b) consists in detecting the geographical position of the terminal at the instant it captures the first image 1 and in selecting a data set from a larger data collection, the selection being performed as a function of the location of the terminal 3 as determined at the time the first image 1 is captured. Naturally, this selection step is preferably performed automatically by means of a computer program. The data collection stored in the main database includes at least:
- information representative of the visual appearance of real objects 2A, 2B, and 2C forming part of a real environment;
- information representative of the locations of said real objects 2A, 2B, and 2C; and
- virtual information 6, 7, and 8 relating to said real objects 2A, 2B, and 2C.
The data collection stored in the main database thus presents a data structure that is organized as at least three fields, namely a first information field representative of the visual appearance of real objects forming part of a real environment, a second field relating to information representative of the locations of said real objects, and a third field relating to virtual information about said real objects. The selection step thus advantageously consists in detecting the geographical position of the terminal 3 substantially at the moment it captures the first image 1 , and then in searching for said instantaneous geographical position of the terminal 3 in the above-mentioned second field of the data collection stored in the main database, and finally in selecting a data set that is extracted from said data collection centered on the geographical position of the terminal 3. The data set as selected in this way thus advantageously itself includes the following data, extracted from the data collection:
- information representative of the visual appearance of real elements 2A, 2B, and 2C included in a geographical zone covering said first real scene 2;
- information representative of the locations of said real elements 2A, 2B, and 2C, said information representative of locations including data characteristic of the location of the terminal 3 at the time the first image 1 is captured; and
- virtual information 6, 7, and 8 relating to said real elements 2A, 2B, and 2C included in the geographical zone covering said first real scene 2.
Advantageously, selection step b) is performed by the terminal 3 itself, which is thus designed, as a function at least of the location of the terminal 3 at the time it captures the first image 1 , to select a data set that is extracted from a data collection stored in a main database, said data collection including at least:
- information representative of the visual appearance of real objects 2A, 2B, and 2C forming part of a real environment;
information representative of the locations of said real objects 2A, 2B, and 2C; and
- virtual information 6, 7, and 8 relating to said real objects 2A, 2B, and 2C.
Preferably, said main database is remote from the terminal 3 and is advantageously stored on a server 9 that is independent of and remote from the terminal 3. The augmented reality system in accordance with the invention thus advantageously includes said server 9 hosting the main database. Advantageously, the terminal 3 is designated to communicate via a wireless connection with said server 9, e.g. by radio type communication. This facility for communication that advantageously exists between the terminal 3 and the server 9 enables data to be transferred between them. Such a principle for providing communication between a portable terminal and a remote fixed server is itself known and it is implemented by most portable terminals of the smart phone type. Such an architecture makes it possible to use the large capacity of a fixed server in terms of storage and computation, thus avoiding making excessive use of the more limited resources of the terminal 3. Thus, the main database is advantageously remote from the terminal 3, the terminal including a local database of size that is generally smaller than that of the main database. The location of the terminal 3 used in selection step b) preferably includes a step of determining the geodetic coordinates of the terminal 3. In this advantageous implementation, determining the geodetic coordinates of the terminal 3 enables the position of the terminal 3 to be known at any point on the terrestrial globe. Once this geographical position of the terminal 3 is known, the selection step enables the small data set relating solely to the geographical zone in which the terminal 3 is located to be extracted from the data collection (advantageously covering the entire terrestrial globe).
This makes it possible, during subsequent steps of the method, to avoid any need to process a large volume of data, most of which is of no pertinence since it does not relate to the geographical zone corresponding to the instantaneous position of the terminal 3. Preferably, the location of the terminal 3 used in the selection step b) is determined using a global positioning system (GPS) incorporated in the terminal 3. The terminal 3 forming part of the system in accordance with the invention thus advantageously includes a GPS system for locating said terminal 3, preferably in absolute manner on the terrestrial globe. Naturally, it is possible to envisage making use of means other than a GPS system for establishing the geographical location of the terminal 3. In this preferred implementation, said information representative of the locations of real objects 2A, 2B, and 2C (included in the data collection preferably hosted on the server 9) comprises the geodetic coordinates of said real objects 2A, 2B, and 2C. Selection step b) then advantageously includes an operation of comparing the instantaneous geodetic coordinates of the terminal 3 (at the moment it captures the first image 1 ) with the geodetic coordinates stored in the data collection housed in the main database. For example, selection step b) may consist in selecting the data collection hosted in the main database for the latitude, longitude, and altitude values that correspond to those of the instantaneous position of the terminal 3, and then extracting the data corresponding to said position and also to the neighborhood thereof.
In order to reduce the size of the data set selected during the selection step b) to as small as possible, thereby contributing to the speed and the effectiveness of the method, it is advantageous during selection step b) for the data set to be selected as a function not only of the location of the terminal 3 at the moment the first image 1 is captured, but also as a function of one or more other parameters, e.g. as a function of the orientation of the terminal 3 at the moment the first image 1 is captured. Preferably, the orientation of the terminal 3 is determined by the terminal 3 itself, which terminal is thus designed to select the data set as a function not only of its location at the moment the first image 1 is captured, but as a function of its orientation at the moment said first image 1 is captured. Knowledge of the orientation of the terminal 3, which corresponds to knowledge of the orientation of the optical axis X-X' (and preferably of a direction along said axis), then makes it possible to reduce the size of the data set that is extracted from the data collection during selection step b), so as to retain only the most pertinent data, e.g. corresponding to an angular sector of at least 180 degrees centered on the optical axis X-X'.
Advantageously, the orientation of the terminal 3 used in selection step b) is determined using a magnetometer incorporated in the terminal 3. Nevertheless, using a terminal 3 that incorporates a magnetometer for determining the orientation of the terminal 3 is not absolutely essential, and it is entirely possible to envisage having recourse to means other than a magnetometer for determining the orientation of the terminal 3.
The augmented reality method in accordance with the invention further comprises a first comparison step c) of comparing the first image 1 of the first real scene 2 with the information representative of the visual appearance of real objects 2A, 2B, and 2C included in said selected data set, in order to determine whether at least one of said real objects is present in said first image 1. Preferably, said comparison step c) is performed by the terminal 3, which is thus advantageously designed to compare the first image 1 of the first real scene 2 with the information representative of the visual appearance of real objects 2A, 2B, and 2C included in the selected data set, in order to determine whether at least one of said real objects 2A, 2B, and 2C is present in said first image. In order to implement the first comparison step c) as quickly and as effectively as possible, thereby contributing to the fluidity of the method, this method inc!udes, prior to the first comparison step c), a step of storing in the local database (advantageously incorporated in the terminal 3) the data set that was selected in selection step b). The information representative of the visual appearance of real objects 2A, 2B, and 2C on the basis of which the first comparison step c) is to be performed is thus locally stored in the terminal 3, thereby increasing the speed of execution of the first comparison step c). In this advantageous implementation, the terminal 3 thus incorporates a local database in which it stores the data set it selected as a function of its instantaneous position at the moment of capturing the first image 1 , prior to comparing the first image 1 of the first real scene 2 with the information representative of the visual appearance of real objects 2k, 2B, and 2C.
The first comparison step c) enables the terminal 3 to identify its surroundings accurately, relying not only on the location and orientation information of the terminal 3, but also on recognizing the visual appearance of the surroundings in question. The first comparison step c) first comprises analyzing the first image 1 so as to identify in said first image 1 the real objects 2A, 2B, and 2C from their visual appearance. Preferably, said information representative of the visual appearance of real objects 2A, 2B, and 2C is representative of the shapes and/or the outlines of said real objects 2A, 2B, and 2C, with the first comparison step c) then including an operation of identifying the real objects 2A, 2B, and 2C present in the first image 1 by shape and/or outline recognition. In other words, the augmented reality method in accordance with the invention makes it possible, on the basis of data contained in the data set extracted from the data collection stored in the main database, to recognize one or more real objects 2A, 2B, and 2C that are assumed to be present in the image 1 given the location and the orientation of the terminal 3 at the moment the first image 1 is captured, and do so on the basis of their shapes and/or outlines. Preferably, the terminal 3 is designed to identify real objects 2A, 2B, and 2C present in the first image 1 by shape and/or outline recognition, i.e. the terminal 3 includes means, preferably software means, capable of analyzing the first image 1 to find the real objects 2A, 2B, and 2C that it is assumed to contain on the basis of knowledge of the information representative of the visual appearances of said real objects 2A, 2B, and 2C as contained in the data set selected during selection step b). Preferably, said information representative of the visual appearances of real objects 2A, 2B, and 2C is formed by a digital terrain model (DTM), i.e. by a digital data collection representative of the surface relief of the terrestrial globe and from which it is possible automatically to calculate the slopes, the crest lines, contour lines, etc. Under such circumstances, corresponding to that which is shown in Figure 1 , the first comparison step c) advantageously comprises:
an operation of analyzing the first image 1 to identify at least one real crest line in the real scene 2; and
an operation of comparing the shape of said real crest line with the shape of a model crest line obtained from said digital terrain model.
Preferably, the first comparison step c) is performed entirely by the terminal 3 or within it, said terminal 3 thus being designed to:
analyze the first image 1 to identify at least one real crest line in the rea! scene 2; and compare the shape of said real crest line with the shape of a model crest line obtained from the digital terrain model.
For example, the digital terrain mode! may be constituted by a set of adjacent rectangular cells (e.g. having dimensions of 38 kilometers (km) by 18 km) each formed by a matrix of points with an altitude value being associated with each of said points.
Recognizing outlines and/or crest lines present in the image 1 on the basis of the information in the data set selected as a function of the location of the terminal 3 enables the terminal to identify accurately the positions in the image 1 of various real objects 2A, 2B, and 2C (mountains, buildings, bodies of water, or any other element of the surroundings), thereby improving the accuracy, the speed, and the fluidity of the augmented reality rendered to the user. In this respect, the augmented reality method in accordance with the invention includes a first rendering step d) in which the terminal 3 displays, preferably on its screen 5, the first image 1 and issues substantially simultaneously at least some of the virtual information 6, 7, 8 relating to at least one of the real objects 2A, 2B, and 2C present in said first image 1 , and does so in a manner that is perceptible to a user. The first rendering step d) thus consists in making perceptible to the user simultaneously both the first image 1 and virtual information 6, 7, 8 relating to the real objects 2A, 2B, and 2C that has been detected as being present in the first image 1 during the preceding steps a), b), and c). Preferably, said virtual information 6, 7, 8 is designed so as to be perceived visually by the user. In this implementation, which corresponds to that shown in the figures, the virtual information 6, 7, 8 is thus of a visual nature, such that the first rendering step consists in displaying on the screen 5 of the terminal 3 the first image 1 with virtual information 6, 7, 8 superposed thereon relating to the real objects 2A, 2B, and 2C present in the first image 1. Thus, by the instantaneous overlay of virtual information 6, 7, 8 on the first image 1 , the user has access to visual elements that are simulated (i.e. absence from the first real scene 2) and that serve to improve the user's perception of the first real scene 2. Thus, in the example shown in Figure 1 , the user obverses a real scene 2 constituted by mountain scenery through the terminal 3. A real image (a photographic rendering) is displayed in real time on the screen 5, with this real image being enhanced, as shown in Figure 1 , by simulated visual elements (e.g. of toponymic and topographic type) constituted specifically by the names and the altitudes of the mountains observed by the user through the terminal 3. Because of the accuracy obtained by the first comparison step c), the virtual information 6, 7, 8 is positioned on the screen 5 in extremely reliable and accurate manner relative to the positions on the same screen 5 of the real objects 2A, 2B, and 2C (here mountains) to which the virtual information 6, 7, 8 relates. Advantageously, the time that elapses between the first capture step a) and the first rendering step d) is substantially imperceptible to the user, i.e. it is short enough to give the user the impression that it takes place instantaneously. This speed with which the method runs between the first capture step c) and the first rendering step d) is made possible in particular by locally storing (in the terminal 3) a portion only of the data collection.
This gives the user access in real time, i.e. substantially simultaneously with the terminal 3 capturing the first image 1 , to the content of the virtual information 6, 7, 8 that is displayed overlaying the first image 1 displayed by the screen 5. The augmented reality method in accordance with the invention further includes a second capture step e) of capturing a second image of a second real scene using the terminal 3. This section capture step is advantageously similar to the first capture step, except that the second real scene differs from the first real scene 2. The difference between the first real scene 2 and the second real scene may possibly result merely from changing the picture-taking angle, with the observed real objects 2A, 2B, and 2C remaining the same. The first image 1 and the second image may be constituted by photographic views obtained as stills, or on the contrary they may form part of a set of views taken continuously with a video device. Preferably, the terminal 3 takes pictures permanently, as a video, thereby enabling the user to view, on the screen 5, continuously and in real time, any real scene lying within the capture field of the camera 4. The screen 5 with which the terminal 3 is advantageously provided is thus designed also to display the second image, the terminal 3 preferably being designed to capture said second image of the second real scene using the same camera means 4 as described above.
Preferably, the first image 1 and the second image 2 include at least one real element 2A, 2B, or 2C in common, i.e. said first and second images show real scenes that are geographically close. Advantageously, the first and second real scenes thus both form portions of a single geographical zone of an extent that is defined, for example, substantially by the horizon seen through the camera 4. In particularly preferred manner, the data set that is extracted from the data collection stored in the main database includes at least: information representative of the visual appearance of all of the real objects 2A, 2B, and 2C included in the geographical zone in question, information representative of the locations of said real objects 2A, 2B, and 2C, and virtual information about said real objects 2A, 2B, and 2C. By virtue of this characteristic, the data relating to the second real scene is fully included in the data set that was selected in the selection step b), thereby improving the fluidity of the method, in particular when the data set in question is stored locally in the terminal 3.
The augmented reality method in accordance with the invention also includes a movement measurement step f) of measuring the movement performed by the terminal 3 between the first capture step and the second capture step. Preferably, the terminal 3 is itself designed to measure the movement it performs between the first and second capture steps. For this purpose, the measurement step f) is advantageously performed using an accelerometer incorporated in the terminal 3. Nevertheless, it is entirely possible to envisage the terminal 3 including some device for measuring the movement of the terminal other than an accelerometer as in the above-mentioned variant.
The method in accordance with the invention also includes a determination step g) of determining the real objects 2A, 2B, and 2C that should be assumed to be present in said second image on the basis of said movement performed by the terminal 3 and on the basis of the information representative of the locations of the real objects 2A, 2B, and 2C. This step g) is advantageously performed by the terminal 3 itself, which may be designed to respond to the movement it performs between the first capture step and the second capture step and to the information representative of the locations of the real objects 2A, 2B, and 2C to determine which real objects 2A, 2B, and 2C should be assumed to be present in said second image. Knowing the path it follows between the first capture step and the second capture step thus enables the terminal 3 to identify the real objects 2A, 2B, and 2C that ought theoretically to be present in its field of observation, and thus in the second image. The determination step g) thus enables the terminal 3 to select data from the data set stored locally in the terminal 3 (or possibly from the data collection stored in the main database), which data relates to the real objects that ought to be in the second image given the movement to which the terminal 3 has been subjected between the first and second capture step. The determination step g) thus does not make use of an absolute determination of the location of the terminal 3 as in step b), nor does it make use of a simple step of shape recognition as in the first comparison step c), but rather it makes use of a relative location step on the basis of an accurately documented starting situation as determined by the first capture step a) and the selection, comparison, and rendering steps b), c), and d). This serves to optimize the use made of the resources available in the terminal 3, and contributes to making the method fluid and fast. Nevertheless, it is entirely possible, and even preferred, to envisage the method in accordance with the invention including a step after the determination step g) but before the second comparison step i) that follows it (described in greater detail below), in which said second image of the second real scene is compared with the information representative of the visual appearance of said real objects 2A, 2B, and 2C in order to verify the actual locations of the real elements 2A, 2B, and 2C that are assumed to be present in the second image. Preferably, the second comparison step i) is substantially identical, at least in principle, to the first comparison step c) as described above. It serves to verify that the theoretical locations of the real objects that are assumed to be present in the second image are indeed true, and to make any adjustments that might be necessary if the match is not perfect.
This second comparison step i) may nevertheless be cut down relative to the first comparison step c), i.e. the processing of the second image in terms of shape recognition may be based on an algorithm that is less accurate, simplified (and therefore consumes less in terms of resources), since the essential purpose of the second comparison step i) is to confirm the content of the second image. This second comparison step i) is advantageously performed by (or in) the terminal 3, preferably using suitable shape and/or outline recognition software.
Finally, the method in account with the invention includes a second rendering step h) in which the terminal 3 displays said second image, and substantially simultaneously issues at least some of the virtual information 6, 7, 8 relating to at least one of the real objects 2A, 2B, and 2C that are assumed to be present in said second image, and does so in a manner that is perceptible to the user. This second rendering step h) is advantageously performed by the terminal 3, which is thus advantageously designed to display said second image (preferably on its screen 5) and substantially simultaneously to issue at least some of the virtual information 6, 7, 8 relating to at least one of the real objects 2A, 2B, and 2C that are assumed to be present in said second image as a result of the determination step g), and to do so in a manner that is perceptible to the user. The second rendering step h) is advantageously similar to the first rendering step d), i.e. it preferably consists in displaying pertinent virtual information relating to the real objects 2A, 2B, and 2C present in the second image on the screen 5 of the terminal 3 superposed on said second image. As a result, the user sees no difference of rendering, even though the means for rendering the second image with its virtual information overlaid thereon results from a process that is different from that implemented to display the first image with its virtual information overlaid thereon. The process implemented for displaying the second image and its virtual information simultaneously and in overlay is thus much less demanding in resources than the process used to achieve superposed display of the first image with the corresponding pertinent virtual information. The method in accordance with the invention thus relies on implementing a main process corresponding to steps a) to d) in which accuracy is optimized, thereby enabling the first image to be displayed with augmented reality, and a secondary process (or tracking process), corresponding to steps e) to h) that relies at least in part on the results of the main process in order to obtain an augmented reality display of a second image. Naturally, these two processes are particularly advantageous when the first and second images relate to real scenes that are geographically close, or at least to real scenes that do not require the terminal 3 to move through a large amplitude between the first capture step and the second capture step. If the real scenes corresponding respectively to the first and second images are too far apart, such that the terminal 3 needs to make a large movement in order to observe one after the other, then the data stored locally in the terminal 3 cannot be reused by the tracking process so the main process must be restarted. The same situation applies if the user causes the terminal 3 to move extremely suddenly, thereby preventing the terminal 3 from suitably detecting its own movement between the first and second capture steps.
In contrast, the method in accordance with the invention is particularly well suited to conventional observation in which the user begins by observing an initial given scene in a fixed and stable manner, and then moves the terminal 3 slowly, essentially with a panning movement, in order to observe the neighborhood of the initial scene. Advantageously, the length of time that elapses between the second capture step e) and the second rendering step h) is substantially imperceptible to the user, in the same manner as for the duration that elapses between the first capture step a) and the first rendering step d), as mentioned above.
By means of this characteristic, which is made possible in particular by the small processing requirements of the tracking process, the augmented reality method is a real-time method, thereby enabling a user to observe various scenes in real time in successive and continuous manner while benefiting continuously and immediately from the display of simulated information elements relating to the real elements observed through the terminal 3. Naturally, the invention is advantageously implemented by computer means that control the operation of the terminal 3 and of peripherals incorporated therein (camera 4, magnetometer, accelerometer, screen 5, etc.). The invention thus provides, in independent manner, a computer program having computer program code means adapted to execute the steps of a method in accordance with the above description when the program is executed on a computer, including on a pocket computer such as a smart phone or a PDA. Preferably, the computer program in question is recorded on a computer-readable medium, regardless of the nature of the medium in question, the essential point being that the medium on which the program is recorded can be read by a computer.
SUSCEPTIBILITY OF INDUSTRIAL APPLICATION
The invention finds industrial application in the design and implementation of an augmented reality method and system.

Claims

An augmented reality method comprising at least:
a) a first capture step of capturing a first image (1 ) of a first real scene (2) using a terminal (3);
b) a selection step of selecting a data set as a function of at least the location of the terminal (3) at the moment the first image (1 ) is captured, the data set comprising an extract from a data collection stored in a main database, said data collection including at least: information representative of the visual appearance of real objects (2A, 2B, 2C) forming part of a real environment, information representative of the locations of said real objects (2A, 2B, 2C), and virtual information (6, 7, 8) relating to said real objects (2A, 2B, 2C);
c) a first comparison step of comparing said first image of the first real scene (2) with information representative of the visual appearance of real objects (2A, 2B, 2C) included in said selected data set in order to determine whether at least one of said real objects (2A, 2B, 2C) is present in said first image (1 );
d) a first rendering step in which the terminal displays said first image (1 ) and substantially simultaneously issues at least some of the virtual information (6, 7, 8) relating to at least one of the real objects (2A, 2B, 2C) present in said first image (1 ), and does so in a manner that is perceptible for a user;
e) a second capture step of capturing a second image of a second real scene using said terminal (3);
f) a movement measurement step of measuring the movement performed by the terminal (3) between the first capture step and the second capture step;
g) a determination step of determining which real objects (2A, 2B, 2C) ought to be assumed to be present in said second image on the basis of said movement performed by the terminal (3) and on the basis of the information representative of the locations of real objects (2A, 2B, 2C); and
h) a second rendering step in which the terminal (3) displays said second image and simultaneously issues at least some of the virtual information (6, 7, 8) relating to at least one of the real objects (2A, 2B, 2C) assumed to be present in said second image, and does so in a manner that is perceptible to the user.
2. A method according to claim 1 , characterized in that the main database is remote from the terminal (3), the terminal incorporating a local database, said method including, prior to comparison step c), a step of storing in the local database said data set selected in selection step b).
3. A method according to claim 1 or claim 2, characterized in that it further comprises, prior to the second rendering step h), a second comparison step i) of comparing said second image of the second real scene with information representative of the visual appearance of said real objects (2A, 2B, 2C) in order to verify the actual locations of real elements (2A, 2B, 2C) that are assumed to be present in the second image.
4. A method according to any one of claims 1 to 3, characterized in that said terminal (3) is designed to be held and manipulated in the hand by the user.
5. A method according to any one of claims 1 to 4, characterized in that said terminal (3) is fitted with a camera (4) and a screen (5) for displaying said first and second images, said screen (5) extending in a mean plane (5A) and the camera (4) presenting an optical axis (Χ-Χ') that is substantially normal to the mean plane (5A) in which the screen (5) extends.
6. A method according to any one of claims 1 to 5, characterized in that the location of the terminal (3) used in selection step b) is determined by a step of determining the geodetic coordinates of the terminal (3).
7. A method according to claim 6, characterized in that the location of the terminal (3) used in selection step b), is determined by means of of a GPS system incorporated in the terminal (3).
8. A method according to any one of claims 1 to 7, characterized in that during selection step b), said data set is selected as a function not only of the location of the terminal (3) at the moment the first image (1) is captured, but also as a function of the orientation of the terminal (3) at the moment the first image (1) is captured.
9. A method according to claim 8, characterized in that the orientation of the terminal (3) used in selection step b) is determined with the help of a magnetometer incorporated in the terminal (3).
10. A method according to any one of claims 1 to 9, characterized in that said information representative of the locations of said objects (2A, 2B, 2C) comprises the geodetic coordinates of said real objects (2A, 2B, 2C).
11. A method according to any one of claims 1 to 10, characterized in that said information representative of the visual appearance of said real objects (2A, 2B, 2C) is representative of the shapes and/or the outlines of said real objects (2A, 2B, 2C), the first comparison step c) including an operation of identifying real objects (2A, 2B, 2C) present in the first image (1) by shape and/or outline recognition.
12. A method according to claim 11 , characterized in that said information representative of the visual appearance of real objects (2A, 2B, 2C) is constituted by a digital terrain model.
13. A method according to claim 12, characterized in that the first comparison step c) comprises:
- an operation of analyzing the first image (1 ) to identify at least one crest line in the real scene (2); and
- an operation of comparing the shape of said real crest line with the shape of a model crest line obtained from said digital terrain model.
14. A method according to any one of claims 1 to 13, characterized in that said virtual information (6, 7, 8) is designed to be perceived visually by the user.
15. A method according to any one of claims 1 to 14, characterized in that the measurement step f) is performed using an accelerometer incorporated in the terminal (3).
16. A method according to any one of claims 1 to 15, characterized in that the terminal (3) is designed to communicate over a wireless connection with a server (9) hosting the main database.
17. A method according to any one of claims 1 to 16, characterized in that the time that elapses between the first capture step a) and the first rendering step b) is substantially imperceptible to the user, as is the time that elapses between the second capture step e) and the second rendering step h).
18. A computer program including computer program code means adapted to execute the steps of a method according to any one of claims 1 to 17 when said program is executed on a computer.
19. A computer program according to claim 18, implemented on a medium that is readable by a computer.
20. A medium capable of being read by a computer and having a program according to claim 18 recorded thereon.
21. An augmented reality system comprising at least one terminal (3) that is designed to:
- capture a first image (1 ) of a first real scene (2);
- as a function of at least the location of the terminal (3) at the moment the first image (1) is captured, select a data set that is an extract from a data collection stored in a main database, said main database including at least: information representative of the visual appearance of real objects (2A, 2B, 2C) forming part of a real environment, information representative of the locations of said real objects, and virtual information (6, 1, 8) relating to said real objects (2A, 2B, 2C); - compare said first image (1 ) of the first real scene (2) with the information representative of the visual appearance of real objects included in said selected data set in order to determine whether at least one of said real objects is present in said first image (1 );
- display said first image (1) and substantially simultaneously issue at least some of the virtual information relating to at least one of the real objects (2A, 2B, 2C) present in said first image (1), and does so in a manner that is perceptible to a user;
- capture a second image of a second real scene;
- measure the movement performed by the terminal (3) between the first capture step and the second capture step;
- on the basis of said movement performed by the terminal (3) and on the basis of information representative of the locations of the real objects (2A, 2B, 2C) determine which real objects (2A, 2B, 2C) should be assumed to be present in said second image; and
- display said second image and substantially simultaneously issue at least some of the virtual information (6, 7, 8) relating to at least one of the real objects assumed to be present in said second image, and does so in a manner that is perceptible to the user.
22. A system according to claim 21 , characterized in that the terminal (3) incorporates a local database in which it stores said selected data set prior to comparing said first image (1) of the first real scene (2) with the information representative of the visual appearance of real objects.
23. A system according to claim 21 or claim 22, characterized in that said terminal (3) is designed to be held and manipulated in the hand by the user.
24. A system according to any one of claims 21 to 23, characterized in that said terminal (3) is fitted with a camera (4) and with a screen (5) for displaying said first and second images, said screen (5) extending in a mean plane (5A) and the camera (4) presenting an optical axis (Χ-Χ') that is substantially normal to said mean plane (5A) in which the screen (5) extends.
25. A system according to claim 24, characterized in that the terminal (3) includes a GPS system for locating said terminal (3).
26. A system according to any one of claims 21 to 25, characterized in that the terminal (3) is designed to select said data set as a function not only of the location of the terminal (3) at the moment the first image (1 ) is captured, but also as a function of the orientation of the terminal at the moment the first image (1) is captured, said terminal (3) including a magnetometer in order to determine the orientation of the terminal (3).
27. A system according to any one of claims 21 to 26, characterized in that said information representative of the visual appearance of real objects is representative of the shapes and/or the outlines of said real objects, said terminal being designed to identify the real objects (2A, 2B, 2C) present in the first image by shape and/or outline recognition.
28. A system according to claim 27, characterized in that said information representative of the visual appearance of real objects (2A, 2B, 2C) is formed by a digital terrain modem.
29. A system according to claim 28, characterized in that the terminal (3) is designed to:
- analyze the first image (1 ) to identify at least one real crest line in the real scene (2); and
- compare the shape of said crest line with the shape of a model crest line obtained from said digital terrain model.
30. A system according to any one of claims 21 to 29, characterized in that the terminal (3) includes an accelerometer for measuring the movement of the terminal (3).
31. A system according to any one of claims 21 to 30, characterized in that it includes a server (9) hosting the main database, the terminal (3) being designated to communicate via a wireless connection with said server (9).
PCT/IB2010/002032 2010-07-07 2010-07-07 An augmented reality method, and a corresponding system and software WO2012004622A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/IB2010/002032 WO2012004622A1 (en) 2010-07-07 2010-07-07 An augmented reality method, and a corresponding system and software

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/IB2010/002032 WO2012004622A1 (en) 2010-07-07 2010-07-07 An augmented reality method, and a corresponding system and software

Publications (1)

Publication Number Publication Date
WO2012004622A1 true WO2012004622A1 (en) 2012-01-12

Family

ID=43127639

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2010/002032 WO2012004622A1 (en) 2010-07-07 2010-07-07 An augmented reality method, and a corresponding system and software

Country Status (1)

Country Link
WO (1) WO2012004622A1 (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2728538A1 (en) * 2012-11-02 2014-05-07 Sony Corporation Method and system for providing content based on location data
KR20160020033A (en) * 2014-08-12 2016-02-23 전자부품연구원 Flight path guiding method based on augmented reality using mobile terminal
EP2685427A3 (en) * 2012-07-10 2017-01-25 LG Electronics, Inc. Mobile Terminal and Control Method Thereof
WO2018007245A1 (en) 2016-07-04 2018-01-11 Stamford Devices Limited An aerosol generator
CN110544311A (en) * 2018-05-29 2019-12-06 百度在线网络技术（北京）有限公司 Safety warning method, device and storage medium
WO2020136633A1 (en) 2018-12-26 2020-07-02 Elbit Systems Land And C4I Ltd. Methods and systems for camera 3d pose determination

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030160867A1 (en) * 2002-01-17 2003-08-28 Yasunori Ohto Information providing apparatus, information providing method, storage medium, and computer program

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030160867A1 (en) * 2002-01-17 2003-08-28 Yasunori Ohto Information providing apparatus, information providing method, storage medium, and computer program

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
MAHA EL CHOUBASSI ET AL: "An Augmented Reality Tourist Guide on Your Mobile Devices", ADVANCES IN MULTIMEDIA MODELING, SPRINGER BERLIN HEIDELBERG, BERLIN, HEIDELBERG, vol. 5916, 6 January 2010 (2010-01-06), pages 588 - 602, XP019137891, ISBN: 978-3-642-11300-0 *
T-MOBILE: "T-mobile G1 User Manual", 27 May 2009 (2009-05-27), XP002612874, Retrieved from the Internet <URL:http://support.t-mobile.com/knowbase/root/public/tm30234.pdf> [retrieved on 20101203] *

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2685427A3 (en) * 2012-07-10 2017-01-25 LG Electronics, Inc. Mobile Terminal and Control Method Thereof
KR101899977B1 (en) 2012-07-10 2018-09-19 엘지전자 주식회사 Mobile terminal and control method thereof
CN103942694A (en) * 2012-11-02 2014-07-23 索尼公司 Method and system for providing content based on location data
US9317173B2 (en) 2012-11-02 2016-04-19 Sony Corporation Method and system for providing content based on location data
CN103942694B (en) * 2012-11-02 2017-07-21 索尼公司 Method and system for providing content based on location data
EP2728538A1 (en) * 2012-11-02 2014-05-07 Sony Corporation Method and system for providing content based on location data
KR101994898B1 (en) 2014-08-12 2019-07-01 전자부품연구원 Flight path guiding method based on augmented reality using mobile terminal
KR20160020033A (en) * 2014-08-12 2016-02-23 전자부품연구원 Flight path guiding method based on augmented reality using mobile terminal
WO2018007245A1 (en) 2016-07-04 2018-01-11 Stamford Devices Limited An aerosol generator
CN110544311A (en) * 2018-05-29 2019-12-06 百度在线网络技术（北京）有限公司 Safety warning method, device and storage medium
CN110544311B (en) * 2018-05-29 2023-04-25 百度在线网络技术（北京）有限公司 Security warning method, device and storage medium
WO2020136633A1 (en) 2018-12-26 2020-07-02 Elbit Systems Land And C4I Ltd. Methods and systems for camera 3d pose determination
EP4296943A2 (en) 2018-12-26 2023-12-27 Elbit Systems Land and C4I Ltd. Methods and systems for camera 3d pose determination

Similar Documents

Publication Publication Date Title
US9454847B2 (en) System and method of indicating transition between street level images
US9525964B2 (en) Methods, apparatuses, and computer-readable storage media for providing interactive navigational assistance using movable guidance markers
US9269191B2 (en) Server, client terminal, system and program for presenting landscapes
US9836886B2 (en) Client terminal and server to determine an overhead view image
CN110555883A (en) repositioning method and device for camera attitude tracking process and storage medium
US20160042570A1 (en) Display control device, display control method, and program
US20090115862A1 (en) Geo-tagging of moving pictures
KR20110032765A (en) Method and apparatus for providing service using image recognition and sensor in portable terminal
KR20050078136A (en) Method for providing local information by augmented reality and local information service system therefor
US9600932B2 (en) Three dimensional navigation among photos
KR20120017293A (en) Device and method for providing augmented reality
CN107084740B (en) Navigation method and device
WO2012004622A1 (en) An augmented reality method, and a corresponding system and software
CN112749613A (en) Video data processing method and device, computer equipment and storage medium
JP2012038153A (en) Surrounding information retrieval device and surrounding information retrieval method
US9635234B2 (en) Server, client terminal, system, and program
WO2011096343A1 (en) Photographic location recommendation system, photographic location recommendation device, photographic location recommendation method, and program for photographic location recommendation
Luley et al. Mobile augmented reality for tourists–MARFT
CN108235764B (en) Information processing method and device, cloud processing equipment and computer program product
US10976179B1 (en) Geolocating contents of a video with device orientation, and application thereof
CN112887793A (en) Video processing method, display device, and storage medium
CN108574801A (en) A kind of image-pickup method and terminal
CN113599809A (en) Virtual map display method, device, equipment and storage medium
CN115796321A (en) Method, device and equipment for selecting address of storage site and storage medium
Tran A real-time mediated reality platform for outdoor navigation on mobile devices and wearable computers

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 10768056

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 10768056

Country of ref document: EP

Kind code of ref document: A1