WO2021090714A1

WO2021090714A1 - Information provision service program and information terminal device for information provision service

Info

Publication number: WO2021090714A1
Application number: PCT/JP2020/040022
Authority: WO
Inventors: 伊原　正典
Original assignee: 伊原　正典
Priority date: 2019-11-04
Filing date: 2020-10-25
Publication date: 2021-05-14
Also published as: WO2021090715A1

Abstract

[Problem] To implement a software, a program, and an information terminal device that enable an information terminal user, in using a position-linked information assistance system, to enjoy a versatile position-linked information assistance service without having to use a dedicated distribution server, carry out a complicated information user registration or information provider registration, or the like. [Solution] Markup language information publicly available on a network is collected on the basis of an information terminal device, information acquired as a collection result is selectively chosen, and the information is collected, divided, and shared in order to display AR information appropriate to the user, with the chosen information in a layout appropriate for display and matching the direction and position of the user information terminal. Thus, it is possible to implement an AR information user terminal device without carrying out a special user registration or information distributor registration, or using a dedicated information distribution system, and it is also possible to provide information acquired by a user information terminal to an intermediate information distribution device, thereby improving an information distribution service.

Description

Information provision service program and information terminal device for information provision service

The present invention relates to an information providing service program and an information terminal device for an information providing service.

The present invention refers to an arbitrary image captured by an imaging unit associated with an information terminal, including the latitude and longitude of the position detection unit (GNSS, GPS, quasi-zenith satellite, wireless LAN position estimation, acceleration sensor, etc.) of the information terminal. Linked to the direction detector (any method that detects the direction, direction, and rotation status of the information terminal by any method such as an electronic compass or electronic gyro sensor by geomagnetic detection) Then, a CG image is generated by a CG (computer graphic) generation unit as a processing unit that performs an image processing generation unit from the information configured by the calculation unit and the storage unit, and the captured image (real landscape, etc.) obtained by the imaging unit is generated. By synthesizing the CG image generated by the image generation unit with the CG image generated by the image generation unit (whether outdoors or indoors), the information terminal display unit displays the terminal position of the user information terminal, the imaging direction, and the three-dimensional imaging range based on the captured image. It is an invention in the operation of an information support device for urban / indoor / outdoor areas by a user information terminal by displaying a composite image of an captured image linked to a coordinate system and a CG image. Multiple applications have been filed.

More specific contents of the application described later are, for example, a service of acquiring advertisements / promotions from a predetermined distribution server with image features based on positions and performing advertisements / advertisements, or a situation in which the terminal is tilted. At the same time, it is a service that acquires the necessary information from the predetermined distribution server to synthesize the information in the imaging screen, or sets the CG information to synthesize based on the ID acquired from the two-dimensional marker and sets the necessary information. It is configured as a service acquired from a distribution server or a service that acquires management information on which object the CG to be synthesized is licensed from a predetermined server, and is used for those terminals. A special distribution device that provides the configured dedicated information to the user is required, and a dedicated server is installed for these services or a dedicated markup language is used.

JP-A-2002-027529 Japanese Unexamined Patent Publication No. 2010-118019 Japanese Unexamined Patent Publication No. 2013-149209 Japanese Unexamined Patent Publication No. 2013-196616 Japanese Unexamined Patent Publication No. 2014-142847 Japanese Unexamined Patent Publication No. 2019-020162 Japanese Patent Application Laid-Open No. 2014-505896 WO2013 / 108549

The main subject of the present invention is to provide a sufficient amount of content at the global level at the same time by utilizing the information on the existing network for the lacking content in the newly developed network service infrastructure. To be able to build a new network service infrastructure provision environment. More specifically, the problem to be solved by the present invention relates to services such as AR (Augmented Reality) / MR (Mixed Reality) that synthesizes a CG image with live-action information during shooting in conjunction with the position of the terminal described above. There is a problem that the user information terminal must receive the dedicated information according to the specifications of the service provider from the dedicated distribution system configured for distribution by the service provider and display it according to the specifications of the distributor. ..

In addition, for new outdoor information spaces where content is insufficient in new Internet services, servers as dedicated distribution devices can be installed, and information distribution environments using a dedicated markup language can be prepared. If it is a service for AR / MR as described in the prior art, there are existing markup languages such as ARML and KML as XML for AR space, and existing AR solutions such as Wikitube (registered trademark) are available. Since these dedicated markup languages are used, there is a problem that a number of new dedicated servers must be installed in the current situation where the dedicated markup languages are not widespread.

And, due to the essential problem that the content necessary for the implementation of such a service is not enough, the content and / or information for users sufficient for constructing the AR / MR information service system and / or the information service infrastructure by location interlocking. There is a fundamental problem of whether chicken (information user) or egg (information provider) is needed first, because it is not possible to prepare the service from the start of the service.

In order to solve the above-mentioned problems, the present invention does not rely only on the information provided by the above-mentioned service provider, but obtains information configured on a generally popular network based on the location information of the information terminal. CG image information generated by collecting, extracting necessary information from the collected information, and appropriately combining the extracted information on the user information terminal based on the imaging direction and tilt state of the user information terminal. By combining and displaying the information on the live image, even if the information provider does not have a special information distribution service system, the live image obtained from the user's terminal and the user's terminal can be used for the information terminal user. By providing the information configured for the user by synthesizing the CG image generated by reconstructing the information acquired by searching the existing network information based on the obtained location information, the real space By appropriately superimposing the information on the non-three-dimensional network on the above three-dimensional coordinate system, the problem that the dedicated information for the above-mentioned AR / MR service is insufficient is solved.

Then, by means of solving such a problem, sufficient contents and / or sufficient information for provision for configuring the information service system and / or information service infrastructure as the above-mentioned basic problem are provided from the start of service provision. It is possible to solve basic and fundamental problems by using it as a means to solve the problem of being unable to prepare. Then, according to the present invention, it is possible to generate information to be developed in an AR / MR space without content information by scraping processing or mashup processing from existing content information on the Internet, and further, according to the present invention. It is also possible to make the display information easier to handle as a processing system by converting and / or using it to ARML or KML at the time of mashup.

In addition, the mashup process collects predetermined contents into a predetermined format according to the search conditions used for the scraping process and the personal information of the user, and generates information that is likely to be given serendity to the user who specified the search conditions. It also includes information generation processing that can improve user convenience by generating recommendation information and personalized information, and specific mashup and recommendation methods are Various implementation methods such as collaborative filtering have been proposed, and the present invention is characterized in that it is carried out not on the server side but on the client user terminal.

Then, the information generated by the mashup is classified based on the position of the target information in the database (structure example is FIG. 6) in the user terminal, and further, the search condition information used for scraping and the mashup are used. Mashed-up information based on the search condition information used can be appropriately registered in the user information database (configuration example is FIG. 6), or image features obtained from image information or depth image information obtained by the imaging unit. If necessary, upload the mashed-up information as cache information to the information distribution device associated with the user information terminal, or download it from the information distribution device related to the cache information uploaded in the past. It may be configured to do so. In addition, the user can use it to generate CG information to be described later from the database in the distribution device (structure example is FIG. 6) associated with the user terminal database (structure example is FIG. 6) composed of these mashed-up information. The mashed-up information that meets the conditions specified from the information terminal may be acquired.

The CG information provided in this section and the above-mentioned section may be a two-dimensional image, a three-dimensional image, or a moving image, and the image format does not have to be a specific one. Instead, any other information such as voice may be provided at the same time, and of course, a method of giving different images to both eyes of the HMD (head mount display) and displaying them three-dimensionally may be taken, or a smartphone display may be used. It may be present, or by registering the mashed-up information in the database of the user terminal device or information distribution device (the structure is shown in Fig. 6), it can be used as a cached database system used for efficient information provision services. You can also do it.

Furthermore, the information terminal is positioned by its own information terminal device while maintaining sufficient privacy without receiving exclusive service provision by acquiring information from a special information distribution device and without registering as a user. By acquiring appropriate information support based on any information distribution device, it is possible to receive more convenient location-linked information support and, if necessary, information sharing with other information terminals and information mediation distribution devices. By doing this, you can enjoy more convenient services. It is also possible for these information mediation distribution devices and user information terminals to be able to enjoy decentralized services by configuring a grid communication environment.

According to the present invention, in a new network service infrastructure, it is not necessary to use the information on the existing Internet and prepare the information configured exclusively for the new technology, and the new network service infrastructure is available at the global level. It is possible to provide sufficient content to the Internet. Then, the service application user of the present invention can enjoy the optimum position-linked information support service without special user registration, and the information sender who wants to provide information by the service is also a user to the special position-linked information distribution system. Since it is possible to provide information to service application users without registration, distribution information registration, or equipment installation, a highly convenient location-linked service can be sent without user registration or distributor registration for the location-linked service. It has the effect of enabling people and users to provide and enjoy information. In addition, as a side effect, the service provider does not have to bother to install a dedicated distribution server for distribution to the user information terminal, and a virtual CG object is synthesized in an appropriate real space 3D coordinate system. It has the advantage of being able to be displayed.

In this way, it is possible to expand the existing content information on the Internet to the AR / MR space where there is no content information by the scraping process and mashup process linked to the position, and it is possible to create a new outdoor information space where the content is insufficient. On the other hand, without installing servers as a dedicated distribution device or preparing information distribution information and distribution environment using a dedicated markup language, suddenly provide sufficient content at the world level. Not only can it be possible, but it is also possible to define the configuration of the computer graphic for displaying the user information terminal at the time of mashup processing.

There are existing markup languages such as ARML and KML as XML for AR space, but the present invention has an essential feature in that such a dedicated processing system or distribution system is not particularly required, and the existing markup language is available. Since Wikitude and the like use these dedicated markup languages, they are not widely used at present. Further, the display information may be easily handled as a processing system by converting it into ARML or KML at the time of mashup of the present invention, or it may be written in a markdown language and an effect may be added to the display content.

Further, even when the information terminal and the service information provision distribution system cooperate to provide information, the information provision applicant does not register the predetermined information in the information provision distribution system in the service information provision distribution system. There is an effect that the convenience of being able to provide service information to information terminal users can be enjoyed. In order to make these effects optimal and reliable, various techniques such as statistical processing, machine learning, and AI methods such as deep learning may be combined to improve the effects of the present invention, or AR. It is also possible to realize public notice by / MR.

Then, as described in the above section, according to the present invention, in a service system that provides information for synthesizing a real landscape image and a CG image linked to the position and direction of an information terminal, one's own according to the position. Distribution of information distribution system for users who want to distribute their own information to users and user information terminals who want to enjoy the information for composing CG images on the user information terminal. By realizing a service that synthesizes CG images etc. into the actual landscape and provides information to the user terminal without registering the user in the system, the problem of chickens and eggs, whether the infrastructure comes first or the content comes first, is fundamental. Solving the problems has the effect of the invention that the convenience of the service system in the present invention is greatly improved. In the case of software that implements the present invention, it may be stored in a storage medium associated with a storage unit or the like, and as a result, content enhancement in a new environment with little content can be realized.

An example of a device configuration of a single user information terminal used as a client is shown. An example of a device configuration of an information distribution device used as a distribution device is shown. The processing procedure according to the present invention by a single information terminal is shown. The processing procedure according to the present invention by interlocking the information distribution server and the information terminal is shown. An example of the system processing procedure by linking the user information terminal for the client and the information distribution server is shown. An example of table configuration when the position-linked information is configured as a database is shown.

First, the procedure of what kind of processing is performed in the position / direction-linked information providing application software and the user information terminal performed by the user information terminal alone will be described step by step.

First, the user information processing terminal (FIG. 1-1001) handled by the user will be described. An HMD (head mounted display) is connected not only to the user information processing terminal device of the present invention, more specifically to an information processing device such as a mobile phone, a smartphone, a tablet, or a notebook PC, but also to an appropriate information processing device. It may be the one in which the information processing device is built in the HMD. Then, the position detection unit of the information terminal (GPS as GNSS, quasi-zenith satellite, DGPS, eLORAN, wireless LAN position estimation, arbitrary method capable of acquiring arbitrary position information including latitude and longitude such as acceleration sensor, etc. ) And the direction detection unit (an arbitrary method for detecting the direction, direction, and rotation status of the information terminal by any method such as an electronic compass (1064) by geomagnetic detection or an electronic gyro sensor (1062)) A CG image is generated by a CG (computer graphic) generation unit as a processing unit that performs an image processing generation unit and the like from information configured by the storage unit and an image pickup unit (1030) obtained by the image pickup unit (1030). Yes, it does not matter whether it is outdoors or indoors, and it may also include distance image information as various depth sensors [ToF "Time-of-Flight", structure light, multi-camera, LiDAR "Laser Imaging Detection and Ringing"]). By synthesizing the feature information consisting of the image and / or the distance image and the CG image generated by the image generation unit, the information terminal display unit is displayed as a tertiary based on the terminal position of the user information terminal, the imaging direction, and the imaging range of the captured image. This is an invention in the operation of an information support device for urban / indoor / outdoor areas by a user information terminal by displaying a composite image of an captured image and a CG image linked to the original coordinate system.

Further, these information terminals are electronic devices and structure lights that can acquire an image generally called an image sensor, a three-dimensional object image sensor device that can image a three-dimensional object by capturing the reflection of LiDAR, a depth camera, and a depth sensor. , An image sensor (1030) using an image sensor other than visible light, an image sensor including a depth image, a GPS (1061), a gyro (1062), and acceleration as various sensors in general. (1063), electronic compass (1064), GNSS, ambient brightness, various detectors (1060) configured by any sensing device that can electronically collect information around various terminals such as voice input microphones. Is also good. Then, based on the information obtained from these image pickup elements and sensors, the calculation unit (1010) linked with the storage unit (1020) performs calculation processing.

In the storage unit (1020), storage using an arbitrary storage medium such as a volatile memory (1021) such as a RAM or a cache memory, a ROM, a flash memory, a ReRAM, or a non-volatile memory (1022) such as an HDD or SSD as an external storage. The imaging unit (1030) may be configured as a unit, or may be provided with a ToF as a depth sensor, a structure light, a multi-camera, a light coding, a rider, an infrastructure depth, and the like.

Further, the display unit (1040) displays information on the display unit (1040) based on the information processed by the storage unit and the calculation unit described above, and the image generation unit (1070) is based on the processed information described above. ) May be used to generate display information such as 3DCG, 2D graphics, and moving images, and the display unit (1040) may be configured to display visual information according to the user's terminal. Specifically, if it is a smartphone, an arbitrary display such as a 3D display may be performed on the screen of the smartphone, and if it is an HMD (head mounted display).

Furthermore, the communication unit (1050) can cooperate and exchange information by communicating with other information processing devices regardless of whether it is wired or wireless, and generally, wireless LAN, wired LAN, or PAN (personal). -Area network), NFC (Near Field Communication) may be configured, or one that can communicate with any short-range small device or one that can communicate with a non-powered small device.

Further, the operation unit (1080) may be a touch panel, a cursor key, a physical keyboard, a touch pad, or a gesture recognition device such as an image pickup or an operation combination on the touch pad. ..

Then, as the first step of the present invention, the user information terminal has a position detection unit and / or an acceleration output unit and / or a direction (direction (1064) or gyro sensor (1062) by a geomagnetic sensor as an electronic compass) attached to the terminal. The step (3001) of generating terminal position-related information is carried out by specifying the current position and direction (direction) of the information terminal by various detection units (1060) such as a detection unit (relative direction change due to). Here, the device for detecting the orientation may be configured so that a more accurate orientation can be extracted by, for example, an electronic compass (1064) or a cooperation between the electronic compass and the gyro (1062). The acceleration detection unit (1063) may be able to detect the inclination by detecting the downward direction of the user terminal from the gravitational acceleration, or detect the moving speed of the user terminal by changing the acceleration.

More specifically, the various detection units (1060) as the above-mentioned position detection units include GNSS, GPS (1061), A-GPS, position identification by WiFi, and wireless communication base stations (for example, mobile phone line base stations). Various methods can be used as long as the location can be identified by any method such as latitude / longitude, address information by latitude / longitude, postal code, telephone number (for example, area code). Conversely, the latitude and longitude may be calculated by extracting the address, postal code, etc. from the text information. Tilt detection can be realized by a gyro sensor (1062), an acceleration sensor (1063), etc., and is the user terminal showing the upper floors of the building? It is possible to make an appropriate display on the display unit (1040) of the user information terminal (1001). Further, if the terminal seems to be moving, it is possible to continuously perform position correction and the like using an acceleration sensor. In this way, the user terminal first carries out the step (3001) of acquiring the current position information.

Subsequently, the user information terminal performs a step (3002) of capturing an image around the terminal using the imaging unit (1030). Then, a feature extraction step (3003) is performed in which the captured image information is acquired in the storage unit (1020), and the features are extracted from the acquired image information by the calculation in the calculation unit (1010). When performing this feature extraction step (3003), in the case of a specification that the image generation unit (1070) can perform arithmetic storage processing other than the purpose of CG generation called GPGPU (General-Purpose computing on Graphics Processing Units) for the calculation for feature extraction. If so, it is also possible to perform feature extraction using the storage unit and calculation unit configured in the image generation unit (1070), and in this image feature extraction step, the depth image information obtained by a depth camera or the like can be used. It may include the generated depth information, and if it is other than GPGPU, it is further provided with a calculation unit for realizing a matrix or neural network processing by analog calculation processing using a tensor processor dedicated to tensor processing or ReRAM. You may also perform feature extraction as an application such as machine learning.

Next, as a target selection information input step for selecting the target of the desired information, tap the user interface such as the icon displayed on the screen from the operation unit (1080) combined with the display unit such as a touch screen to input. By doing something, entering appropriate text as a keyword, or something that is contraindicated for the user (it is religious (such as Haram)), by making recommendations or curations based on the information obtained from the application or server. A step (3004) is performed in which menus and icons for configuring personalized information are selected and operated to set search target selection information related to search conditions. This operation unit (1080) is other than a touch screen. In some cases, it may be a cursor operation key or a physical keyboard. And when the input from the operation unit consists of a GUI and a touch screen, operations such as text input by tapping, icon selection, and menu selection are performed. By doing so, it is possible to configure instruction information based on the user's operation.

The two steps of the imaging step (3002) and the imaging feature extraction step (3003), the sensor information acquisition step (3001), and the user's operation input step affect the entire processing even if the order is different. Since it does not work, there is no problem in configuring it arbitrarily. In this image feature extraction step, depth information generated by depth image information obtained by a depth camera or the like may be included.

Communication times in which any search can be performed based on any combination of the three pieces of information of the user information terminal position information and / or image feature information and / or search target selection information acquired in the above four steps. The step (3005) of providing the position information and / or the image feature information and / or the search target selection information of the user information terminal to an arbitrary information providing distribution device connected on the line and requesting the search is carried out. .. These steps are steps in which the communication unit (1050) transmits information for requesting acquisition of position-linked information.

Then, based on each information provided by the user information terminal in the search request step (3005), an external search for information linked to the search request step (3005) is performed for any information distribution device connected by a communication line. The step (3101) performed by the information distribution device of the above is carried out. After that, the user information terminal carries out a step (3006) of receiving the information group obtained according to the step (3101) performed by the information distribution device by the communication unit (1050). These steps are the steps for receiving the position interlocking information. At this time, the information group based on the acquired position-linked information can be recorded in the database configured in the user terminal with the configuration as shown in FIG.

As for the information acquired by the user information terminal in the step (3006), a step (3007) is carried out in which predetermined information is extracted in the form of searching for a specific keyword or the like in the user's information terminal. In the step (3007) of extracting more specific information, if the acquired information is composed of simple XML, the information surrounded by a predetermined tag such as <ADRESS> JAPAN </ / ADRESS>. Evaluate whether the information is necessary for the user by extracting. The notation in each of the examples described in this specification has a high degree of abstraction for the sake of simplicity, but more detailed address, product field, product name, store genre, restaurant field, product field for sale, and sales. Detailed search conditions such as price range, inventory status, screening performance / time zone, and exhibition content may be specified, and the user can operate the operation unit (by displaying these conditions on the display unit of the user terminal). It may be configured so that the selection operation can be performed in 1080).

For example, when the user information terminal is located in Tokyo, there may be cases such as <ADRESS> TOKYO </ ADRESS> or <address> Tokyo </ address>. Similarly, any markup language can be used by using a zip code such as <ZIP> 1060837 </ ZIP> as an extraction condition or a telephone number such as <TEL> 03-XXXX-XXXXX </ TEL>. Appropriate information may be extracted from the written information, and specifically, JSON, YAML, etc. can be assumed.

As a notation other than simple XML, the distribution device has a configuration that can acquire information on position-linked search results in various formats such as JSON format and CSV format, extract the parts necessary for the user information terminal, organize them, and display them. The information acquired from the above may be edited, and the received information display processing step (3008) for configuring the necessary information so that the information can be displayed on the user information terminal based on the extracted / edited information is carried out. In this step, it is also possible to preferentially configure information only in the direction in which the imaging unit of the display device is facing.

The user terminal device (and / or software in the user terminal) in the present invention provides information to be provided in the terminal based on the location information or the like acquired from the user information terminal (or software operating on the terminal). Step (3007) to perform the information extraction process such as collecting, separating, extracting, editing, and listing the necessary information in the user terminal device by being able to search as needed from the communication line connected to the information and communication unit (3007). Is carried out. These steps are the received position-linked information extraction steps.

The process of collecting / separating / extracting the information as the above-mentioned position-linked extraction information step as scraping, and editing / listing as the mashup uses the terminal using the target selection information as the information surrounded by predetermined tags. Filtering processing and scraping processing to determine whether the information is necessary for the user by extracting the information necessary for the user, crawling processing across multiple sites may be performed, and mashup processing is used for scraping processing. According to the search conditions and the personal information of the user, the predetermined contents are put together in a predetermined format to generate information that is likely to give serendipity to the user who specified the search conditions, and to generate recommendation information and personalized information. It includes information generation processing that can improve user convenience by generating it, and various implementation methods such as collaborative filtering are proposed as specific mashup and recommendation methods. The present invention is characterized in that it is carried out not on the server side but on the client user terminal.

In this extraction process (3007), not only XML but also HTML may be used, or the data may be described in a text having an arbitrary markup language or grammar for easy processing, or an appropriate database-like list structure. The information may be necessary for display by arbitrary scraping processing because it is configured in a data description language having the above, and as a result, the information is captured by the imaging unit based on the information obtained in the step (3007). The step (3008) of generating the image information necessary for generating the CG image for synthesizing the image / video is carried out.

The step (3009) of synthesizing a CG image based on the direction and orientation of the user terminal with the actual scenery captured by the user information terminal in the above step (3008) is carried out, and the position and orientation inclination of the user information terminal are performed. The step (3010) of displaying the image information obtained by synthesizing the CG image with the actual landscape according to the information such as is displayed on the display unit of the user information terminal is carried out.

More specifically, if it is a distant object, the display size of the CG image to be combined is small, and if it is a nearby object, the display size is increased to make it larger according to the distance, or the direction of the user's information terminal. When the upper floor of a tall building is imaged according to the information obtained by various detection units (1060) such as and inclination, the position-linked information related to the upper floor of the building is processed so as to synthesize a CG image with the actual landscape. The CG image composition display step (3010) to the actual landscape is carried out by doing so.

The radius of the information to be acquired may be specified by the operation unit (1080) of the user terminal for these series of processes, and those operations are attached to the user terminal described above. It may be sound re-recognition by voice obtained from the microphones, or touch operation input by a touch sensor attached to the surface of the display unit, and these accessory functions are very well recognized in society. Therefore, no particular restraint is made in the various examples according to the present invention.

In this way, based on the search result information collected and classified (crawling and scraping) in the user information terminal and the spatial coordinate system optimized according to the position and direction of the user terminal. By synthesizing and displaying the actual scenery and CG images, the information acquired (3101) from various general distribution systems other than the dedicated information distribution system is collected, extracted, edited, and combined (crawling and scraping:) on the user terminal. By doing 3007), it is possible to configure more convenient display information for the user terminal and to configure information providing software for the user information terminal having an appropriate function as the user terminal. Therefore, it is possible to solve the fundamental problem that the necessary information to be provided is insufficient in the new information infrastructure.

Then, based on the scraped information, the information generated by the mashup process (3007) in order to further personalize the information terminal user is the target information in the database (FIG. 6) in the user terminal. Furthermore, the mashed-up information can be appropriately registered in the user information database based on the search condition information used for scraping and the search condition information used for mash-up. .. Then, personalization may be performed for the terminal user at the time of this mashup.

Furthermore, if necessary, the mashed-up information can be uploaded to the information distribution device associated with the user information terminal as cached information of the mashed-up information, or downloaded from the related information distribution as the cached information uploaded in the past. It may be configured to do so, and it meets the conditions for use in the generation of CG information (3008), which will be described later, from the database in the user terminal consisting of these mashed-up information and the database in the associated distribution device. The mashed-up information may be extracted.

The CG information provided / generated (3008) in this section and the above-mentioned section may be a two-dimensional image, a three-dimensional image, or a moving image, and the image format is particularly specific. It does not have to be, and other arbitrary information such as voice may be provided at the same time, and of course, a method of giving different images to both eyes of the HMD (head-mounted display) and displaying them three-dimensionally may be taken. However, it may be displayed on a smartphone.

In addition, when the software is implemented on the user information terminal, the user can display a logo or icon indicating a specific brand on the user information terminal and perform a position-linked search related to those logos or icons. Information related to companies and websites related to the selected logo or icon is collected in conjunction with the position, and based on the collected information, information related to the above-mentioned logo or icon according to the position and imaging direction can be enjoyed. Application software may be configured in.

Although the examples so far have been described on the premise that the user information terminal (1001) is a non-transparent device, the image information obtained from the image pickup unit (1030) is combined with the CG image synthesis step (3009). In the above, the transmissive device does not necessarily require the surrounding image as the background, and detects and calculates the subtle line-of-sight deviation between the camera position line-of-sight and the transparent screen in-line line-of-sight from the three-dimensional coordinate system of the imaging range of the captured surrounding image. , The step (3010) of correcting the CG image for compositing according to the deviation and projecting the CG image on the transmissive screen to realize and display the CG image for the real landscape that is not the captured image may be performed. .. In the case of software that implements the present invention, it may be stored in a storage medium associated with a storage unit or the like.

In addition, by transmitting and sharing the scraped and / or mashed-up information generated in this embodiment to a predetermined information distribution device, the information that can be enjoyed by other users is increased, and the information of the new service is increased. This may improve convenience.

Further, as information suitable for AR / MR, the surrounding image features obtained from the user terminal may be used, or SLAM (Simultaneus) is used to use the image feature information acquired by the user terminal for AR / MR. It may be used as information for localization and mapping), and depth information associated with imaging may be added to the image features. Using the user terminal of the configuration example of FIG. 1, FIGS. 3 and 5 It should be understood as the flow of carrying out the processes of 5001 to 5009.

Next, in conjunction with the information distribution device, an information distribution service system that obtains mashup information and scraping information of arbitrary information combinations linked to information such as position, direction, and inclination from the user information terminal and distributes the information. Explain how to use it.

An information distribution service system for information terminal users who wants to distribute information using the information distribution device without performing special information distribution provider registration (for example, registration procedure such as store registration). In order to realize this, what kind of device configuration information distribution device is used to provide information to the user information terminal on the consumer side is implemented more specifically based on the step configuration as an information distribution system. An example and what kind of step configuration the software to be operated when a hardware configuration similar to the information distribution device in the information distribution system is used will be described step by step.

As shown in FIG. 2, the information distribution device (2001) according to the information distribution system of the present invention includes at least a calculation unit (2010), a storage unit (2020), and a communication unit (2030). Further, the storage unit may be regarded as having a volatile storage unit (2021) and a non-volatile storage unit (2022). The arithmetic unit may be composed of a CPU, GPGPU, or any other arithmetic device (TPU, ReRAM analog arithmetic device, etc.), and the storage unit is a volatile storage unit such as a RAM or cache memory or a storage medium such as a GPGPU memory. It may be provided, and the non-volatile memory is preferably composed of a storage medium including an arbitrary electronic storage device such as a ROM, a flash memory, or a ReRAM, or an arbitrary storage such as an HDD, an SSD, or an optical storage device. Further, the keyboard, mouse, display unit, and the like may be provided so that the operation can be performed, and other auxiliary parts may be optionally provided.

Then, the user information terminal can detect various information such as the position, direction, and inclination provided in the user terminal with information on the position and / or direction and / or inclination and / or the captured image and / or the captured image feature. Various detection units (1060) detect and acquire each information. More specifically, the service information provision and distribution device according to the present invention based on information acquired from appropriate electronic devices such as GPS (including GNSSs such as quasi-zenith satellites), accelerometers, gyro sensors, and electronic compasses (1064). In (2001), the user terminal device (1001) searches and / or extracts and / or sorts from the first position-linked information stored in the information distribution device of the information distribution system of the present invention including position information and the like. A step (3005) of issuing a request to do so is performed.

Then, the information distribution service using the information distribution device according to the information distribution system of the present invention is a step of receiving the first information such as the position-related information and the search condition information of the user information terminal from the user terminal based on the request. (3006) is carried out. The step of receiving this first information is to search and / or extract and / or select the second position-linked information required for the user information terminal from the first position-linked information collected in advance in the information distribution system. This makes it possible to transmit the information necessary for providing the user information terminal.

As described above, the information distribution device (and / or information distribution system software) used in the information distribution system as a service in the present invention in order to provide the second position-linked information to the user information terminal is an arbitrary user information terminal. Collection and / or extraction and / or editing of information that can be searched from other information distribution devices previously connected via a communication line by the information distribution system based on information acquired from (or software that operates on the terminal). / Or, the first position-linked information group generation step used for searching from an arbitrary user terminal by listing (crawling or scraping) is carried out by the information distribution device (2002) used in the information distribution system, and the first The user information terminal (1001) searches for the second position-linked information group from the first position-linked information acquired by executing the position-linked information group generation step of the second position-linked information group for the user terminal. The step of acquiring the position-linked information group is carried out.

Then, it can be distributed to the storage unit of the information distribution device (2001) used in the information distribution system of the present invention regardless of the presence or absence of a request issued from the user information terminal by the steps of collection, extraction, editing, and listing. By saving the distribution information as the first position-linked information group obtained as the search result, it is possible to prepare the information that is likely to be needed in advance even if there is no particular request from the user terminal. , User information In response to a request for location-related information and search target selection information provided by the terminal (which may be application software of the terminal, of course), the user appropriately obtains information in the vicinity of the location of the user terminal. A step (inside 5102) of transmitting information from the information distribution device side in the information distribution system of the present invention to the information terminal is performed.

As described above, the information distribution device used in the information distribution system of the present invention autonomously connects to another information distribution device using the communication unit provided in the information distribution device even if there is no particular request from the user information terminal. It is possible to collect and store the information necessary for distribution.

At this time, the purpose is to collect and store in advance the information necessary for the information distribution system to distribute the location-linked information to the user information terminal (1001), and search according to the geographical location to which the information belongs. The step of selecting the condition (4001) is carried out, and the step of determining the predetermined position-linked search condition (4002) is carried out in the step. It is possible to collect information in advance without performing such special geographical attribution, or to compose it from geographical information included in the information obtained based on general keyword search results.

Then, the information distribution terminal performs a step (4003) of searching and collecting information on the general network according to the conditions determined in the confirmation step (4002) in order to collect information for distribution to the external information distribution device. Will be done. In response to the request in the search collection step (4003), the information as the search result is transmitted from the external information distribution device to the information distribution device of the present invention, and the information distribution device of the present invention receives the necessary information (4004). Is carried out.
The external information distribution device performs the search process (4101) in response to the request step (4003) of the information distribution device of the present invention.

After the step (4004) of receiving the search target information as the position-linked search result is carried out, the information distribution device of the present invention carries out the step (4005) of classifying the necessary information contents from the received search target information. .. At this time, a database (DB: information that can confirm an arbitrary physical position such as latitude / longitude, postal code, address, telephone number, and information related to the location in the company such as a delivery company) accompanying the information is used. It is preferable that the structural example is organized and stored in Fig. 6), etc., or included in the storage items. Furthermore, the update from the user information terminal to the database of the information distribution device is not the packet communication of the smartphone, but the Wi-. Fi connection is preferable

Subsequently, the information classified in the step (4005) is organized and recorded in, for example, a DB (database: FIG. 6) according to the required specifications (4006). As described above, this step uses the location information (information that can confirm any physical location such as latitude / longitude, postal code, address, telephone number, and location-related information within the vendor such as the delivery company) that accompanies the information. It is preferable that the information is organized and stored in a database (DB: FIG. 6) or included in the storage items.

The process of collecting, extracting, editing, and listing this information, more specifically, the information to be handled specifically when it is implemented in step 4005, is the identification that the information user can handle in the user information terminal. Keywords (information on service provision such as product information, store information, inventory information, screening performance information) and location information (this is the same in other examples, but latitude / longitude, address, postal code, telephone number, and other locations. The step of extracting predetermined information is carried out in the form of searching for an identification code (including, for example, an area code used internally by the delivery company).

More specifically, if the information acquired in the above step is composed of simple XML, the information enclosed in predetermined tags and the target selection information such as <ADRESS> JAPAN </ / ADRESS> are displayed. By extracting the information necessary for the terminal user by using it, it is judged whether or not the information is necessary for the user. Of course, this extraction process may be performed from HTML, may be described in any markup language, or may be configured in a data description language having a list-up structure such as an appropriate database structure. Further, by registering the mashup information in the database of the user terminal device and the information distribution device (the structure is shown in FIG. 6), it can be used as a caching system used for an efficient information providing service. This information may be composed of JSON, YAML, CSV and the like.

Then, for example, when the user's information terminal is located in Tokyo, cases such as <ADRESS> TOKYO </ ADRESS> or <address> Tokyo </ address> can be considered. Similarly, a postal code such as <ZIP> 1060837 </ ZIP> may be used as an extraction condition, or a telephone number such as <TEL> 03-XXXX-XXXXX </ TEL> may be used as an extraction condition. As a notation other than simple XML, it is possible to acquire information on position-linked search results in various formats such as JSON format and CSV format, extract and organize (scraping) the necessary parts for the user information terminal, and display it. The information acquired from the distribution device may be edited in the configuration, and the received information display processing step of configuring the information so that the information can be displayed on the user information terminal based on the edited information is performed. In this step, as in the other embodiments, it is possible to preferentially configure information only in the direction in which the image pickup unit of the display device in the user terminal device is facing.

For example, in this way, the information distribution system of the present invention and the information distribution device used in the information distribution system acquire the first position-linked information. Not limited to such a method, any information can be collected by any method as long as the information terminal of the present invention has a configuration that can be searched and / or collected. Further, although autonomous collection on the user information terminal (1001) side is described in another embodiment, a position-linked information distribution device (2001) that uses the information collected by such a method in an information distribution system. By providing the information to the information distribution device, a configuration for efficiently collecting the information required by the position-linked information distribution device may be realized.

Regarding the search target selection information as the above-mentioned search condition information, the search condition information is configured by selecting an appropriate icon or inputting characters as in the other examples, and the search target selection information as the configured search condition information. It can also be configured to carry out information retrieval for the user information terminal. In this case, it can be applied when the search input screen or the like is provided from the information distribution device side like a Web browser.

Based on the search target selection information as the search condition information, the above-mentioned information terminal user service system is the first position interlocking in the information distribution device (2001) arranged for the user information terminal from the information distribution device (2001). By providing the second position-linked information acquired from the information to the user information terminal (1001), the processing performed as the processing in the user terminal in other embodiments is performed in advance in the service system for the information terminal user. This has the effect of reducing the processing load in the user terminal and making it possible to provide CG configuration information for the user terminal as an efficient position-linked information terminal, and improves convenience.

Then, the user information terminal is based on the information acquired from the service system for the user information terminal user, and the information for CG configuration for the user terminal for providing information in the direction of the imaging unit based on the current position of the user information terminal. By appropriately constructing a CG image from the user information terminal and synthesizing and displaying it in the user information terminal based on the search result information and the spatial coordinate system optimized according to the position and direction of the user terminal, the user information terminal By synthesizing the CG image with the actual landscape captured by the imaging unit, it is possible to realize a CG image composition information providing service for the actual landscape by the user terminal using the information for the information terminal user service system according to the present invention.

Therefore, even in a situation where the applicant does not pre-register the distributor registration in the service infrastructure in the information distribution system in such a new service infrastructure, the information distribution device of the present invention collects and provides the information in advance. By implementing it, it is possible to solve the fundamental problem of lack of information provided at the initial stage of the new information infrastructure.

Further, the present invention is defined as an information distribution system in which a plurality of information distribution devices cooperate to realize position-linked information distribution of the present invention, and as an example, one information distribution device (2001) is used. Since it is assumed that the realization will be carried out by a group of a plurality of information distribution devices, the examples are summarized as an information distribution system. In the case of software that implements the present invention, it may be stored in a storage medium associated with a storage unit or the like.

Subsequently, the first position-linked information and the second position-linked information will be described with reference to FIG. First, the first position-linked information, the first position-linked information group, the second position-linked information, and the second position-linked information group are used in the user information terminals (5001, 1001) according to another embodiment of the present invention. This is a concept required when the information distribution device (5101, 2001) of the present invention is used.

The second position-linked information and the second position-linked information group are information obtained as a result of performing processing such as searching and / or extracting and / or sorting from the first position-linked information and the first position-linked information group. Since it refers to a group, it can be considered that the requirement is included in the first position-linked information and the first position-linked information group. Further, the second position-linked information group collected from (5201) with the progress of the steps (5004, 5005, 5102) is stored in the storage unit (FIG. 2) of the information distribution device of (5101) for the user terminal. The information may be recorded as information on the structure of the database (FIG. 6).

The first position-linked information and the first position-linked information group are generally composed of the following information. More specifically, as shown in FIG. 6, the table structure of the database (FIG. 6).
), And has location-related information such as latitude, longitude, zip code, telephone number, and address as the first search key. Next, it has information indicating the attributes related to location information such as whether the thing existing at that position is a store, a house, a school, or a park as a key, and then the preliminary classification information of the attribute information related to location information is used as a key. These preliminary information may be configured to have a plurality of items in the same layer or to have a plurality of layers.

More specifically, if it is a store, it is a "restaurant," "electric store," "sports goods store," "sports gym," etc., if it is a house, it is a "detached house," "apartment house," etc. There are "elementary school", "junior high school", "high school", "university", etc., and item information is added based on the classification related to each attribute. These item information may have any configuration as long as it can be discriminated from both numerical information and textual information.

The first position-linked information configured in this way is appropriately registered in the information distribution device alone or a plurality of linked distribution systems of the present invention, and constitutes the position-linked information distribution device (5101, 2001) as a system. Then, the user information terminals (5001, 1001) request the first position-linked information group stored in the position-linked information distribution device (5101, 2001) using the information acquired and generated in the terminal as a key, such as a search. I do. As a result of a request such as a search request to the position-linked information distribution device (5101, 2001), the position-linked information distribution device (5101, 2001) of the present invention uses a second position-linked information group as a user information terminal (5001, 1001). ) To provide.

As a result, the user information terminals (5001, 1001) that received the second position-linked information group appropriately use the user information terminals (5001) as in other embodiments based on the received second position-linked information. , 1001) It becomes possible to provide information in the form of appropriately image-synthesizing the actual scenery and indoor / outdoor information to the user's vision. In the case of software that implements the present invention, it may be stored in a storage medium associated with a storage unit or the like.

Further, based on the collection, classification, analysis, and editing of position-linked information in other embodiments, it is appropriate to associate the position-linked information database (FIG. 6) with the position as an example of the information configuration of the records in the database according to the predetermined specifications. The information organized in is shown in (Since the detailed numerical values of each field are unintended and rough values, values that do not match the original latitude and longitude may be described). Information to be classified, analyzed, and stored in the storage unit (2020) of the position-linked information distribution device (2001) as a storage location of this information by using an AI method such as a statistical method, a machine learning method, or deep learning. Generates information groups such as variables, arrays, matrices, and tensors that can be used as indicators of, and provides physical position-related information to arithmetic elements such as latitude and longitude, appropriate walking and vehicle road route distance information, and appropriate public information. Fields such as information providing stores that process information provided by users and the user's intentions estimated from operations as evaluable information by converting it to information such as institutional transportation network distance information. It can also be configured as a position-linked information distribution device (2001) that appropriately provides and processes information for service users by appropriately associating the information.

More specifically, the information to be provided by the user information terminal is the user information terminal (1001) in another embodiment, and the position-linked information distribution device (2001) as the present invention in another embodiment. Information provided to the user of the user information terminal (1001) by using the above-mentioned statistical processing, machine learning, and deep learning in the steps when performing information retrieval, collection, classification, analysis, editing, and extraction in). The optimization of the information is realized by machine learning and AI. By optimizing the information provided to these users, a curation service for position-linked information that is effective for the user's aha experience and serendipity may be realized, and SNS as information that will be a resource used for their optimization. In addition to the site, food and drink, accommodation, product review sites, etc. may be extracted based on information linked to the location, optimized, and provided to the user.

Further, although not shown in FIG. 6, if it is described as an example, the image information and / or the image feature and / or the information indicating the location of the image information (for example, a file) is described as a field in the record that summarizes the position information related information. Used by analyzing, recording, and saving information indicating the location of image feature information (for example, file name or URL) and / or information indicating the location of image feature information (for example, file name or URL) in association with position-linked information. It may be possible to provide more efficient position-linked information to the service user from the person information terminal or the position-linked information distribution device. Of course, if the position information and the image information and / or the image feature information are associated by any method, the implementation method is not the information in the same record but the information that is appropriately associated. You don't have to.

At this time, image information and image feature information classified by position interlocking are classified according to the direction information acquired from the user information terminal, and improvements are made by processing efficiency and optimization by AI such as machine learning and deep learning based on the direction information. The information distribution environment of the present invention can also be configured so that it can be applied to performance improvement for providing position-linked information more appropriately according to directions such as direction information and inclination information. More specifically, position information and / or tilt information and / or direction information and / or movement speed information and / or image information and / or image feature information obtained from the above-mentioned user information terminal or arbitrary information distribution device. By forming a three-dimensional map by AI processing that associates image information and position information, which is called SLAM, with SLAM and configuring it to assist the self-position estimation of the user terminal, the self-position of the user terminal by SLAM interlocking. You may configure an AR / MR service that uses a system that assists in estimation.

In addition, the information collected independently by the user information terminal itself in the user information terminal (1001) is linked to the storage unit (1020) of the user information processing terminal (1001) by the position interlocking of the position interlocking information distribution device (2001). A database having the same configuration as the information database (FIG. 6) is constructed in the storage unit (1020) of the user information terminal (1001), and some information is shared or used with the position-linked information distribution device (2001). A more efficient and optimal series of service systems may be operated by uploading from the personal information terminal (1001) to the database (FIG. 6) of the location-linked information distribution device (2001).

Further, the position-linked information distribution device (2001) and the user information terminal (1001) in the present invention each include a communication unit, and the communication unit has a wired LAN, a wired WAN, regardless of whether it is wired or wireless. It is also possible to use any various communication technologies such as a communication carrier line, an in-store wireless LAN, a wireless LAN via a PAN (Personal Area Network), and information communication via a communication carrier line. In the case of software that implements the present invention, it may be stored in a storage medium associated with a storage unit or the like.

In the device and / or system and / or software / program as the above-described embodiment, the position information and / or direction information and / or inclination information and / or image information and / or image feature information and / or the position information and / or direction information and / or inclination information collected by the information distribution device and / or Alternatively, arbitrary text information collected based on the position of the user information terminal is selected by an arbitrary method, combined, hashed or appropriately encrypted, and information (distributed ledger information) is added to the arbitrary ledger information. As an information management method such as describing (etc.), the information distribution system or user information terminal of the present invention may be configured so that the reliability of information and the management status of information can be grasped by a blockchain or the like. The computing resources required for distributed operations such as blockchain may be secured by using the computing resources of the terminal or the distribution system. In the case of software that implements the present invention, it may be stored in a storage medium associated with a storage unit or the like.

The services provided by these user information terminals, information distribution devices, and information distribution systems refer to past historical information based on location, and games linked to location information related to content such as arbitrary works. Information is provided as a service, real estate sales information and open house information are provided in a position-linked manner, restaurant evaluations and seat availability are provided in a position-linked manner, inventory information of any store, etc. It is possible to provide services by associating any information linked to the position, such as bargain sale information, sales fair, sales event information, price information, seminar information by any operator, and information inside the event venue. It is also possible to make it possible to introduce related content and to provide information that is historically related to the location.

To summarize the processing procedure as a client-side embodiment of the user information terminal device (1001) of the present invention, it can be obtained from various detection units (1060, 1061, 1062, 1063, 1064) of the user information terminal device (1001). The user information is based on the position information and the direction information or the instruction information obtained from the operation unit (1080) (may be configured as text information input from 1080 or selected icon selection information). First information from an arbitrary information distribution device (for example, a web search site, an SNS site, an arbitrary evaluation site such as a hotel or a gourmet) and / or an information distribution device system via the communication unit (1050) of the terminal device (1001). Based on the collecting steps (3005, 3006) and the position information of the user information terminal device (1001) and the instruction information from the collected first information, the user information is obtained with respect to the first information. The step (3007) of performing scraping processing using the arithmetic unit (1010) and the storage unit (1020) of the terminal device (1001) to generate the second information, and the above-mentioned obtained from the user information terminal device (1001). Based on the position information and / or the instruction information, the second information is mashed up by using the calculation unit (1010) and the storage unit (1020) of the user information terminal device (1001) to obtain the third information. Based on the step (3008) to be generated, the third information, and the imaging direction of the user information terminal device (1001), the user is displayed on the display unit (1040) of the user information terminal device (1001). The computer graphic image information generated by the image generation unit (1070) of the information terminal device (1001) and the captured image information obtained by the image pickup unit (1030) of the user information terminal device (1001) are tertiary in the imaging range. It is configured as a user information terminal device (1001) characterized in that the steps (3009, 3010) of synthetic display according to the original coordinate system are executed, and can be similarly implemented as a program.

The first information is used for the search conditions required from the external distribution device, the information that serves as the selection criteria for generating the second information by the scraping process that extracts the necessary information from the acquired search results, and the display after selection. It is also used as a reference when displaying a CG image based on the condition setting information of the third information mashed up and the third information on the display unit of the user information terminal.

The second information is information that has been organized and processed based on the first condition after processing for collecting raw information from an external information distribution device or the like based on the condition of the first information. Generated. As a specific example, an HTML file is extracted from a predetermined Web server, and by further accessing the link destination of the file, more detailed information is collected and the address information is extracted from the collected HTML file. Then, by searching the address information, the latitude / longitude information extracted from the address information and the link address such as a predetermined URL can be set as the second information.

Then, by repeating such processing, the second information itself may be configured as an information group having a list structure, and further, HTML title tags and the like may be extracted and associated with links, etc. It may be a list structure, or by associating the HTML file itself as a list structure, a link to the display content in the third information generated by the mashup process, a file name, or the direct HTML content can be used. Second information may be generated as information having a list structure.

Then, a database (FIG. 6) is configured based on the list structure as described above, and is appropriately recorded in the storage unit, and the recorded information is used as a database cache or a database file in the volatile storage unit or non-volatile storage unit. Recorded in volatile storage. It should be noted that the volatile storage unit and the non-volatile storage unit are merely illustrated as abstract concepts, and more specifically, rewritable ones such as HDD, SSD, and RAM and non-rewritable ones such as ROM are implemented. It may be any medium, and its implementation is not particularly constrained.

The third information is information constructed based on the second information, and conditions for compositing with the live-action image are added based on the first information, and any direction and / or relative coordinates in the display. Information according to the distance and information necessary for display (for example, 3D model information, text information for display, voice information, coloring information, CSS information, and link information and files that indicate the relevance of those information. Information, URL information, etc.) are configured. The captured image information is configured as image information obtained by the imaging unit (1030) of the user information terminal. The computer graphic (CG) image information is configured as image information generated by the image generation unit (1070) for synthesizing on the captured image information based on the third information. Furthermore, the scraping process shows the process of extracting specific keywords and HTML tags, and the extraction process arranges them in a predetermined format and stores them, and the mashup process organizes the collected information in a form that is easy to provide to users. Indicates the process of specifying the display format so that it can be displayed as organized information. Providing such processing according to the user's preference analysis may be called curation.

In other words, the user information terminal device is based on the position information obtained from various detection units of the user information terminal device, the direction information, or the instruction information obtained from the operation unit. The step of collecting the first information from an arbitrary information distribution device and / or the information distribution device system via the communication unit, and the position information and the instruction information of the user information terminal device from the collected first information. Based on the above, a step of performing scraping processing on the first information by using the calculation unit and the storage unit of the user information terminal device to generate the second information, and the step obtained from the user information terminal device. A step of mashing up the second information based on the position information and / or the instruction information using the calculation unit and the storage unit of the user information terminal device to generate the third information, and the third information. Based on the information and the imaging direction of the user information terminal device, the computer graphic image information generated by the drawing calculation unit of the user information terminal device and the user information are displayed on the display unit of the user information terminal device. It can also be said to be a user information terminal device characterized in that a step of synthesizing and displaying the captured image information obtained by the imaging unit of the terminal device according to the three-dimensional coordinate system of the imaging range is performed. Further, it is preferable that the update to each database is not the packet communication of the smartphone but the Wi-Fi connection regardless of whether the database is configured in the smartphone as the user terminal or the database of the distribution device.

Then, when displaying the scraped and mashed information from these user terminal devices at an appropriate three-dimensional position accompanying the captured image displayed in the user terminal display unit, the imaging unit (1030) By recording the imaging information acquired in the above-mentioned imaging unit (1030) and the information from the depth sensor provided in the above-mentioned imaging unit (1030) in a database for SLAM, the convenience of the user may be improved. SLAM is performed by accumulating the image feature information extracted from the information obtained by scraping in the database shown in FIG. 6, and the AR / MR information is synthesized and displayed at a more appropriate position in the database shown in FIG. Weather information, date and time information, and astronomical information such as the sun and moon may be recorded in association with each other, and the efficiency and quality of scraping and mashup may be improved based on the information.

Furthermore, by sharing information within the movement range of the user terminal in the same manner as the distribution device, the position information collected by the information terminal device in the device and / or system and / or software / program as the above-described embodiment. And / or direction information and / or inclination information and / or image information and / or image feature information and / or arbitrary text information collected based on the position of the user information terminal is selected and combined by an arbitrary method. As an information management method, such as hashing and appropriate encryption processing, and describing information (distributed ledger information, etc.) in arbitrary ledger information, grasp the reliability of information and the management status of information by blockchain etc. The information distribution system and the user information terminal of the present invention may be configured so as to be possible, and the calculation resources of the user terminal and the distribution system may be used to secure the calculation resources required for distributed operations such as blockchain. .. In the case of software that implements the present invention, it may be stored in a storage medium associated with a storage unit or the like, or position-linked data related to the installation location of the distribution device in parallel distributed processing for wide area deployment. You may collect and realize an appropriate location-linked service by parallel and distributed processing.

According to the present invention, in a new AR / MR network service infrastructure, a new network can be suddenly used on a global scale without preparing information configured exclusively for a new technology by using information on an existing network. It will be possible to provide the necessary and sufficient content for the service infrastructure. Furthermore, in the past, a complicated processing system is used to encourage users who wish to use the information support system to register, and the location-linked information support service system is used to encourage the registration of system users who wish to provide information. Although it was necessary to provide the information, by using the method of the present invention, the information terminal device of the service user autonomously collects the position-linked information, so that it is not necessary to perform the conventional registration procedure or the like. , It becomes possible to realize a position-linked information support service system that is appropriate and highly convenient according to the position and direction of the user terminal. Then, based on the data uploaded from the user information terminal to the distribution device dedicated to this service, the distribution side information when providing a completely new service infrastructure is temporarily stored as a cache, thereby chicken (service infrastructure). By solving the problem of which comes first, egg (contents in the service infrastructure) and accelerating the spread of new service infrastructure, the possibility of using AR / MR in the outdoor information industry will be dramatically improved. You will be able to do it.

1001 User information terminal device 1010 Calculation unit 1020 Storage unit 1030 Imaging unit 1040 Display unit 1050 Communication unit 1060 Various detection units 1061 GPS
1062 Gyro sensor 1063 Accelerometer 1064 Electronic compass 1070 Image generator 1080 Operation unit
2001 Information distribution device 2010 Calculation unit 2020 Storage unit 2030 Communication unit
3001 Various detection steps 3002 Imaging step 3003 Image feature extraction step 3004 Search condition setting operation step 3005 Search processing request step 3006 Search result acquisition step 3007 Search result content extraction step 3008 CG display content generation step 3009 CG is synthesized with the captured image. Step 3010 Composite image display step
4001 Position-linked save target selection step 4002 Position-linked search condition confirmation step 4003 Position-linked search request step 4004 Position-linked search acquisition step 4005 Position-linked search result content classification step 4006 Classification search result DB storage step
5001 Processing system of information processing terminal of the present invention 5101 Processing system of information distribution device of the present invention 5201 Processing system of general information distribution device 5002 Various sensors / imaging in information terminal and feature extraction step 5003 In information terminal Search condition setting step 5004 Information search processing request step 5005 related to the position of the information terminal Information search result acquisition step 5006 related to the position of the information terminal Content extraction step 5007 According to the extracted content CG generation step 5008 for information terminal Step 5009 Combining the generated CG with a background image according to the situation of the information terminal 5009 Step 5102 Displaying the composite image on the information terminal Search step 5111 Position in the information distribution device Linked information search step 5112 External search step 5113 When appropriate information is not found in the information distribution device Search result reply step 5103 by the information distribution device Temporary storage (cache) processing step 5202 of information acquired by the external search Search processing to an external distribution device 5203 Providing information to this information distribution device of search results acquired from a general external distribution device

Claims

According to the necessary information acquisition conditions based on the position information obtained from various detection units of the user information terminal device, the direction information, or the instruction information obtained from the operation unit.
A step of collecting first information from a plurality of arbitrary general-purpose Web information distribution devices via a communication unit associated with the user information terminal device, and
The first information collected in the collection step,
First information to generate second information by performing scraping processing based on the position and range of the user information terminal and the necessary information acquisition condition using the calculation unit and the storage unit provided in the user information terminal device. Steps to generate and
The second information by scraping obtained in the first information generation step,
Using the calculation unit and the storage unit provided in the user information terminal device,
A second information generation step of performing mashup processing based on the position information and imaging range of the user information terminal and the necessary information acquisition condition to generate a third information, and
Based on the third information obtained in the second information generation step,
Using the storage unit and the calculation unit in the user terminal,
Perform the mashup step as the third information,
The display content is selected from the plurality of third information groups on the display unit of the user information terminal device, and the display content is selected.
Using the drawing calculation unit of the user information terminal device,
A step of generating computer graphic image information based on the necessary information acquisition conditions of the user information terminal device, and
On the display unit associated with the user information terminal device,
The captured image information obtained by the imaging unit associated with the user information terminal device and the captured image information.
According to the computer graphic image information generated by the drawing calculation unit of the user information terminal device,
According to a three-dimensional coordinate system based on the imaging range information of the captured image obtained by using the image feature information obtained from the captured image information, the position information obtained from the user information terminal, and the direction information. ,
A step of combining and displaying the captured image information and the computer graphic image information,
User information terminal device that implements a program for user terminals of services related to MR and / or AR, which is characterized by implementing
According to the necessary information acquisition conditions based on the position information obtained from various detection units of the user information terminal device, the direction information, or the instruction information obtained from the operation unit.
A step of collecting first information from a plurality of arbitrary general-purpose Web information distribution devices via a communication unit associated with the user information terminal device, and
The first information collected in the collection step,
First information to generate second information by performing scraping processing based on the position and range of the user information terminal and the necessary information acquisition condition using the calculation unit and the storage unit provided in the user information terminal device. Steps to generate and
The second information by scraping obtained in the first information generation step,
Using the calculation unit and the storage unit provided in the user information terminal device,
A second information generation step of performing mashup processing based on the position information and imaging range of the user information terminal and the necessary information acquisition condition to generate a third information, and
Based on the third information obtained in the second information generation step,
Using the storage unit and the calculation unit in the user terminal,
Perform the mashup step as the third information,
The display content is selected from the plurality of third information groups on the display unit of the user information terminal device, and the display content is selected.
Using the drawing calculation unit of the user information terminal device,
A step of generating computer graphic image information based on the necessary information acquisition conditions of the user information terminal device, and
On the display unit associated with the user information terminal device,
The captured image information obtained by the imaging unit associated with the user information terminal device and the captured image information.
According to the computer graphic image information generated by the drawing calculation unit of the user information terminal device,
According to a three-dimensional coordinate system based on the imaging range information of the captured image obtained by using the image feature information obtained from the captured image information, the position information obtained from the user information terminal, and the direction information. ,
A step of combining and displaying the captured image information and the computer graphic image information,
A storage medium in which a program for a user terminal of a service related to MR and / or AR is recorded.
According to the necessary information acquisition conditions based on the position information obtained from various detection units of the user information terminal device, the direction information, or the instruction information obtained from the operation unit.
A step of collecting first information from a plurality of arbitrary general-purpose Web information distribution devices via a communication unit associated with the user information terminal device, and
The first information collected in the collection step,
First information to generate second information by performing scraping processing based on the position and range of the user information terminal and the necessary information acquisition condition using the calculation unit and the storage unit provided in the user information terminal device. Steps to generate and
The second information by scraping obtained in the first information generation step,
Using the calculation unit and the storage unit provided in the user information terminal device,
A second information generation step of performing mashup processing based on the position information and imaging range of the user information terminal and the necessary information acquisition condition to generate a third information, and
Based on the third information obtained in the second information generation step,
Using the storage unit and the calculation unit in the user terminal,
Perform the mashup step as the third information,
The display content is selected from the plurality of third information groups on the display unit of the user information terminal device, and the display content is selected.
Using the drawing calculation unit of the user information terminal device,
A step of generating computer graphic image information based on the necessary information acquisition conditions of the user information terminal device, and
On the display unit associated with the user information terminal device,
The captured image information obtained by the imaging unit associated with the user information terminal device and the captured image information.
According to the computer graphic image information generated by the drawing calculation unit of the user information terminal device,
According to a three-dimensional coordinate system based on the imaging range information of the captured image obtained by using the image feature information obtained from the captured image information, the position information obtained from the user information terminal, and the direction information. ,
A step of combining and displaying the captured image information and the computer graphic image information,
An information distribution device that distributes a program for user terminals of services related to MR and / or AR, which is characterized by carrying out
According to the necessary information acquisition conditions based on the position information obtained from various detection units of the user information terminal device, the direction information, or the instruction information obtained from the operation unit.
A step of collecting first information from a plurality of arbitrary general-purpose Web information distribution devices via a communication unit associated with the user information terminal device, and
The first information collected in the collection step,
First information to generate second information by performing scraping processing based on the position and range of the user information terminal and the necessary information acquisition condition using the calculation unit and the storage unit provided in the user information terminal device. Steps to generate and
The second information by scraping obtained in the first information generation step,
Using the calculation unit and the storage unit provided in the user information terminal device,
A second information generation step of performing mashup processing based on the position information and imaging range of the user information terminal and the necessary information acquisition condition to generate a third information, and
Based on the third information obtained in the second information generation step,
Using the storage unit and the calculation unit in the user terminal,
Perform the mashup step as the third information,
The display content is selected from the plurality of third information groups on the display unit of the user information terminal device, and the display content is selected.
Using the drawing calculation unit of the user information terminal device,
A step of generating computer graphic image information based on the necessary information acquisition conditions of the user information terminal device, and
On the display unit associated with the user information terminal device,
The captured image information obtained by the imaging unit associated with the user information terminal device and the captured image information.
According to the computer graphic image information generated by the drawing calculation unit of the user information terminal device,
According to a three-dimensional coordinate system based on the imaging range information of the captured image obtained by using the image feature information obtained from the captured image information, the position information obtained from the user information terminal, and the direction information. ,
A step of combining and displaying the captured image information and the computer graphic image information,
Program for user terminals of services related to MR and / or AR, which is characterized by implementing