JP6958795B1 - Information processing methods, computer programs and information processing equipment - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing method, a computer program and an information processing device for providing an image of a user of a facility together with information other than the image. In the information processing method according to the present embodiment, an information processing device accepts selection of a facial expression related to a target, and the selected facial expression is obtained from an image taken by a camera installed in a facility used by the target. The image extracted accordingly is displayed, and the correspondence between the event carried out at the facility and the facial expression of the target is displayed. Further, the information processing method may display a plurality of characters or images associated with facial expressions as selection candidates, accept selections for the selection candidates, and accept selections of facial expressions related to the target. [Selection diagram] Fig. 6

Description

The present invention relates to an information processing method, a computer program, and an information processing device that perform processing related to display of an image captured by a camera.

Parents of children or students attending facilities such as nursery schools, kindergartens or elementary schools want to know in detail how their children spend their time in these facilities. For example, nursery teachers or teachers engaged in these facilities take pictures of children or students, and post the pictures taken on the bulletin board of the facility or organize the pictures to create an album etc. for parents. Parents can know the state of the child in the facility by handing it over, but the burden on the nursery teacher or teacher will increase.

In Patent Document 1, a specialized cameraman is dispatched to a kindergarten or school to take a picture of a kindergarten child or a student, the picture is posted on the website, an application for a photo album is accepted from the user, and the application for a photo album is accepted through the website. A photo album creation system that distributes photo albums has been proposed.

Japanese Unexamined Patent Publication No. 2008-21191

The pictures taken at the facility are taken from the moments of daily life of children or students, and it is difficult to know the state of the child throughout the day. The same applies to the photo album creation system described in Patent Document 1. Further, the photo album creation system described in Patent Document 1 is not easy to use frequently because it is necessary to dispatch a specialized cameraman to the facility. For example, it is possible to ask a professional cameraman to take pictures of special events such as entrance ceremonies, entrance ceremonies, athletic meet, graduation ceremonies or graduation ceremonies, but it is difficult to ask a professional cameraman to take pictures of daily life.

The present invention has been made in view of such circumstances, and an object of the present invention is an information processing method, a computer program, and information that can be expected to provide images taken on a daily basis to users of a facility. The purpose is to provide a processing device.

In the information processing method according to one embodiment, the information processing apparatus accepts the selection of facial expressions related to the target, and the images taken by the camera installed in the facility used by the target are extracted according to the selected facial expressions. An image is displayed, and the correspondence between the event carried out at the facility and the facial expression of the target is displayed.

In the case of one embodiment, it can be expected to provide images of facility users on a daily basis.

It is a schematic diagram for demonstrating the outline of the information processing system which concerns on this Embodiment. It is a block diagram which shows the structure of the server apparatus which concerns on this embodiment. It is a schematic diagram which shows one configuration example of a user DB. It is a schematic diagram which shows an example of the aggregated information generated by the information aggregation processing unit. It is a block diagram which shows the structure of the terminal apparatus which concerns on this embodiment. It is a schematic diagram which shows an example of the UI screen displayed by the terminal apparatus which concerns on this embodiment. It is a flowchart which shows the procedure of the image storage processing performed by the server apparatus which concerns on this embodiment. It is a flowchart which shows the procedure of the image display processing performed by the terminal apparatus which concerns on this embodiment. It is a flowchart which shows the procedure of the image display processing performed by the server apparatus 3 which concerns on this Embodiment.

A specific example of the information processing system according to the embodiment of the present invention will be described below with reference to the drawings. It should be noted that the present invention is not limited to these examples, and is indicated by the scope of claims, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

<System overview>
FIG. 1 is a schematic diagram for explaining an outline of the information processing system according to the present embodiment. The information processing system according to the present embodiment transmits an image (still image or moving image) taken by one or a plurality of cameras 1 installed in the facility 100 to the server device 3, and transmits the image received from the camera 1 to the server. This is a system in which the device 3 transmits to the user's terminal device 5.

The camera 1 may be, for example, a device such as a smartphone or a tablet terminal device having a camera. Further, the camera 1 may be a device having a variable shooting direction or angle of view, and may be, for example, a camera that automatically tracks and shoots a predetermined subject. Further, the camera 1 may be a camera capable of photographing a range of, for example, 360 °. Further, the camera 1 may be a camera mounted on a moving body such as a drone, and the camera 1 set in the facility 100 may be used to take pictures inside and outside the facility 100 with the camera mounted on the moving body. A device that performs the above may be included. Further, a plurality of cameras 1 having various configurations as described above may be installed in the facility 100.

In the illustrated example, a nursery school, a kindergarten, or the like is shown as the facility 100. A camera 1 installed in a facility 100 such as a nursery school or a kindergarten automatically and repeatedly takes a picture of the inside of the facility 100 and a user (a child in this example) who uses the facility 100, for example, repeatedly at a predetermined cycle. The camera 1 has a function of performing communication via a mobile phone communication network and a network such as the Internet, and the camera 1 transmits the captured image to the server device 3.

The server device 3 receives an image from the camera 1 via the network, and stores and stores the received image in the storage device. Further, the server device 3 transmits the stored image to the terminal device 5 such as a smartphone or a PC (personal computer) used by the user. In the present embodiment, the terminal device 5 may include a terminal device 5 used by a guardian or the like of a child who uses the facility and a terminal device 5 used by an employee such as a nursery teacher who is engaged in the facility. .. As a result, for example, a guardian who entrusts a child to a facility 100 such as a nursery school or a kindergarten can view an image of a child spending time at the facility 100 taken by the camera 1 on his / her own terminal device 5. Further, for example, a nursery teacher or the like in the facility 100 can acquire an image taken by the camera 1 of the child spending time in the facility 100 by the terminal device 5 without performing the work of taking a picture of the child. For example, a photo album can be created or a photo can be posted in the facility 100.

In the information processing system according to the present embodiment, the server device 3 detects, for example, a person with respect to the image captured by the camera 1, the process of identifying the detected person, and the facial expression (emotion) of the identified person. Performs processing to identify actions. The server device 3 attaches these processing results to an image as incidental information and stores them. The server device 3 may perform scoring (scoring) of each image based on these processing results, store only the images whose scores exceed the threshold value, and discard the images whose scores do not exceed the threshold value. The server device 3 receives an image display request, an image condition to be displayed, and the like from the terminal device 5, extracts an image according to the request from a large number of stored and accumulated images, and transmits the image to the terminal device 5. The extracted image is displayed on the terminal device 5.

Further, in the information processing system according to the present embodiment, the employee of the facility 100 creates schedule information about the event (curriculum) performed at the facility on the terminal device 5, and the server device 3 acquires the schedule information. And remember. Based on the results of the various processes described above, the server device 3 aggregates the facial expressions and actions of the users at the event, for example, for each event performed at the facility 100 and for each user (child) of the facility 100. Then, the aggregated information is stored in association with the schedule information.

Further, in the information processing system according to the present embodiment, it is possible for an employee or the like of the facility 100 to create and register information such as a shooting location or an installation location of the camera 1 in advance on the terminal device 5. You may. For example, when the facility 100 is a nursery school, a kindergarten, or the like, information such as a sandbox, a garden, a pool, a playset, a Sakura-gumi classroom, or a tulip-gumi classroom can be registered by an employee or the like as a shooting location of the camera 1. .. For example, the camera 1 stores the registered information, and the camera 1 attaches the registered information to the captured image and transmits the registered information to the server device 3. Alternatively, the registered information may be stored in the server device 3 in association with, for example, the ID of the camera 1, and the information may be added to the image acquired from the camera 1. These information attached to the image may be used, for example, for scoring the image by the server device 3, or may be specified by the user as, for example, a condition of the image to be displayed, and the server device 3 may be specified according to the condition from the user. May be used in extracting the image.

In the information processing system according to the present embodiment, the user uses his / her own terminal device 5 to communicate with the server device 3, acquires images stored in the server device 3 and stores the images in the terminal device 5. It can be displayed. At this time, the terminal device 5 accepts the conditions and the like of the image acquired from the server device 3 and displays them from the user, specifies the conditions, and requests the server device 3 to transmit the image. The server device 3 extracts an image satisfying the conditions specified in the request from the terminal device 5 and transmits it to the request source terminal device 5. The terminal device 5 receives the image transmitted from the server device 3 and displays it on the display unit. In the present embodiment, the user can specify the facial expression or action of the person (child) shown in the image as a condition, and the server device 3 can specify the facial expression or action attached to the image as incidental information. An image with specified conditions is extracted based on the identification result.

Further, in the information processing system according to the present embodiment, the terminal device 5 acquires the schedule information of the event to be performed at the facility and the aggregated information such as facial expressions and actions associated with the schedule information from the server device 3. Display schedule information and aggregated information. The terminal device 5 displays, for example, a list of events for one day, and displays in characters or images what kind of facial expressions or actions the target user (child) was doing at each event.

Further, in the information processing system according to the present embodiment, the images and information that can be displayed on the terminal device 5 are different depending on the type of user of the terminal device 5 (for example, a guardian or a nursery teacher). There is. That is, the terminal device 5 used by the facility user (child) or its guardian or the like displays an image of this user, and displays the identification result of the user's facial expression or behavior. , Images of other users (only) and identification results such as facial expressions or actions of other users are not displayed. On the other hand, the terminal device 5 used by the employees engaged in the facility can display images of all users (children) who use the facility and identification results such as facial expressions or actions.

Further, in the information processing system according to the present embodiment, the user can add information such as a comment to the image displayed on the terminal device 5 (the image stored in the server device 3). The terminal device 5 receives an input such as a comment from the user and transmits it to the server device 3, and the server device 3 stores the information such as the comment in association with the image. When transmitting this image to the terminal device 5, the server device 3 transmits the image to the terminal device 5 with information such as a comment associated with the image. When the image received from the server device 3 is accompanied by information such as a comment, the terminal device 5 displays the information such as a comment together with the image. As a result, for example, information such as comments can be exchanged between the guardian of the facility user (child) and the facility worker.

In this example, the case where the child is photographed at the facility 100 such as a nursery school or a kindergarten has been described, but the application of the information processing system according to the present embodiment is not limited to the facility 100 such as a nursery school or a kindergarten. The facility 100 may be, for example, a commercial facility such as an amusement park or a theme park, or may be a facility such as a venue where a sport or an event is held, or may be, for example, a user's home. good. Further, the object to be photographed by the camera 1 does not have to be a person, and may be, for example, an animal in a zoo or a pet at home.

<Device configuration>
FIG. 2 is a block diagram showing a configuration of the server device 3 according to the present embodiment. The server device 3 according to the present embodiment includes a processing unit 31, a storage unit (storage) 32, a communication unit (transceiver) 33, and the like. In the present embodiment, it is assumed that the processing is performed by one server device, but a plurality of server devices may be distributed to perform the processing.

The processing unit 31 is configured by using an arithmetic processing unit such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit) or a GPU (Graphics Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory) and the like. Has been done. The processing unit 31 reads and executes the server program 32a stored in the storage unit 32 to acquire an image taken by the camera 1, and identifies a person, a person's facial expression, an action, and the like captured in the image. Processing, processing for selecting and storing images, processing for acquiring and storing schedule information created by employees of facility 100, processing for extracting and transmitting images in response to a request from the terminal device 5, etc. Performs various processes.

The storage unit 32 is configured by using a large-capacity storage device such as a hard disk. The storage unit 32 stores various programs executed by the processing unit 31 and various data required for processing by the processing unit 31. In the present embodiment, the storage unit 32 stores the server program 32a executed by the processing unit 31, the image storage unit 32b that stores and stores the image received from the camera 1, and the user DB that stores information about the user. (Database) 32c is provided.

In the present embodiment, the server program 32a is provided in a mode of being recorded on a recording medium 99 such as a memory card or an optical disk, and the server device 3 reads the server program 32a from the recording medium 99 and stores it in the storage unit 32. However, the server program 32a may be written to the storage unit 32, for example, at the manufacturing stage of the server device 3. Further, for example, in the server program 32a, the server device 3 may acquire what is distributed by another remote server device or the like by communication. For example, in the server program 32a, the writing device may read what has been recorded on the recording medium 99 and write it in the storage unit 32 of the server device 3. The server program 32a may be provided in the form of distribution via the network, or may be provided in the form of being recorded on the recording medium 99.

The image storage unit 32b stores an image received from the camera 1 and selected by the server device 3. The image storage unit 32b classifies a plurality of images according to, for example, the ID of the facility 100 in which the camera 1 is installed, the ID of each camera 1 when a plurality of cameras 1 are installed, the date and time when the images were taken, and the like. And remember. Further, in the present embodiment, the image storage unit 32b attaches and stores the identification result of the person captured in the image and the identification result such as the facial expression or behavior of the person as incidental information to the image.

Further, in the present embodiment, the image storage unit 32b stores the schedule information created by the facility worker or the like in the terminal device 5 and acquired by the server device 3. The schedule information includes, for example, information such as date information, information such as the name of an event to be performed on that day, and information such as time zone (start time and end time) of each event. The schedule information is, for example, free play from 9:00 to 10:00 on November 25, 2020, class time from 10:00 to 12:00, rice cooked from 12:00 to 13:00, and so on. Further, in the present embodiment, the image storage unit 32b stores information on the recognition result such as the facial expression and the behavior of the user (child) in each event in association with each event of the schedule information. For example, in a one-hour event from 10:00 to 11:00, user A has the most smiles, user B has the most angry expressions, user C has the most sad expressions, and so on. The information is stored in association with each event of the schedule information.

The user DB 32c is a database that stores information about a user who has registered for use of the service provided by the information processing system according to the present embodiment. FIG. 3 is a schematic view showing a configuration example of the user DB 32c. The user DB 32c of the server device 3 according to the present embodiment stores, for example, information of "user ID", "facility ID", and "target person" in association with each other. The "user ID" is identification information uniquely assigned to a user who uses the service provided by the information processing system according to the present embodiment. The user is, for example, a guardian or a nursery teacher of a child who uses a nursery school, and in this example, IDs such as guardian A, guardian B, nursery teacher X, and nursery teacher Y are attached as user IDs. The "facility ID" is identification information uniquely attached to the facility 100. In this example, an ID such as a nursery school a is attached as the facility ID.

The "target person" is information about a user (child) to be photographed by the camera 1 installed in the facility 100, and is information about a target person who can view an image or the like. The information of the "target person" may include information such as "name" and "feature amount". The "name" is the real name or nickname of the target person, and in this example, nicknames such as "Aoi-chan" and "Takuya-kun" are set. The "feature amount" is, for example, vector information indicating the facial features of the subject, which is extracted from a photograph of the face of the subject. This feature amount is used, for example, in a process of identifying who is the person shown in the image. In this example, for nursery teacher X and nursery teacher Y who are employees of facility 100, images can be viewed for all users of this facility 100, and "all" is set as "target person". ing.

The configuration of the user DB 32c included in the server device 3 is not limited to the one shown in the figure. In addition to the above information, the user DB 32c also includes, for example, a user's name, a facility 100's name, a terminal ID attached to the terminal device 5 used by the user, a user's address, a contact information such as a telephone number or an email address, and the like. Alternatively, various information such as a password for logging in to the service can be stored.

The communication unit 33 of the server device 3 communicates with various devices via a network N including a mobile phone communication network, a wireless LAN (Local Area Network), the Internet, and the like. In the present embodiment, the communication unit 33 communicates with the camera 1 and the terminal device 5 via the network N. The communication unit 33 transmits the data given by the processing unit 31 to another device, and gives the data received from the other device to the processing unit 31.

The storage unit 32 may be an external storage device connected to the server device 3. Further, the server device 3 may be a multi-computer including a plurality of computers, or may be a virtual machine virtually constructed by software. Further, the server device 3 is not limited to the above configuration, and may include, for example, a reading unit that reads information stored in a portable storage medium, an input unit that accepts operation input, a display unit that displays an image, and the like. ..

Further, in the server device 3 according to the present embodiment, the processing unit 31 reads out and executes the server program 32a stored in the storage unit 32, so that the image receiving processing unit 31a, the image identification processing unit 31b, and the information aggregation processing The unit 31c, the image selection unit 31d, the image transmission processing unit 31e, and the like are realized in the processing unit 31 as software-like functional units. In this figure, as the functional unit of the processing unit 31, the functional unit that performs processing on the image captured by the camera 1 is illustrated, and the functional unit related to other processing is omitted.

The image reception processing unit 31a performs a process of receiving the image transmitted by the camera 1 by the communication unit 33. For example, the image reception processing unit 31a associates the image with the shooting date and time of the image, the ID of the camera 1 that shot the image, the ID of the facility 100 in which the camera 1 is installed, and the like, based on the data attached to the received image. It is classified and temporarily stored in the storage unit 32. The image temporarily stored in the storage unit 32 by the image reception processing unit 31a is sorted by the image selection process and stored and stored in the image storage unit 32b of the storage unit 32, or the storage unit 32. Is erased from.

The image identification processing unit 31b has a process of detecting a person in the image, a process of identifying who is in the image, and a person in the image of the image acquired from the camera 1. The process of identifying the facial expression of the person, the process of identifying the behavior of the person captured in the image, and the like are performed. The image identification processing unit 31b attaches information regarding these processing results to the image as metadata. In the present embodiment, these processes are performed by the server device 3, but the present invention is not limited to this, and the camera 1 may perform some or all of these processes in advance.

The image identification processing unit 31b detects a person from an image by using, for example, a learning model that has been machine-learned in advance. The learning model receives, for example, image data as input, and machine learning is performed in advance so as to output an image area (data such as coordinates indicating) in which a person included in the image is captured. The learning model is machine-learned using, for example, teacher data in which image data and data indicating an image area in which a person included in the image is captured are associated with each other. In addition, when the target person to be detected based on the image taken at the facility 100 such as a nursery school or kindergarten is a child, the detection accuracy of the child is obtained by creating the teacher data using the image data of the child. Can be expected to increase. The image identification processing unit 31b inputs the image acquired from the camera 1 to the learning model, acquires the detection result of the person output by the learning model, and performs the process of detecting the person captured in the image. The image identification processing unit 31b removes this image and excludes it from the target of subsequent processing when a person is not detected from the image, that is, when no person is captured in the image.

The image identification processing unit 31b performs a process of identifying who is the person captured in the image, for example, based on the feature amount of the face of the person captured in the image. The image identification processing unit 31b extracts, for example, the feature amount of the human face captured in the image and compares it with the feature amount of the target person registered in the user DB 32c. The image identification processing unit 31b searches for a target person (child) whose feature amount matches or is similar to the feature amount of the human face captured in the image of the camera 1 and is registered in the user DB 32c, and the corresponding target person. Is judged to be the person shown in this image. The image identification processing unit 31b, for example, uses a learning model that accepts an image as an input and outputs multidimensional vector information as a feature amount of the human face captured in this image, and captures the image on the image captured by the camera 1. The feature amount of the person's face is extracted. The same learning model can be used when registering facial features in the user DB 32c. The image identification processing unit 31b calculates, for example, the distance between two vectors corresponding to the two feature quantities, and the one whose distance is equal to or less than the threshold value and whose distance is the shortest has the same or similar facial features. Then it can be judged.

The image identification processing unit 31b performs processing for identifying a person's facial expression captured in an image, for example, by using a learning model that has been machine-learned in advance. For example, the learning model accepts an image of a person as input, and machine learning is performed in advance so as to output a certainty that the facial expression of the person captured in this image is a predetermined facial expression. A given facial expression is, for example, a smile, anger or sadness. The image identification processing unit 31b may, for example, prepare a plurality of learning models that output the certainty of one type of facial expression to identify the plurality of types of facial expressions, and output the certainty of each of the plurality of types of facial expressions, for example. One learning model may be prepared for identification. The learning model is, for example, an image data and a label indicating whether or not the facial expression of the person captured in this image is a predetermined facial expression (label 1 if the facial expression is predetermined, and label 1 if the facial expression is other than the predetermined facial expression). Machine learning is performed using the teacher data associated with (label 0, etc.). In the present embodiment, the image identification processing unit 31b inputs the image taken by the camera 1 into the learning model, acquires the certainty of each facial expression output by the learning model, and the facial expression with the highest certainty is copied to the image. Judge as the facial expression of the person. If the highest certainty does not exceed the threshold value, the image identification processing unit 31b may determine that the facial expression of the person captured in the image is not any facial expression that can be identified by the learning model.

The image identification processing unit 31b performs a process of identifying an action performed by a person captured in an image by using, for example, a learning model that has been machine-learned in advance. The learning model accepts, for example, image data as input, and machine learning is performed in advance so as to output the degree of certainty that the behavior of the person captured in the image is a predetermined behavior. In the present embodiment, a learning model is created in advance for each action, such as a learning model that outputs the certainty of running and a learning model that outputs the certainty of eating as predetermined actions. Will be done. In the learning model, machine learning is performed using, for example, teacher data in which image data and a label indicating whether or not a person captured in the image is performing a predetermined action are associated with each other. In the present embodiment, the image identification processing unit 31b inputs the image captured by the camera 1 into each learning model, and acquires the certainty of each action output by the learning model. The image identification processing unit 31b compares the certainty of a plurality of actions, and determines that the action with the highest certainty is the action performed by the person captured in the image. If the highest certainty does not exceed the threshold value, the image identification processing unit 31b may determine that the behavior of the person captured in the image is not any behavior that can be identified by the learning model.

The information aggregation processing unit 31c performs a process of aggregating the information of the identification result by the image identification processing unit 31b. The information aggregation processing unit 31c reads the schedule information created in advance by the employee of the facility 100 from the storage unit 32, and for each event included in the schedule information, for example, any facial expression of each user (child) of the facility 100. Aggregate which action was the longest, which action was the longest, and so on. The information aggregation processing unit 31c stores these aggregation results as aggregation information in the storage unit 32 in association with each event of the schedule information.

FIG. 4 is a schematic diagram showing an example of aggregated information generated by the information aggregation processing unit. This figure shows the aggregated result of the number of times the user's facial expression is recognized from the image, but it is not limited to this, and for example, the user's behavior may be aggregated, and for example, the facial expression. Aggregation may be done for time rather than number of times. The aggregated information generated by the information aggregation processing unit 31c is, for example, information stored in association with information such as "date", "event", "target person", and "number of facial expressions".

The "date" is the date when the image was taken by the camera 1. The "event" is the information of the event set for the "date" extracted from the schedule information. For example, in the case of a facility 100 such as a nursery school or a kindergarten, "going to a kindergarten" and "free play" etc. Events can be set. The "target person" is a person identified as a person photographed in the image taken by the camera 1, and the "name" of the "target person" stored in the user DB 32c can be set. In this example, the names of "Aoi-chan" and "Takuya-kun" are set as "target persons". The "number of facial expressions" is the number of facial expressions of each user identified from the image taken by the camera 1, and in this example, the facial expressions are identified for the three types of "egao", "kana", and "occurrence". It is supposed to be.

According to the aggregated information of this example, in the event of "going to the park" on "November 25, 2020", "Aoi-chan" had "Egao" eight times and "Kanai" once, and "Takuya-kun". "" Is counted as "Egao" twice and "Occurrence" four times. Also, at the "Free Play" event on the same day, "Aoi-chan" had "Egao" 7 times and "Okake" once, and "Takuya-kun" had "Egao" 6 times and "Kanai" 2 times. Is aggregated.

The image selection unit 31d performs a process of selecting an image stored and accumulated in the image storage unit 32b from the images acquired from the camera 1 based on the identification result by the image identification processing unit 31b. The image selection unit 31d selects a predetermined number of images for one user for each event of the schedule, for example. At this time, the image selection unit 31d can select a predetermined number of images with high certainty output by the learning model, for example, when identifying facial expressions. Further, for example, the image selection unit 31d may select the image having the highest certainty at intervals such as 5 minutes or 10 minutes. The method of image selection by the image selection unit 31d is not limited to the above method, and any method may be adopted.

The image transmission processing unit 31e performs a process of transmitting the image stored in the image storage unit 32b to the user's terminal device 5. The image transmission processing unit 31e acquires the conditions given together with the image transmission request from the terminal device 5, extracts the images satisfying the acquired conditions from the plurality of images stored in the image storage unit 32b, and requests the request source. Is transmitted to the terminal device 5 of. The condition given by the terminal device 5 is, for example, "an image of Aoi-chan's smile on November 25, 2020". The image transmission processing unit 31e extracts an image satisfying the conditions from the terminal device 5 based on the metadata attached to each image by the image identification processing unit 31b.

FIG. 5 is a block diagram showing the configuration of the terminal device 5 according to the present embodiment. The terminal device 5 according to the present embodiment includes a processing unit 51, a storage unit (storage) 52, a communication unit (transceiver) 53, a display unit (display) 54, an operation unit 55, and the like. The terminal device 5 is a device used by a user of the facility 100, a guardian watching over the user, or a user such as an employee of the facility 100, and is an information processing device such as a smartphone, a tablet terminal device, or a personal computer. Can be configured using.

The processing unit 51 is configured by using an arithmetic processing unit such as a CPU or MPU, a ROM, and the like. The processing unit 51 reads and executes the program 52a stored in the storage unit 52 to accept input of conditions for the image to be displayed, and various processes such as displaying the image acquired from the server device 3. Perform processing.

The storage unit 52 is configured by using a non-volatile memory element such as a flash memory. The storage unit 52 stores various programs executed by the processing unit 51 and various data required for processing by the processing unit 51. In the present embodiment, the storage unit 52 stores the program 52a executed by the processing unit 51. In the present embodiment, the program 52a is distributed by a remote server device or the like, which is acquired by the terminal device 5 by communication and stored in the storage unit 52. However, the program 52a may be written in the storage unit 52, for example, at the manufacturing stage of the terminal device 5. For example, in the program 52a, the terminal device 5 may read the program 52a recorded on the recording medium 98 such as a memory card or an optical disk and store it in the storage unit 52. For example, in the program 52a, the writing device may read what was recorded on the recording medium 98 and write it in the storage unit 52 of the terminal device 5. The program 52a may be provided in a mode of distribution via a network, or may be provided in a mode recorded on a recording medium 98.

The communication unit 53 communicates with various devices via a network N including a mobile phone communication network, a wireless LAN, the Internet, and the like. In the present embodiment, the communication unit 53 communicates with the server device 3 via the network N. The communication unit 53 transmits the data given by the processing unit 51 to another device, and gives the data received from the other device to the processing unit 51.

The display unit 54 is configured by using a liquid crystal display or the like, and displays various images, characters, and the like based on the processing of the processing unit 51. The operation unit 55 accepts the user's operation and notifies the processing unit 51 of the accepted operation. For example, the operation unit 55 accepts a user's operation by an input device such as a mechanical button or a touch panel provided on the surface of the display unit 54. Further, for example, the operation unit 55 may be an input device such as a mouse and a keyboard, and these input devices may be configured to be removable with respect to the terminal device 5.

Further, in the terminal device 5 according to the present embodiment, the processing unit 51 reads out and executes the program 52a stored in the storage unit 52, so that the display processing unit 51a, the condition acquisition unit 51b, and the like become software-like functional units. It is realized in the processing unit 51. The program 52a may be a program dedicated to the information processing system according to the present embodiment, or may be a general-purpose program such as an Internet browser or a web browser.

The display processing unit 51a performs a process of displaying various characters, images, and the like on the display unit 54. In the present embodiment, the display processing unit 51a displays the UI (User Interface) screen described later on the display unit 54, and also displays the image acquired from the server device 3 in a predetermined area of the UI screen. Further, in the information processing system according to the present embodiment, the UI screen displayed on the terminal device 5 used by the user of the facility 100 or its guardian and the UI screen displayed on the terminal device 5 used by the employee of the facility 100. The display processing unit 51a displays the UI screen according to the user who is using the terminal device 5.

The condition acquisition unit 51b acquires conditions related to one or more images to be displayed on the UI screen based on the user's operation on the operation unit 55 when the display processing unit 51a displays the UI screen on the display unit 54. Perform the processing. In the information processing system according to the present embodiment, as conditions for the image to be displayed, the date when the image was taken, the facial expression (smile, anger, sadness, etc.) of the person shown in the image, and the use shown in the image. We accept people (children) and scheduled events. The condition acquisition unit 51b gives a transmission request for an image with the acquired conditions to the server device 3. In response to this request, the server device 3 transmits an image conforming to the conditions to the terminal device 5, and the display processing unit 51a of the terminal device 5 that receives this image displays the image.

<Image display processing>
FIG. 6 is a schematic view showing an example of a UI screen displayed by the terminal device 5 according to the present embodiment. The UI screen shown in FIG. 6 is a UI screen displayed by the terminal device 5 used by the employee of the facility 100. The UI screen displayed by the terminal device 5 according to the present embodiment includes a date display area 101, a facial expression selection area (first area) 102, a target person selection area 103, and a main image display area (third area) 104. It is configured to include a secondary image display area (second area) 105, a schedule display area (fourth area) 106, and the like.

The date display area 101 is provided at the upper part of the UI screen, and in this example, the date and the day of the week are displayed as in "November 25, 2020 wed." In the information processing system according to the present embodiment, the illustrated UI screen is displayed after the user activates the program 52a on the terminal device 5 in order to use the service. At this time, the date of yesterday is displayed in the date display area 101 of the UI screen, and the user can browse the image taken by the camera 1 of the facility 100 yesterday. The user can change the date by performing a touch operation, a click operation, or the like on the date display area 101. When the terminal device 5 accepts an operation on the date display area 101, it displays, for example, a calendar or the like and accepts the date selection. The terminal device 5 that has received the selection of the date gives a request for transmitting the image of this date to the server device 3, and receives and displays the image transmitted from the server device 3 in response to the request.

The facial expression selection area 102 is provided at the left end of the UI screen, and for example, character strings indicating facial expressions such as “Kana”, “Egao”, and “Okori” are displayed side by side as options. Further, the terminal device 5 displays a character string indicating a facial expression for each option, an icon (image, figure, symbol, etc.) indicating the facial expression, and the number of images corresponding to the facial expression. In the illustrated example, it is shown that there is one image of "Kana", eight images of "Egao", and two images of "Okashi", and "Egao" is selected. The terminal device 5 accepts facial expression selection by accepting a touch operation, a click operation, or the like for any of the options displayed in the facial expression selection area 102. The terminal device 5 that has received the selection of the facial expression gives a request for transmitting the image of the facial expression to the server device 3, and receives and displays the image transmitted from the server device 3 in response to the request.

The target person selection area 103 is provided in the upper part of the main image display area 104, which will be described later, provided in the center of the UI screen, and is an area for accepting the selection (switching) operation of the target person in the form of a so-called tab. In the illustrated example, the names of four selectable subjects such as "Aoi-chan", "Takuya-kun", "Hana-chan" and "Ren-kun" are displayed on each tab, and "Aoi-chan" is displayed in each tab. Is selected. Further, the terminal device 5 displays the name of the target person as a character string for each tab, and also displays the number of images corresponding to the target person. In the illustrated example, it is shown that there are four images of "Aoi-chan", two images of "Takuya-kun", one image of "Hana-chan", and one image of "Ren-kun". There is. The terminal device 5 accepts the selection of the target person to display the image by accepting the touch operation, the click operation, or the like for any tab provided in the target person selection area 103. The terminal device 5 that has received the selection of the target person gives a request for transmitting the image of the target person to the server device 3, and receives and displays the image transmitted from the server device 3 in response to the request.

Since the illustrated example is a UI screen displayed by the terminal device 5 used by the employee of the facility 100, all the users (children) who use the facility 100 can be selected in the target person selection area 103. .. However, for example, when the facility 100 is a nursery school or the like and the users are classified, the employee can select all the users of the class in charge, and the users of other classes. You do not have to be able to select.

Although not shown, the target person selection area 103 of the UI screen displayed by the terminal device 5 used by the guardian of the user (child) is provided as a tab that can be selected only by the guardian's child. No other child can be selected. For example, if there is only one child using the facility 100, there is only one tab provided in the target person selection area 103, and the target person does not have to be switched. Further, for example, when a plurality of children such as siblings are using the facility 100, the UI screen displayed by the terminal device 5 used by the guardians of these children is a tab about the plurality of children of the guardian. Is provided, and the target person is switched. The server device 3 or the terminal device 5 can determine who can be selected in the target person selection area 103 of the UI screen based on the information of the "target person" of the user DB 32c.

The main image display area 104 is provided in the center of the UI screen, and displays one image selected in the sub-image display area 105 described later. Further, the image stored and displayed by the information processing system according to the present embodiment may be either a still image or a moving image, and the still image and the moving image may be mixed. When displaying a moving image in the main image display area 104, the terminal device 5 displays a button for accepting operations such as playback and stop, a slide bar indicating the playback time, and the like at the bottom of the main image display area 104. In the illustrated example, a moving image is displayed in the main image display area 104. When displaying a still image in the main image display area 104, these buttons, slide bar, and the like may not be displayed.

The sub-image display area 105 is provided on the right side of the main image display area 104, and is an area for displaying a list of one or more images matching the conditions selected by the user side by side in the vertical direction. Each image displayed in the list may be a reduced version of the original image, a so-called thumbnail, and the thumbnail image may be created by either the server device 3 or the terminal device 5. In the illustrated example, three images are listed side by side in the sub-image display area 105 in the vertical direction. When displaying a list of more images in the slave image display area 105, the terminal device 5 displays, for example, a scroll bar or the like on the side of the slave image display area 105, and performs a scroll operation on the slave image display area 105. You may accept and scroll the image to be displayed in the slave image display area 105 to switch. Further, the display order of the plurality of images listed in the sub-image display area 105 may be determined according to, for example, the shooting time, and for example, when the facial expression condition is selected as the facial expression condition, the degree of certainty of a smile is determined. It may be determined accordingly, or it may be determined by a method other than these. The terminal device 5 accepts a user's selection of an image by accepting a touch operation, a click operation, or the like on one or a plurality of images listed in the sub-image display area 105. The terminal device 5 displays the image selected in the sub-image display area 105 in the main image display area 104.

The schedule display area 106 is provided at the bottom of the UI screen, and displays the schedule of the event (curriculum) implemented (planned to be implemented) at the facility 100 on the date displayed in the date display area 101. In the illustrated example, a schedule is shown by arranging a plurality of events in the horizontal direction in a band graph with the horizontal axis as time. The events shown in the schedule of this example are "going to the park" from 8:00 to 9:00, "free play" from 9:00 to 10:00, and 10:00 to 12:00. Class time is from 12:00 to 13:00 for "lunch" and from 13:00 to 14:00 for "free play".

Further, the terminal device 5 displays icons corresponding to facial expressions such as “egao”, “kana”, and “occurrence” in association with each event in the schedule display area 106. The facial expression icon display can be performed by the terminal device 5 based on the facial expression information aggregated by the server device 3 for the target person selected in the target person selection area 103. In the illustrated example, an icon with the expression "Kana" is displayed for "going to the park" from 8:00 to 9:00, "free play" from 9:00 to 10:00, and 12:00 to 13:00. An icon with the expression "Egao" is displayed for "Lunch" and "Free play" from 13:00 to 14:00. For "free play" from 13:00 to 14:00, two icons with the expression "Egao" are displayed. For example, this is the certainty of the smile identified from the image taken at this event. It is highlighted by increasing the number of icons to be displayed when it is higher than a predetermined threshold value or when the smiling face during the time of this event is longer than a predetermined threshold value.

Further, in the left side portion of the schedule display area 106, the number of images for facial expressions such as "egao", "kana", and "occurrence" is shown. In the illustrated example, it is shown that there are four images of "Egao", one image of "Kanai", and zero images of "Okashi".

Further, the terminal device 5 accepts the selection of an event by accepting a touch operation, a click operation, or the like for the event of the schedule displayed in the schedule display area 106. The terminal device 5 that has received the selection of the event gives a request for transmitting the image of this event to the server device 3, and receives and displays the image transmitted from the server device 3 in response to the request.

In the information processing system according to the present embodiment, the conditions of the image to be displayed can be set by the user performing an operation using the UI screen displayed by the terminal device 5. For example, the user can set the date condition in the date display area 101, set the facial expression condition in the facial expression selection area 102, and set the target person's condition in the target person selection area 103. The event conditions can be set in the schedule display area 106. The user does not have to set all of these plurality of conditions, but may set one or more conditions as appropriate. The terminal device 5 transmits an image satisfying the conditions set in these areas to the server device 3. The server device 3 extracts an image satisfying the conditions in response to the request from the terminal device 5 from the images stored in the image storage unit 32b, and transmits the extracted image to the request source terminal device 5. The terminal device 5 receives the image transmitted from the server device 3 and displays a list of the received one or a plurality of received images in the sub-image display area 105 of the UI screen. The user selects one image from the images listed in the sub-image display area 105 by a touch operation, a click operation, or the like, and the terminal device 5 selects the selected image in the main image display area 104 in response to this selection. Display on.

Although not shown, in the information processing system according to the present embodiment, information such as a comment can be added to the image displayed in the main image display area 104 or the sub-image display area 105. For example, when a predetermined operation (for example, double-clicking) is performed on the image displayed in the main image display area 104, the terminal device 5 displays a screen for inputting information such as a comment, and a character, an image, or the like is displayed. Accepts input. The terminal device 5 transmits information such as a comment that has received the input to the server device 3, and the server device 3 associates this information with the image and stores it in the image storage unit 32b.

Further, when the server device 3 transmits an image in response to a request from the terminal device 5, if information such as a comment associated with the image is stored, the server device 3 attaches this information to the image and attaches the information to the terminal device. Send to 5. When the terminal device 5 receives an image with information such as a comment, for example, when displaying this image in the main image display area 104, the information such as a comment is superimposed on the image and displayed. Information such as comments is displayed at predetermined locations such as up, down, left, and right of the area 104, or an icon or the like indicating that information such as comments is attached is superimposed on the image and displayed, and a touch operation or touch operation on this icon is performed. Information such as comments can be displayed by various methods such as displaying information such as comments in response to a click operation or the like.

FIG. 7 is a flowchart showing a procedure of image storage processing performed by the server device 3 according to the present embodiment. The image reception processing unit 31a of the processing unit 31 of the server device 3 according to the present embodiment determines whether or not the image transmitted by the camera 1 installed in the facility 100 has been received (step S1). When the image is not received (S1: NO), the image reception processing unit 31a waits until the image is received from the camera 1.

When an image is received from the camera 1 (S1: YES), the image identification processing unit 31b of the processing unit 31 performs a process of detecting a person captured in the image captured by the camera 1 (step S2). Based on the result of this processing, the image identification processing unit 31b determines whether or not a person is captured in the image (step S3). When no person is shown in the image (S3: NO), the image identification processing unit 31b discards this image (step S11) and returns the processing to step S1.

When a person is captured in the image (S3: YES), the image identification processing unit 31b acquires the feature amount of the person captured in the image and copies it to the image based on the feature amount stored in the user DB 32c. A process of identifying who the person is (step S4) is performed. The image identification processing unit 31b performs a process of identifying a person's facial expression reflected in the image (step S5). Further, the image identification processing unit 31b performs a process of identifying the behavior of the person captured in the image (step S6). Next, the information aggregation processing unit 31c of the processing unit 31 associates the person identified in step S4 with each event of the schedule information on the day when the image was taken, and the facial expression identified in step S5 and the facial expression identified in step S6. A process of aggregating information such as identified actions is performed (step S7). Based on the result of the aggregation process in step S7, the information aggregation processing unit 31c stores the aggregation result of the facial expression and action of the person captured in the image as aggregation information in the storage unit 32 in association with each event of the schedule information. (Step S8).

The image selection unit 31d of the processing unit 31 performs image scoring based on information such as the facial expression identified in step S5 and the behavior identified in step S6, and determines whether or not the score exceeds the threshold value. By doing so, it is determined whether or not to store this image (step S9). When it is determined that the image is to be stored (S9: YES), the image selection unit 31d stores this image in the image storage unit 32b (step S10), and returns the process to step S1. When it is determined that the image is not stored (S9: NO), the image selection unit 31d discards this image (step S11) and returns the process to step S1.

In the present embodiment, a process of detecting a person from an image captured by the camera 1, a process of identifying who is the person captured in the image, a process of identifying the facial expression of the person captured in the image, In addition, the server device 3 shall perform processing for identifying the behavior of the person shown in the image, but the present invention is not limited to this. Part or all of these plurality of processes may be performed by the camera 1. For example, the camera 1 may perform a process of detecting a person from an image taken by itself, and may discard the image in which the person is not photographed without transmitting it to the server device 3. Further, for example, the camera 1 may perform processing such as identification of the facial expression of a person captured in the image, and the processing result may be attached to the image and transmitted to the server device 3.

FIG. 8 is a flowchart showing a procedure of image display processing performed by the terminal device 5 according to the present embodiment. In the terminal device 5 according to the present embodiment, when a program 52a such as an application program for using this service is activated, the display processing unit 51a of the processing unit 51 displays the UI screen shown in FIG. 6 ( Step S21). The UI screen immediately after startup may be in a state in which no image is displayed in, for example, the main image display area 104 and the sub-image display area 105, and may be set to the default conditions defined for the conditions such as the facial expression and the target person. The image extracted based on may be displayed. When displaying an image based on the default conditions, the terminal device 5 requests the server device 3 to transmit an image under the default conditions prior to displaying the UI image, and the server device 3 transmits in response to this request. The image to be displayed is acquired and displayed in the main image display area 104 and the sub-image display area 105.

In the state where the UI screen is displayed, the condition acquisition unit 51b of the processing unit 51 determines whether or not the target person for image display has been selected in the target person selection area 103 of the UI screen (step S22). When the target person has not been selected (S22: NO), the condition acquisition unit 51b determines whether or not the facial expression has been selected in the facial expression selection area 102 of the UI screen (step S23). When the facial expression is not selected (S23: NO), the condition acquisition unit 51b determines whether or not the event has been selected in the schedule display area 106 (step S24). If no event has been selected (S24: NO), the condition acquisition unit 51b proceeds to step S28.

When the target person is selected on the UI screen (S22: YES), the facial expression is selected (S23: YES), or the event is selected (S24: YES), the condition acquisition unit 51b The selected condition is acquired, and the server device 3 is requested to transmit an image that matches this condition (S25). The condition acquisition unit 51b receives the image transmitted by the server device 3 in response to this request (step S26). The display processing unit 51a displays a list of one or more images received in step S26 in the sub-image display area 105 (S27).

Next, the condition acquisition unit 51b determines whether or not one image has been selected from the images listed in the sub-image display area 105 (step S28). If no image has been selected (S28: NO), the condition acquisition unit 51b returns to step S21 and continues to display the UI image. When an image is selected (S28: YES), the display processing unit 51a displays the selected one image in the main image display area 104 (step S29), returns the processing to step S21, and returns the processing to the UI image. Continue displaying.

FIG. 9 is a flowchart showing a procedure of image display processing performed by the server device 3 according to the present embodiment. The image transmission processing unit 31e of the server device 3 according to the present embodiment determines whether or not an image transmission request has been given from the terminal device 5 (step S41). When the transmission request is not given (S41: NO), the image transmission processing unit 31e waits until the transmission request is given.

When an image transmission request is given from the terminal device 5 (S41: YES), the image transmission processing unit 31e acquires the image conditions attached to the transmission request (step S42). The image transmission processing unit 31e extracts an image of the target date set under the conditions acquired in step S42 from the plurality of images stored in the image storage unit 32b (step S43).

The image transmission processing unit 31e extracts the image of the target person set under the conditions acquired in step S42 from the image extracted in step S43 (step S44). The image transmission processing unit 31e extracts an image of the facial expression of the target set under the conditions acquired in step S42 from the image extracted in step S44 (step S45). The image transmission processing unit 31e extracts an image of the target event set under the conditions acquired in step S42 from the image extracted in step S45 (step S46).

The conditions acquired in step S42 may not be set for all of the target date, target person, facial expression, or event, for example. The image transmission processing unit 31e may omit the processing of the corresponding steps S43 to S46 for the conditions that have not been set. Further, for example, a plurality of target persons or a plurality of facial expressions may be set as conditions, and in this case, the image transmission processing unit 31e may extract images corresponding to each condition.

The image transmission processing unit 31e transmits one or a plurality of images finally extracted in step S46 to the requesting terminal device 5 (step S47), and ends the process. If the image corresponding to the set condition does not exist, the image transmission processing unit 31e notifies the terminal device 5 that the image does not exist.

<Others>
When identifying a person captured in an image captured by the camera 1, the identification accuracy may decrease depending on, for example, the size or orientation of the person's face in the image. Therefore, the server device 3 may identify the person by using, for example, a process of tracking the person (tracking process) captured in the image. The server device 3 can be expected to have high identification accuracy, such as when a person's face is enlarged in the image or when the person's face in the image is facing the front (camera 1). Based on the identification result of a person based on the image, tracking processing is performed on one or more images following this image. The tracking process is a process of tracking the movement of an object in a plurality of continuously captured images, for example, Boosting, MIL (Multiple Instance Learning), TLD (Tracking Learning and Detection), Medical Flow, or KCF (Kernelized Correlation). Methods such as Filters) can be used. The server device 3 uses the tracking process to determine the identification result of a person (for example, the certainty of identification) in the reference image for a plurality of images that can be determined to be the same person, and the identification result for the subsequent image. Then, the information of the identification result may be attached to the subsequent image and stored.

<Summary>
In the information processing system according to the present embodiment having the above configuration, the terminal device 5 receives the facial expression selection regarding the target person who displays the image in the facial expression selection area 102 of the UI screen. The terminal device 5 acquires an image extracted by the server device 3 according to the selected facial expression from the image taken by the camera 1 installed in the facility 100 used by the target person, and displays the main image display area 104 and the secondary image. Displayed in area 105. Further, the terminal device 5 displays the correspondence between the event carried out at the facility 100 and the facial expression of the target person in the schedule display area 106. As a result, the information processing system can provide an image of the user of the facility 100 taken on a daily basis together with the user's facial expression, schedule information, and the like.

Further, in the information processing system according to the present embodiment, the terminal device 5 displays a plurality of characters and icons associated with facial expressions as selection candidates in the facial expression selection area 102 of the UI screen, and accepts a user's selection operation for the selection candidates. And accepts choices regarding the subject's facial expressions. As a result, the user can easily set the image to be displayed on the condition of the facial expression of the target person.

Further, in the information processing system according to the present embodiment, the terminal device 5 displays each selection candidate of the facial expression selection area 102 and the number of images that can be displayed when the selection candidate is selected. As a result, the user can know in advance the number of images to be displayed according to the facial expression conditions, and can be expected to facilitate the selection of the conditions.

Further, in the information processing system according to the present embodiment, the terminal device 5 accepts the selection of the target person to display the image by the tab provided in the target person selection area 103 of the UI screen. The terminal device 5 acquires an image extracted from the image storage unit 32b by the server device 3 according to the selected target person, and displays the image in the main image display area 104 and the sub-image display area 105. Further, the terminal device 5 displays the correspondence between the event and the facial expression for the selected target person in the schedule display area 106. As a result, when there are a plurality of target persons, the user can easily select the target person and browse the image.

Further, in the information processing system according to the present embodiment, the terminal device 5 accepts, for example, a touch operation or a click operation for any of the events based on the correspondence between the events and the facial expressions displayed in the schedule display area 106 of the UI screen. By doing so, the selection of the event is accepted. The terminal device 5 acquires an image extracted from the image storage unit 32b by the server device 3 in response to the selected event and displays it in the main image display area 104 and the sub-image display area 105. As a result, the user can easily set an event as a condition for the image to be displayed.

Further, in the information processing system according to the present embodiment, the terminal device 5 lists and displays one or a plurality of images extracted by the server device 3 according to the conditions in the sub-image display area 105 of the UI screen. It accepts the selection of one image from a plurality of images and displays the selected image in the main image display area 104. As a result, when there are a plurality of images that meet the conditions set by the user, the user can easily select one desired image from the plurality of images.

Further, in the information processing system according to the present embodiment, a facial expression selection area 102 that accepts facial expression selection, a sub-image display area 105 that displays a list of one or more images extracted according to the selected facial expression, and a list. The terminal device 5 has a UI screen including a main image display area 104 for displaying an image selected from the displayed images and a schedule display area 106 for displaying an event carried out at the facility 100 in association with the facial expression of the target person. Is displayed on the display unit 54. As a result, the user can set conditions such as facial expressions and events on the displayed UI screen, acquire images according to the conditions from the server device 3, and display them on the terminal device 5.

Further, in the information processing system according to the present embodiment, the terminal device 5 used by the user (child) of the facility 100 or a related person (guardian) of the facility 100 displays an image related to this user. On the other hand, the terminal device 5 used by the employees of the facility 100 displays images relating to a plurality of users of the facility 100. As a result, the information processing system can be expected to display an appropriate image to the user of the terminal device 5.

In the present embodiment, the facility 100 in which the camera 1 is installed is a nursery school or a kindergarten, and the camera 1 takes a picture of a child who uses the facility 100. However, the facility 100 is limited to the nursery school or the kindergarten. However, it may be any facility. Facility 100 may be indoors or outdoors. Further, a plurality of cameras 1 may be installed in the facility 100, and the images captured by the plurality of cameras 1 may be transmitted to the server device 3. Further, although it is assumed that the server device 3 performs the extraction of the image according to the conditions, the present invention is not limited to this, and the terminal device 5 may perform the extraction.

The UI screen shown in FIG. 6 is an example and is not limited to this, and the arrangement and design of each area can be changed as appropriate. Further, in the present embodiment, the facial expression selection is accepted in the facial expression selection area 102 of the UI screen, but instead of the facial expression selection or together with the facial expression selection, the behavior of the person captured in the image is a condition. It may be selectable.

The embodiments disclosed this time should be considered as exemplary in all respects and not restrictive. The scope of the present invention is indicated by the scope of claims, not the above-mentioned meaning, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

1 Camera 3 Server device 5 Terminal device 31 Processing unit 31a Image reception processing unit 31b Image identification processing unit 31c Information aggregation processing unit 31d Image selection unit 31e Image transmission processing unit 32 Storage unit 32a Server program 32b Image storage unit 32c User DB
33 Communication unit 51 Processing unit 51a Display processing unit 51b Condition acquisition unit 52 Storage unit 52a Program 53 Communication unit 54 Display unit 55 Operation unit 98, 99 Recording medium 100 Facility 101 Date display area 102 Facial expression selection area 103 Target person selection area 104 Main Image display area 105 Subordinate image display area 106 Schedule display area

Claims (13)

Information processing device
Accepts facial expression selections regarding the subject,
An image extracted according to the selected facial expression from the image taken by the camera installed in the facility used by the target is displayed.
Display the correspondence between the event held at the facility and the facial expression of the target.
Information processing method. Multiple characters or images associated with facial expressions are displayed as selection candidates, and
Accepts the selection of facial expressions related to the target based on the selection candidates.
The information processing method according to claim 1. Along with the selection candidate, the number of images that can be displayed when the selection candidate is selected is displayed.
The information processing method according to claim 2. Accept the selection of the target,
Display the extracted image according to the selected facial expression and target,
Display the correspondence between the event held at the facility and the facial expression of the selected target.
The information processing method according to any one of claims 1 to 3. Accepts event selection based on the displayed event and facial expression mapping,
Display the extracted images according to the selected event,
The information processing method according to any one of claims 1 to 4. List one or more extracted images and
Accepts the selection of one image from one or more images displayed in the list,
Display the selected image,
The information processing method according to any one of claims 1 to 5. A first area that displays multiple characters or images associated with facial expressions as selection candidates and accepts selection of facial expressions related to the target, and
A second area that displays a list of one or more images extracted according to the facial expression selected in the first area and accepts the selection of one image, and
A third area for displaying the image selected in the second area, and
A screen including an event carried out at the facility and a fourth area for displaying the association of facial expressions of the target is displayed on the display unit.
The information processing method according to any one of claims 1 to 6. The information processing device used by the user who uses the facility or a person related to the user displays an image related to the user and displays an image related to the user.
The information processing device used by the employee of the facility displays an image relating to a plurality of users who use the facility.
The information processing method according to any one of claims 1 to 7. Display the correspondence between the schedule of a plurality of events carried out at the facility and the facial expression of the target in each event.
The information processing method according to any one of claims 1 to 8. On the computer
Accepts facial expression selections regarding the subject,
An image extracted according to the selected facial expression from the image taken by the camera installed in the facility used by the target is displayed.
A computer program that performs a process of displaying the correspondence between an event carried out at the facility and the facial expression of the target. On the computer
Acquire the image taken by the camera installed in the facility and
Get information on the events held at the facility
Identify the facial expression of the target in the acquired image and
The event carried out at the facility and the facial expression of the identified object are associated with each other and stored in the storage unit.
A computer program that performs a process of transmitting information that associates an event carried out at the facility with the facial expression of the target to a terminal device. The reception department that accepts the selection of facial expressions related to the target,
An image display processing unit that performs processing to display an image extracted according to a selected facial expression from an image taken by a camera installed in the facility used by the target.
An information processing device including a corresponding display processing unit that performs processing for displaying the correspondence between an event carried out at the facility and the facial expression of the target. An image acquisition unit that acquires images taken by cameras installed in the facility,
The information acquisition department that acquires information on events held at the facility,
An identification unit that identifies the facial expression of the target imaged in the image acquired by the image acquisition unit, and an identification unit.
A storage unit that stores the event performed at the facility and the facial expression of the identified object in association with each other.
An information processing device including an information transmission unit that transmits information in which an event carried out at the facility and a facial expression of the target are associated with each other to a terminal device.

Images