KR20240175152A - Smart refrigerator system and food information provision method of smart refrigerator - Google Patents

Abstract

A smart refrigerator system according to an embodiment includes: a server which implements a food judgment model by training a deep learning neural network with a training data set, and distributes the food judgment model; a smart refrigerator which recognizes food detailed information including the type, amount, expiration date, and manufacturing date of food through the food judgment model received from the server, and tracks the location of food inside or outside the refrigerator, and displays the food detailed information and the food location; and a user terminal which outputs the food detailed information and the food location by linking with the smart refrigerator.

Description

{SMART REFRIGERATOR SYSTEM AND FOOD INFORMATION PROVISION METHOD OF SMART REFRIGERATOR}

The present disclosure relates to a smart refrigerator system and a method for providing food information of a smart refrigerator, and more particularly, to a smart refrigerator system and a method for providing food information that provide detailed food information including the type, location, expiration date, and manufacturing date of food stored in a smart refrigerator.

Unless otherwise indicated herein, the materials described in this section are not prior art to the claims of this application and their inclusion in this section is not an admission that they are prior art.

The refrigerator keeps various foods in a low temperature state by repeating cooling for a certain period of time to prevent spoilage. In general, since various foods are stored in the refrigerator, there are many cases where you cannot remember what you stored after a certain period of time. Therefore, when you want to cook or eat food, you have to open the refrigerator and check the type of food you have stored.

In addition, since we do not remember all the foods stored in the refrigerator and the expiration date of each food, we sometimes buy the same food repeatedly. Buying duplicate food increases the frequency of discarding food stored in the refrigerator and has a negative impact on the household economy. In addition, it causes the problem of reducing the space utilization efficiency of the refrigerator.

Meanwhile, smart refrigerators provide various additional functions by connecting to the Internet through a touch screen interface and wireless communication network. In addition, smart refrigerators can be controlled by smart terminals using the Internet of Things, or linked with other home appliances such as smart TVs, smart dishwashers, smart microwave ovens, serving robots, and robot vacuum cleaners.

1. Korean Patent Publication No. 10-2017-0054761 (May 18, 2017) 2. Korean Patent Publication No. 10-2017-0116557 (October 19, 2017)

A smart refrigerator system according to an embodiment and a method for providing food information of a smart refrigerator sense a container for storing food, a food type, and a storage start time, and provides detailed food information including the type, location, expiration date, and manufacturing date of food stored in a smart refrigerator, as well as the storage period for each food and the location of the food.

In addition, the smart refrigerator system according to the embodiment and the food information providing method of the smart refrigerator recognize medicines, ointments, cosmetics, and various processed foods from around the world stored in the refrigerator, read letters and pictures written on the packaging of the processed foods, and provide detailed information including the expiration date to the user through voice and display.

In addition, the smart refrigerator according to the embodiment can receive the type of food and the date of manufacture via voice or text input, and can track the storage location of food inside the refrigerator and the food movement path within the home by linking with various IoT devices in the home.

In addition, the smart refrigerator according to the embodiment can provide food detail information to the user terminal by linking with the user terminal and control the refrigerator temperature through the user terminal.

A smart refrigerator system according to an embodiment may include: a server which implements a food judgment model by training a deep learning neural network with a training data set, and distributes the food judgment model; a smart refrigerator which recognizes food detailed information including the type, amount, expiration date, and manufacturing date of food through the food judgment model received from the server, tracks the location of food inside or outside the refrigerator, and displays the food detailed information and the food location; a user terminal which outputs the food detailed information and the food location in conjunction with the smart refrigerator; and an Internet of Things home appliance which transmits a food image to the smart refrigerator in conjunction with the smart refrigerator to track the location of food outside the refrigerator.

In an embodiment, a smart refrigerator; includes: a sensor for sensing food, text and an image of the food packaging container, and a container containing the food; a communication unit for receiving a food judgment model from a server and linking with a user terminal and an Internet of Things home appliance; an extraction unit for identifying the type and amount of food sensed by the sensor through the food judgment model, and extracting an expiration date and a manufacturing date from the text and image of the food packaging container; a display unit for displaying detailed food information including the type and amount of the food, the expiration date, and the manufacturing date, the food storage period, and the food location;

The smart refrigerator system and the method for providing food information in the smart refrigerator described above enable users to know detailed information such as the type, location, and expiration date of food stored in the refrigerator, thereby reducing the frequency of repeat purchases or disposal of food.

In addition, the utilization of food stored in the refrigerator is increased through examples, and the location of food inside and outside the refrigerator is tracked to store food more freshly and efficiently.

The effects of the present invention are not limited to the effects described above, and should be understood to include all effects that can be inferred from the detailed description of the present invention or the composition of the invention described in the claims.

Figure 1 is a drawing showing the configuration of a smart refrigerator system according to an embodiment.
Figure 2 is a drawing showing the data processing configuration of a smart refrigerator (100) according to an embodiment.
Figure 3 is a drawing showing an example of an internal sensor configuration.
Figure 4 is a drawing showing the data processing configuration of a server (200) according to an embodiment.
Figure 5 is a diagram showing a data processing configuration of a smart terminal according to an embodiment.
Figure 6 is a signal flow diagram of a smart refrigerator system according to an embodiment.

The advantages and features of the present invention, and the methods for achieving them, will become clear with reference to the embodiments described in detail below together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and the present embodiments are provided only to make the disclosure of the present invention complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

In describing embodiments of the present invention, if it is judged that a detailed description of a known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description will be omitted. In addition, the terms described below are terms defined in consideration of functions in embodiments of the present invention, and these may vary depending on the intention or custom of the user or operator. Therefore, the definitions should be made based on the contents throughout this specification.

Figure 1 is a drawing showing the configuration of a smart refrigerator system according to an embodiment.

Referring to FIG. 1, a smart refrigerator system according to an embodiment may be configured to include a smart refrigerator (100), a server (200), a user terminal (300), and an Internet of Things home appliance (400).

The server (200) trains a deep learning neural network with a training data set to implement a food judgment model and distributes the food judgment model. The food judgment model is an artificial intelligence model for judging the type and amount of food based on food images, food shapes, packaging containers, and wrapping paper sensed in a smart refrigerator, and extracting detailed food information such as manufacturing date and expiration date. In the embodiment, the server (200) trains the food judgment model through a training data set including the type of food matched to the food image, expiration date, storage period for each food, etc. In addition, the server (200) can train the food judgment model with new food information received from an external server to update the food judgment model.

The smart refrigerator (100) recognizes food detailed information through the food judgment model received from the server (200). In the embodiment, the food detailed information is information related to food stored in the refrigerator. For example, the food detailed information may include the type of food stored, expiration date, manufacturing date, storage period, etc. In the embodiment, the smart refrigerator (100) recognizes food detailed information sensed from a sensor through the food judgment model, tracks the location of each food located inside or outside the refrigerator, and displays the food detailed information and the food location.

The user terminal (300) outputs food details and food locations in conjunction with the smart refrigerator (100). In the embodiment, at least one user terminal (300) may be implemented as a computer that can access a remote server or terminal via a network. Here, the computer may include, for example, a notebook, desktop, laptop, etc. equipped with a navigation system and a web browser. At this time, at least one user terminal (300) may be implemented as a terminal that can access a remote server or terminal via a network. At least one user terminal (300) may include, for example, all kinds of handheld-based wireless communication devices, such as a navigation, a PCS (Personal Communication System), a GSM (Global System for Mobile communications), a PDC (Personal Digital Cellular), a PHS (Personal Handyphone System), a PDA (Personal Digital Assistant), an IMT (International Mobile Telecommunication)-2000, a CDMA (Code Division Multiple Access)-2000, a W-CDMA (W-Code Division Multiple Access), a Wibro (Wireless Broadband Internet) terminal, a smartphone, a smartpad, a tablet PC, etc., as a wireless communication device that ensures portability and mobility.

The Internet of Things home appliance (400) is an Internet of Things-enabled home appliance installed in a home. In an embodiment, the Internet of Things home appliance (400) is linked to a user terminal (300) and a smart refrigerator (100), and tracks the location of food taken out of the refrigerator and transmits the location to the smart refrigerator (100) and the user terminal (300). In addition, in an embodiment, the Internet of Things home appliance (400) transmits a food image to the smart refrigerator (100), so that the location of food outside the refrigerator can be tracked through the food image.

In an embodiment, the IoT home appliance (400) may include all home appliances that can be controlled via the IoT, such as a wall pad, a television, a robot vacuum cleaner, a serving robot, a cooking robot observation camera, and lighting. In an embodiment, the smart refrigerator (100) may store the manufacturing date and manufacturing time of the food in conjunction with the cooking robot when food is manufactured and stored by the cooking robot.

In addition, in an embodiment, when a series of tasks with a load of a certain level or higher are processed through artificial intelligence in a smart refrigerator (100), the tasks can be divided or collaborated among multiple Internet of Things home appliances (400), including desktops, TVs, and robots in the home.

A smart refrigerator system according to an embodiment and a method for providing food information of a smart refrigerator sense a container for storing food, a food type, and a storage start time, and provides detailed food information including the type, location, expiration date, and manufacturing date of food stored in a smart refrigerator, as well as the storage period for each food and the location of the food.

In addition, the smart refrigerator system according to the embodiment and the food information providing method of the smart refrigerator recognize medicines, ointments, cosmetics, and various processed foods from around the world stored in the refrigerator, read letters and pictures written on the packaging of the processed foods, and provide detailed information including the expiration date to the user through voice and display.

In addition, the smart refrigerator according to the embodiment can receive the type of food and the date of manufacture via voice or text input, and can track the storage location of food inside the refrigerator and the food movement path within the house by linking with various IoT devices in the home.

In addition, the smart refrigerator according to the embodiment can provide food detail information to the user terminal by linking with the user terminal and control the refrigerator temperature through the user terminal.

FIG. 2 is a diagram showing a data processing configuration of a smart refrigerator (100) according to an embodiment.

Referring to FIG. 2, a smart refrigerator (100) according to an embodiment may be configured to include a sensor (110), a communication unit (120), an extraction unit (130), an output unit (140), a display unit (150), a text input unit (160), a location tracking unit (170), and a personal information protection unit (180). The term 'unit' used in this specification should be interpreted as including software, hardware, or a combination thereof, depending on the context in which the term is used. For example, the software may be machine language, firmware, embedded code, and application software. As another example, the hardware may be a circuit, a processor, a computer, an integrated circuit, an integrated circuit core, a sensor, a MEMS (Micro-Electro-Mechanical System), a passive device, or a combination thereof.

The sensor (110) senses users and food around the refrigerator. In an embodiment, the sensor (110) may be configured as a camera, etc. In an embodiment, when the sensor (110) is configured as a camera, an abnormal signal of the camera may be transmitted to the server (200) through a wireless communication network system from a communication unit inside the camera to facilitate maintenance of the camera. In addition, in an embodiment, the camera may be supplied with power through wireless power transmission technology from the main power supply device of the smart refrigerator so that it may operate without a battery.

In an embodiment, the sensor (110) senses text and pictures of food and food packaging containers. In an embodiment, the sensor (110) may be composed of internal sensors and external sensors. In an embodiment, the internal sensor senses the location of food containers and each food stored inside the refrigerator. The external sensor senses the user's motion, voice, food images, and food shapes.

In the embodiment, an external sensor is provided as a voice recognition sensor, so that detailed information about food can be asked to the user without operating a camera, and detailed information about the food can be determined through voice recognition of the user's response.

In addition, in the embodiment, the sensor (110) may be composed of a plurality of cameras, and the number of cameras equipped in the smart refrigerator may be set differently depending on the size of the smart refrigerator (100) and the installation site, so as to enable multiple people to use the smart refrigerator (100) simultaneously.

In an embodiment, when configuring multiple cameras with sensors (110), each camera tracks a different target and can communicate with each other to prevent the tracking targets of each camera from overlapping. At this time, the camera generates a duplicate estimation prevention signal and transmits and receives it to another camera. The duplicate estimation prevention signal is a signal for preventing the tracking targets of each camera from overlapping and can include information on the target tracked by each camera, the field of view of each camera, and the shooting boundary information of each camera.

These cameras can be used for monitoring other devices for malfunctions, relaying signals, and other tasks.

In an embodiment, the internal sensor can recognize the location of the container by identifying a tag configured inside the container. Specifically, in the embodiment, a tag is inserted into a container that is not used in a microwave oven, such as a kimchi container, and the inserted tag is recognized by the internal sensor to track the location of the container storing the food. In addition, in the embodiment, the external sensor senses the user's voice to input detailed food information, such as the type of food and expiration date, through the voice, and stores the same.

In addition, in an embodiment, when the external sensor is a camera and the user blocks the view of the camera in the refrigerator, the user's shape and user motion are sensed and the accurate location of the food is estimated by linking with the internal sensor. For example, even when the view is blocked, the internal sensor tracks the internal movement location of the food in real time and transmits the accurate internal location of the food to the external sensor, which is the camera. Referring to FIG. 3 showing an embodiment of the internal sensor configuration, a plurality of internal sensors (1, 2, 3, 4, 5, 6) according to the embodiment are included, and each internal sensor can be composed of a movable camera (1), an angle adjustment sensor (2), and a camera (3, 4, 5).

In the embodiment, the angle adjustment sensor (2) includes an infrared sensor, etc., to enable tracking of the approximate shape or real-time location of food. In the embodiment, the location of food is tracked even when the refrigerator is closed through an internal sensor, and in conjunction with the photo data of an external sensor, a camera, it is possible to provide image data on the location of the food.

The mobile camera (1) according to the embodiment can be angle-adjusted, and the lens moisture can be removed in the parking area (6). In the embodiment, the mobile camera (1) is used for food shape recognition and food location tracking of the inner door and inner shelves (7, 8, 9). The parking area (6) is a device for removing moisture or foreign substances from the camera lens, which is an internal sensor.

In the embodiment, a movable camera (1) and an angle adjustment sensor (2) track the shape and location of food inside the refrigerator and inside the open door from the outside. In the embodiment, external cameras, which are external sensors, can be used to track food movement and location outside the refrigerator.

The communication unit (120) receives a food judgment model from the server (200) and connects with the user terminal (300) and the IoT home appliance (400). The communication unit (120) transmits food detailed information to the user terminal (300) and receives a food image from the IoT home appliance (400), thereby enabling the location of food outside the refrigerator to be tracked.

The extraction unit (130) determines the type and amount of food, expiration date, and manufacturing date. In the embodiment, the extraction unit (130) determines the type and amount of stored food sensed by the sensor (110) through the food judgment model received from the server (200), and extracts the expiration date and manufacturing date from the text and picture written on the food packaging.

In addition, the extraction unit (130) can extract information about the food contained in the container according to the shape and pattern of the container containing the food. For example, when it is recognized that the food stored in the container above a certain size is kimchi, the food contained in the container that is similar to the size and appearance pattern of the container containing the kimchi to a certain level or more can be extracted as kimchi. In addition, the extraction unit (130) can identify the type of food contained in the transparent container through image recognition and a food judgment model.

The calculation unit (140) calculates the storage period for each food according to the initial storage time of the food stored in the refrigerator. In addition, the calculation unit (140) calculates the food edible period. In the embodiment, the food edible period is the period remaining from the current time to the expiration date, which is the period during which the food can be consumed before it spoils. If the expiration date of the food is set, the calculation unit (140) calculates the period remaining from the current time to the expiration date as the food edible period. If the expiration date of the food is not set, the calculation unit (140) calculates the period determined according to the type of the food as the food edible period.

In the embodiment, the calculation unit (140) can calculate the food consumption period in approximate time units for food with no set expiration date. For example, a time unit such as one week, two weeks, or one month can be calculated as the food consumption period.

In addition, in the embodiment, the output unit (140) predicts the food to be retrieved by the user based on the user's food retrieval pattern information, and identifies the predicted food location. In the embodiment, the user's food retrieval pattern information is information on the type of food that the user takes out of the refrigerator based on time. Specifically, if the user takes out milk and eggs at around 7:00 AM and takes out meat at around 6:00 PM based on the user's regular schedule, the time of retrieval of the food and the type of food can be stored as the food retrieval pattern information.

In the embodiment, the calculation unit (140) stores the user's food retrieval pattern information for a certain period of time, and identifies food information that the user repeatedly retrieves according to a time point, and when the user approaches the refrigerator at a certain time point, predicts the type of food that the user intends to retrieve, and calculates detailed food information including the location of the predicted food, the storage period, the expiration date, and the amount of food remaining. Thereafter, the calculation unit (140) can provide detailed information on the corresponding food to the user.

The display unit (150) displays information generated by the smart refrigerator (100). For example, the display unit (150) may output detailed information on food stored in the refrigerator, the food storage period, location information on food to be retrieved, detailed information on food to be retrieved, the food edible period for each food, and the location of the food outside the refrigerator. In an embodiment, the detailed information on food may include the type and amount of food, expiration date, manufacturing date, and the like. In addition, in an embodiment, the display unit (150) may periodically output the food edible period and the expiration date of the food.

In addition, the display unit (150) enlarges the image of the food and container captured by the sensor (110) and separates the food and container objects from the image so that the user can easily recognize the separated objects. In addition, a virtual screen is displayed on the refrigerator display so that individual clicks on the objects corresponding to the separated food and container are easy. Through this, the user can accurately recognize the food details and the container shape on the external display panel.

In addition, the display unit (150) can be configured as a full-length touch-type mirror and can provide user-customized information such as how to wear clothes, how to put on makeup, how to cook food, and how to decorate a table.

In addition, the display unit (150) can use the image reflected in the full-length mirror on the opposite side of the smart refrigerator (100) to track the shape and location of food on the refrigerator shelf or inside the door by reversing the image through the external camera provided on the top of the refrigerator. In addition, the user can be 3D scanned and linked with an artificial intelligence program to output the user's full body on the refrigerator display, and provide 3D scanned images of the user wearing various outfits.

In an embodiment, a full-length mirror opposite a smart refrigerator (100) serves as an auxiliary role for the internal camera. For example, a camera on the top of a smart refrigerator (100) can be used to capture images of food shapes and track their location on refrigerator shelves or inside a wide-open door as reflected in the full-length mirror.

In addition, the display unit (150) can combine overlapping foods or food containers into one in the display list so that only the type and number of foods can be output.

The text input unit (160) receives detailed food information as text from the user. In the embodiment, the text input unit (160) outputs a text input button on the display panel of the smart refrigerator (100) according to the user's settings, so that detailed food information such as the type of food and expiration date can be input in text form from the user.

The location tracking unit (170) tracks the location of food outside the refrigerator by linking with the IoT home appliance (400) in the home. For example, the location tracking unit (170) can receive an image of food taken from an IoT home appliance (400) in the home where a camera is installed, and track the current location of the food. In addition, the location tracking unit (170) receives real-time captured image information from the IoT home appliance (400) and determines the location of the food.

In the embodiment, the location tracking unit (170) can recognize food from a food image through an image recognition process using artificial intelligence machine learning, and track the location of food outside the refrigerator. Artificial intelligence image recognition is a process in which a machine recognizes objects from a photograph and understands a scene just like a person, and corresponds to one of the computer vision technologies. In the embodiment, the location tracking unit (170) performs a data processing process for classifying and detecting food objects included in the image and segmenting the objects by identifying them on a pixel basis for image recognition. In the embodiment, the location tracking unit (170) can perform an out-of-distribution detection process for processing unlearned patterns other than noise response. Out-of-distribution detection identifies whether an image input to the artificial intelligence is learned probability distribution data or not. In the embodiment, out-of-distribution detection filters out or processes exceptions for images that are difficult for an artificial neural network to judge, thereby increasing stability and reliability. In the embodiment, the probability value indicating how confident the deep learning judgment is for detecting non-learning distribution data is calibrated, or non-learning distribution data is generated and learned using a generative adversarial network (GAN) to improve the detection accuracy of food.

In addition, in the embodiment, the location tracking unit (170) uses a lightweight deep learning technique that simplifies the calculation to reduce the size of the model while maintaining the accuracy of recognizing the food image received from the Internet of Things home appliance (400), so that the location of the food outside the refrigerator can be finally determined. In the embodiment, the convolution filter in the convolutional neural network (CNN) is modified for image recognition to reduce the calculation dimension, or the pruning process that deletes the weight of the neural network that does not have a large effect, and the quantization process that reduces the floating point of the weight value to simplify the calculation are performed, thereby enabling data lightweighting. In addition, in the embodiment, the output of a large neural network that has been trained in advance is imitated by a small neural network to simplify the calculation and maintain accuracy.

In addition, in the embodiment, the location tracking unit (170) can calculate the food location based on the real-time location information of the Internet of Things home appliance (400) that captured the food image, and track the food location outside the refrigerator. In addition, in the embodiment, the location tracking unit (170) can receive the food location from the Internet of Things home appliance (400) that captured the food image, and collect the received food location to track the food location outside the refrigerator.

In addition, in the embodiment, the location tracking unit (170) is linked with the Internet of Things home appliance (400) to track the location of food, and if the location tracking of food outside is missed, the location tracking unit transmits a request for transmitting the location of the food to the user terminal (300). Afterwards, when the location tracking unit (170) receives the location of the food from the user terminal (300), it can confirm and update the location of the food, and transmit a message to the user terminal (300) recommending that the food be brought into the refrigerator.

In addition, in the embodiment, the location tracking unit (170) can adjust the location tracking intensity of food outside the refrigerator. In the embodiment, the location tracking unit (170) can adjust the location tracking intensity of food outside the refrigerator according to the location tracking intensity setting such as high or low. For example, the location tracking unit (170) can adjust the number of Internet of Things home appliances linked to the smart refrigerator (100) according to the location tracking intensity. Specifically, when the location tracking intensity is high, the location tracking unit (170) can set the number of Internet of Things home appliances linked to the smart refrigerator (100) to a certain number or more (e.g., 5 or more), and when the location tracking intensity is low, the number of Internet of Things home appliances linked to the smart refrigerator (100) can be set to a certain number or less. That is, in the embodiment, when the tracking intensity is high, the location tracking unit (170) increases the number of Internet of Things home appliances linked to the smart refrigerator (100), so that more data such as food images can be collected for obtaining food location information outside the refrigerator.

Additionally, the location tracking unit (170) can set the number of linked IoT home appliances to 0 when the location tracking function is set to off.

In addition, the location tracking unit (170) can set the tracking intensity according to the type of food. For example, in the case of food that does not need to be put back into the refrigerator after being taken out of the refrigerator, such as a beverage can, the location tracking function of the food can be turned off. On the other hand, in the case of food that needs to be put back into the refrigerator, such as large-capacity milk or large-capacity food, the tracking intensity can be set strong, and if the location tracking of the food outside is missed, a location transmission request for the food can be transmitted to the user terminal (300).

The personal information protection unit (180) limits the location range of the Internet of Things home appliance (400) that is linked to the smart refrigerator (100), so that food location and image information can be transmitted and received only in a limited space in the house. To this end, the personal information protection unit (180) can limit the range in which the location of the Internet of Things home appliance can be included, or can preset the type of the Internet of Things home appliance (400) that is linked to the smart refrigerator (100). In addition, the personal information protection unit (180) sets the tracking range of the food location to only indoors, so that food location and image information can be transmitted and received only in a limited space in the house. In addition, in the embodiment, the personal information protection unit (180) allows the food location to be transmitted and received through the user terminal (300) when the food location is not recognized inside the house or the food location is outside the house.

In addition, in the embodiment, the personal information protection unit (180) sets the format of the food location information as coordinates and text when storing the food location, thereby preventing personal information leakage indoors.

Figure 4 is a diagram showing a data processing configuration of a server (200) according to an embodiment.

Referring to FIG. 4, a server (200) according to an embodiment may be configured to include a deep learning unit (210) and a communication unit (220).

The deep learning unit (210) trains a neural network with a training data set to implement a food judgment model. In the embodiment, the food judgment model is an artificial intelligence model that identifies the type and amount of food through the image, shape, packaging, etc. of the food, and recognizes detailed food information such as the manufacturing date and expiration date.

In the embodiment, the neural network includes at least one of a DNN (Deep Neural Network), a CNN (Convolutional Neural Network), an RNN (Recurrent Neural Network), and a BRDNN (Bidirectional Recurrent Deep Neural Network), but is not limited thereto. In the embodiment, the deep learning unit (210) can learn food information including food images, expiration dates by food type, storage methods, etc. as a training data set, and implement a food judgment model.

The communication unit (220) distributes a food judgment model implemented as a smart refrigerator, communicates with an external server, and updates the food judgment model. In particular, the communication unit (220) updates the world language, new food brands, and expiration date notation methods in the food judgment model. In addition, in the embodiment, the communication unit (220) can collect abnormal signals from the smart refrigerator, and automatically request an inspection of the smart refrigerator to the AS center based on the collected abnormal signals. In the embodiment, the abnormal signals can include abnormal signals for a series of data processing processes occurring in the smart refrigerator, such as temperature maintenance, food, and user detection abnormalities.

Figure 5 is a diagram showing a data processing configuration of a smart terminal according to an embodiment.

Referring to FIG. 5, a smart terminal (300) according to an embodiment may be configured to include an information providing unit (310) and a recipe extraction unit (320).

The information providing unit (310) outputs food detailed information generated in the smart refrigerator, food storage period, food ingestion period, food detailed information according to the user's food return pattern, and food location. In an embodiment, the information providing unit (310) may sort and output food detailed information according to the storage period, expiration date, food ingestion period, and food quantity of the food. For example, food with a long food storage period may be output in order, or food with a short expiration date may be output first. Specifically, food may be sorted and output in order of the shortest food ingestion period, which is the time between the expiration date and the current point in time. In addition, the information providing unit (310) may output food detailed information according to the food quantity and expiration date. For example, food may be sorted and output in order of the shortest food ingestion period and the largest amount. In addition, the information providing unit (310) may periodically output the food ingestion period and the expiration date of the food according to the time unit.

The recipe extraction unit (320) provides recipe information for dishes that can be made with food stored in a smart refrigerator. For example, if the food stored in the refrigerator is kimchi, eggs, and butter, the recipe extraction unit (320) can provide recipe information for dishes such as kimchi fried rice that can be cooked with the corresponding foods.

In addition, the recipe extraction unit (320) can provide shopping information. In an embodiment, the recipe extraction unit (320) can provide recipe information for a dish that can be made if there is more specific food from the food stored in the refrigerator, and can provide shopping information, which is a list of foods that need to be purchased for the dish. In addition, the recipe extraction unit (320) can provide shopping information for the food that is not much left when the amount of food that is regularly consumed is not much. For example, when the amount of milk or eggs is determined to be less than a certain level, the recipe extraction unit (320) provides milk and eggs as shopping information.

In addition, the recipe extraction unit (320) can extract foods whose storage period exceeds a certain level when the number of foods stored in the refrigerator is a certain number or more, and provide cooking recipe information using the extracted foods. For example, the recipe extraction unit (320) can extract foods whose storage period exceeds a certain period (e.g., one month), such as pork belly and meat, and provide cooking recipe information for using up the extracted foods. Specifically, the recipe extraction unit (320) can provide recipe information for kimchi stew, stir-fried meat, etc. for using up meat whose storage period exceeds one month.

Below, the information providing method of the smart refrigerator will be described in order. Since the operation (function) of the information providing method of the smart refrigerator according to the embodiment is essentially the same as the function of the smart refrigerator system, any description overlapping with FIGS. 1 to 4 will be omitted.

Fig. 6 is a signal flow diagram of a smart refrigerator system according to an embodiment.

Referring to FIG. 6, in step S100, a deep learning neural network is trained on a server using a training data set to implement a food judgment model, and the food judgment model is distributed to a smart refrigerator. In step S200, the smart refrigerator receives the food judgment model from the server, and in step S300, the food detailed information including the type, amount, expiration date, and manufacturing date of the food is recognized through the food judgment model. In an embodiment, step S300 may include a step of sensing food, text and pictures of food packaging containers, and containers containing food by a sensor; a step of receiving the food judgment model from the server by a communication unit and linking it with a user terminal and an Internet of Things home appliance; and a step of identifying the type and amount of food sensed by the sensor through the food judgment model by an extraction unit, and extracting the expiration date and manufacturing date from the text and pictures of the food packaging container.

In step S400, the user terminal receives food detailed information from the smart refrigerator and outputs the food detailed information. In step S500, the Internet of Things home appliance generates an image of food outside the refrigerator and transmits it to the smart refrigerator. In step S600, the food image is collected from the Internet of Things home appliance, and in step S700, the smart refrigerator tracks the location of food inside or outside the refrigerator. In an embodiment, step S700 recognizes food in the food image received from the Internet of Things electronic device through an image recognition process using artificial intelligence machine learning, and tracks the location of food outside the refrigerator. Thereafter, in step S800, the smart refrigerator displays food detailed information and food location. In step S900, the user terminal receives the food location from the smart refrigerator and outputs the food location.

The smart refrigerator system and the method for providing food information in the smart refrigerator described above enable users to know detailed information such as the type, location, and expiration date of food stored in the refrigerator, thereby reducing the frequency of repeat purchases or disposal of food.

In addition, the utilization of food stored in the refrigerator is increased through examples, and the location of food inside and outside the refrigerator is tracked to store food more freshly and efficiently.

The disclosed content is merely an example, and various modifications and implementations can be made by a person skilled in the art without departing from the gist of the claims claimed in the patent, so the scope of protection of the disclosed content is not limited to the specific embodiments described above.

Claims (7)

In the smart refrigerator system,
A server that implements a food judgment model by training a deep learning neural network with a training data set and distributes the food judgment model;
A smart refrigerator that recognizes food detailed information including the type, amount, expiration date, and manufacturing date of food through a food judgment model received from the above server, tracks the location of food inside or outside the refrigerator, and displays the food detailed information and food location; and
A smart refrigerator system including a user terminal that outputs food details and food locations in conjunction with the smart refrigerator.
In the first paragraph, the smart refrigerator system;
A smart refrigerator system further comprising an Internet of Things home appliance that links with a smart refrigerator and transmits food images to the smart refrigerator to track the location of food outside the refrigerator.
In the first paragraph, the smart refrigerator;
Food, text and pictures on said food packaging container, a sensor for sensing the container containing the food;
A communication unit that receives a food judgment model from a server and connects with a user terminal and an Internet of Things home appliance;
An extraction unit that identifies the type and amount of food sensed from the sensor through the food judgment model and extracts the expiration date and manufacturing date from the text and picture of the food packaging container;
A smart refrigerator system characterized by including a display unit that displays food details including the type and amount of the food, expiration date, manufacturing date, food storage period, and food location.
In the third paragraph, the sensor;
Internal sensors that recognize the containers and locations of food stored inside the refrigerator; and
A smart refrigerator system characterized by including an external sensor that recognizes a user's motion and voice.
In the second paragraph, the smart refrigerator;
A text input section for entering food details including the type of food, expiration date, and manufacturing date as text; and
A smart refrigerator system further comprising a location tracking unit that tracks the location of food outside the refrigerator by linking with an Internet of Things device in the home.
In the fifth paragraph, the location tracking unit
A smart refrigerator system characterized by recognizing food in food images received from Internet of Things electronic devices through an image recognition process using artificial intelligence machine learning, and tracking the location of food outside the refrigerator.
In the method of providing information on a smart refrigerator,
(A) A step of implementing a food judgment model by training a deep learning neural network with a training data set on a server and distributing the food judgment model;
(B) A step of recognizing food detailed information including the type, amount, expiration date, and manufacturing date of food through a food judgment model received from the server in a smart refrigerator;
(C) a step of tracking the location of food inside or outside the refrigerator in a smart refrigerator and displaying the food details and the food location; and
(D) A method for providing information of a smart refrigerator, comprising: a step of linking a smart refrigerator with a user terminal to output food details and food locations;