CN116680424A

CN116680424A - Image management system, image management server, image management method, and method for producing image management system

Info

Publication number: CN116680424A
Application number: CN202211080206.0A
Authority: CN
Inventors: 草野慎太郎; 长谷川亮; 船山敦子; 青山胜
Original assignee: Hitachi Global Life Solutions Inc
Current assignee: Hitachi Global Life Solutions Inc
Priority date: 2022-02-22
Filing date: 2022-09-05
Publication date: 2023-09-01
Also published as: JP2023122170A; WO2023162312A1

Abstract

The invention provides an image management system, an image management server, an image management method and a method for producing the image management system. When viewing an image obtained by photographing the inside of a refrigerator of a customer, the customer and the living environment of the customer are hidden. The image management system is provided with: a refrigerator (1) provided with a refrigerating chamber (116) for accommodating food and a camera unit (10) for shooting the range of articles comprising the refrigerating chamber (116); a server (2 a) that acquires a pre-processing image captured by the camera unit (10); a storage server (6 a) that stores a pre-processing image; a server (2 b) that performs masking processing or mosaic processing on the pre-processing image to generate a post-processing image; a storage server (6 b) that stores the processed image; and a deletion unit that deletes the pre-processing image at least before deleting the post-processing image.

Description

Image management system, image management server, image management method, and method for producing image management system

Technical Field

The present invention relates to an image management system, an image management server, an image management method, and a method of producing an image management system.

Background

In recent years, a service is provided in which a user can view the inside of a refrigerator through a portable terminal by uploading an image obtained by capturing the inside of the refrigerator to a server and performing image processing on the uploaded image. Patent document 1 describes an invention capable of capturing a large number of places with a small number of cameras.

Regarding an image obtained by photographing the inside of a refrigerator of a customer, attention must be paid to processing from the viewpoint of protecting personal information. This is because the customer or the living environment of the customer may be inadvertently seen by others.

Patent document 1: japanese patent laid-open No. 2003-42626

Disclosure of Invention

In order to solve the above problems, an image management system of the present invention includes: a storage box having a storage area for storing articles and a photographing unit for photographing an article range including the storage area; a pre-processing image acquisition unit that acquires a pre-processing image captured by the imaging unit; a first storage unit that stores the pre-processing image; a post-processing image generation unit that performs processing to delete at least a part of the range outside the range of the article or reduce the amount of information on the pre-processing image to generate a post-processing image; a second storage unit that stores the processed image; and a deletion unit that deletes the pre-processing image after the post-processing image is generated.

The image management server of the present invention includes: a pre-processing image acquisition unit that acquires a pre-processing image obtained by capturing, by a capturing unit, an article range including a storage area in which articles are stored; a first storage unit that stores the pre-processing image; a post-processing image generation unit that performs processing to delete at least a part of the range outside the range of the article or reduce the amount of information on the pre-processing image to generate a post-processing image; a second storage unit that stores the processed image; and a deletion unit that deletes the pre-processing image at least before deleting the post-processing image.

In the method for producing an image management system according to the present invention, the image management system includes: a storage box having a storage area for storing articles and a photographing unit for photographing an article range including the storage area; a pre-processing image acquisition unit that acquires a pre-processing image captured by the imaging unit; a first storage unit that stores the pre-processing image; a post-processing image generation unit that performs processing to delete at least a part of the range outside the range of the article or reduce the amount of information on the pre-processing image to generate a post-processing image; a second storage unit that stores the processed image; and a deleting unit configured to delete the pre-processing image after the post-processing image is generated, wherein the method for producing the image management system provides a control program for controlling the photographing unit to the storage or a terminal used by a user.

Other means are described in the embodiments for carrying out the invention.

Drawings

Fig. 1 is a block diagram of a refrigerator of the present embodiment.

Fig. 2 is a block diagram of a server of the 2 types.

Fig. 3 is a block diagram of a portable terminal.

Fig. 4 is a logical block diagram of an image management system.

Fig. 5 is a front view of the refrigerator.

Fig. 6 is a side view of the refrigerator.

Fig. 7 is a front view of a state in which left and right doors of a refrigerating chamber of a refrigerator are opened.

Fig. 8 is a top view of a state in which left and right doors of a refrigerating chamber of a refrigerator are opened.

Fig. 9 is a perspective view of the camera unit of the refrigerator as viewed obliquely from below.

Fig. 10 is a sequence chart of the first image processing process.

Fig. 11 shows a pre-process image taken by a camera of the refrigerator.

Fig. 12 shows a pre-process image after partial masking would not be required.

Fig. 13 shows an example of the processed image.

Fig. 14 shows a selection screen displayed on the mobile terminal.

Fig. 15 shows a warning screen.

Fig. 16 shows a registration continuation user selection screen.

Fig. 17 shows a confirmation dialog.

Fig. 18 shows a notice-at-transfer screen.

Fig. 19 shows a notice-at-transfer screen.

Fig. 20 shows an image use attention screen.

Fig. 21 shows a home screen.

Fig. 22 shows a content screen of the refrigerator.

Fig. 23 is a sequence chart of image processing according to the second embodiment.

Detailed Description

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings. In the image management system of the present embodiment, the refrigerator 1 of fig. 1, the servers 2a, 2b and the storage servers 6a, 6b of fig. 2 cooperate with the portable terminal 3 of fig. 3 to provide services to users. The refrigerator 1, the servers 2a, 2b, the storage servers 6a, 6b, and the portable terminal 3 are respectively assigned different IP addresses.

The structure of the refrigerator 1 will be described with reference to fig. 1.

The refrigerator 1 has, as a hardware configuration, a camera unit 10, an arithmetic unit 112, a ROM (Read Only Memory) 113, a RAM (Random Access Memory: random access Memory) 114, a communication unit 115, and a camera LED (Light Emitting Diode: light emitting diode) 100. The refrigerator 1 further includes a left door 12, a right door 13, a door sensor 117, and a refrigerating chamber 116 shown in fig. 1.

The arithmetic unit 112 is, for example, a CPU (Central Processing Unit central processing unit) and comprehensively controls the refrigerator 1. The RAM114 is a work area of the arithmetic unit 112. The RAM114 is a non-transitory or temporary recording medium storing various programs and data, and stores programs not shown.

The camera unit 10 is an imaging unit that images the refrigerator compartment 116, and is, for example, a fisheye camera or a panoramic camera. The camera unit 10 is installed at an upper side of the refrigerating chamber 116. The camera LED100 is an indicator that lights up when the camera unit 10 photographs. The left door 12 and the right door 13 are doors provided in the refrigerator compartment 116, and the open/close state is detected by the door sensor 117. The camera unit 10 operates to photograph the refrigerating compartment 116 by opening the left door 12 or/and the right door 13 by the user. The refrigerating compartment 116 is a storage compartment for storing articles typified by food.

The communication unit 115 is connected to a network, and transmits and receives data to and from the server 2a, which is an external device.

The configuration of the servers 2a, 2b and the storage servers 6a, 6b will be described with reference to fig. 2. The servers 2a, 2b and the storage servers 6a, 6b each have a CPU21, a ROM22, a RAM23, a storage section 24, and a communication section 25.

The CPU21 comprehensively controls the servers 2a, 2b. ROM22 is a non-volatile memory, such as storing BIOS (Basic I/O system). The RAM23 is a work area of the CPU 21. The RAM23 is a non-transitory or temporary recording medium storing various programs and data, and stores programs not shown.

The storage unit 24 is a non-transitory or temporary recording medium that stores various programs and data, and is configured by, for example, a Hard Disk Drive (HDD), a flash memory, or the like. The storage unit 24 of the storage servers 6a and 6b is preferably a large-capacity storage unit that can be accessed at high speed. The communication unit 25 is connected to a network and transmits and receives data to and from an external device.

The structure of the mobile terminal 3 will be described with reference to fig. 3. The portable terminal 3 includes a CPU31, a ROM32, a RAM33, a storage unit 34, a communication unit 35, and a touch panel display 36.

The CPU31 performs integrated control of the portable terminal 3. ROM32 is a non-volatile memory, such as storing BIOS. The RAM33 becomes a work area of the CPU 31. The RAM33 is a non-transitory or temporary recording medium storing various programs and data, and stores programs not shown.

The storage unit 34 is a non-transitory or temporary recording medium that stores various programs and data, and is constituted by, for example, a flash memory or the like. The communication unit 35 is connected to a network and transmits and receives data to and from an external device. The touch panel display 36 is configured by stacking a touch panel on a planar display. The touch screen display 36 is a display unit that displays text, images, or graphics, and is also an input unit that detects a user operation.

Fig. 4 is a logical block diagram of an image management system.

The pre-processing image acquisition unit 81 acquires a pre-processing image captured by the camera unit 10. The pre-processing image acquisition unit 81 is realized by executing a program by the CPU21 of the server 2 a.

The first storage unit 82 stores the pre-processing image. The first storage unit 82 corresponds to the storage server 6a.

The post-processing image generation unit 84 performs processing to delete at least a part of the range out of the range of the article or reduce the information amount on the pre-processing image to generate a post-processing image. The post-processing image generation unit 84 deletes or reduces the information amount including a predetermined range for the pre-processing image to generate a post-processing image. When a human body part is captured in any one of the pre-processing images, the post-processing image generation unit 84 deletes the human body part or reduces the information amount, thereby generating a post-processing image. The post-processing image generation unit 84 is realized by executing a program by the CPU21 of the server 2b.

The deletion unit 83 deletes the pre-processing image after the post-processing image generation. Then, the deletion unit 83 deletes the pre-processing image at least before deleting the post-processing image. By execution of the program by the CPU21 of the server 2 a.

The second storage unit 85 stores the processed image. The second storage unit 85 corresponds to the storage server 6a.

The registration unit 86 registers a user who can view the processed image in association with the individual number of the storage or the imaging unit. The registered user information presenting unit 87 presents, when a user associated with an individual number is newly registered, information about the registered user to the user.

The registered user right deleting unit 88 causes the user to select a registered user when the user associated with the individual number is newly registered, and deletes the viewing right of the selected registered user. The registration unit 86, the registered user information presenting unit 87, and the registered user authority deleting unit 88 are realized by executing programs by the CPU31 of the portable terminal 3.

Fig. 5 is a front view of the refrigerator 1.

The refrigerator 1 is a device for cooling food and the like, and includes a left door 12, a right door 13, and a camera unit 10 of a refrigerating compartment 116. The refrigerator 1 has a plurality of storage compartments inside its cabinet. As the storage compartments of the refrigerator 1, a refrigerating compartment 116, an ice making compartment 14 and an upper freezer compartment 15, a vegetable compartment 16 and a lower freezer compartment 17 are provided in this order from top to bottom.

The refrigerator 1 has a case in which a heat insulating material (not shown) such as foamed polyurethane is filled between an outer case made of a steel plate and an inner case made of a resin (not shown). A plurality of openings corresponding to the respective chambers are provided on a front side (front side) of the cabinet of the refrigerator 1. For example, when food or the like is put in through the opening of the refrigerating chamber 116, the left door 12 and the right door 13 are opened. When the left door 12 and the right door 13 are closed, the opening of the refrigerator compartment 116 is closed. Thus, the left door 12 and the right door 13 have a function of closing the opening of the refrigerating chamber 116, and the like. The same applies to the other doors.

The refrigerator 1 is provided with a left door 12 and a right door 13 as french doors forming a refrigerating chamber 116 together with a cabinet. The left door 12 is rotatable about the axis of a hinge 121 (see fig. 8) at the left end. The same applies to the right door 13. In addition, the refrigerator 1 includes a vegetable compartment door and a lower-stage freezing compartment door as drawer-type doors in addition to an ice-making compartment door and an upper-stage freezing compartment door.

A plurality of partitions for partitioning the refrigerating chamber 116 in a predetermined manner are provided in the refrigerating chamber 116. A plurality of shelves 123 (see fig. 7) for storing food and the like are provided inside the refrigerator of the left door 12. The same applies to the left door 12. The ice making compartment 14 is provided with an ice making compartment container (not shown) that is pulled out integrally with the ice making compartment door. Similarly, an upper-stage freezing chamber container (not shown) is provided in the upper-stage freezing chamber 15, a vegetable chamber container (not shown) is provided in the vegetable chamber 16, and a lower-stage freezing chamber container (not shown) is provided in the lower-stage freezing chamber 17.

Although not shown, the refrigerator 1 includes a compressor, a radiator (condenser), a capillary tube (throttle mechanism), and an evaporator. The refrigerant is circulated through the compressor, the radiator, the capillary tube, and the evaporator in this order, and the air in the storage chamber is cooled by heat exchange with the refrigerant flowing through the evaporator. The camera unit 10 shown in fig. 5 photographs a storage room or the like of the refrigerator 1, and is provided on an upper surface of the cabinet.

Fig. 6 is a side view of the refrigerator 1.

As shown in fig. 6, the camera unit 10 includes a main body 101 and a support 102. The main body 101 has a function of photographing a storage room or the like. The support 102 supports the main body 101 and is provided on the upper surface of the refrigerator 1.

A lens 103 is provided near the front end of the body 101 (see also fig. 6). The lens 103 is an optical element that refracts and focuses light. As such a lens 103, for example, a fisheye lens is used. When the left door 12 and/or the right door 13 (see fig. 5) are/is opened, the lens 103 faces downward, so that the camera unit 10 can capture the refrigerator compartment 116 (see fig. 7). The camera unit 10 is installed at a position where it is possible to capture a living space range including living spaces other than the refrigerating compartment 116.

As shown in fig. 6, the lens 103 is located further forward than the front end of the case (opening of the case). More preferably, the lens 103 is located further forward than the front surfaces of the left door 12 and the right door 13 (refer to fig. 5) in the closed state. Thus, for example, when the left door 12 and the right door 13 of the refrigerating chamber 116 are opened, the refrigerating chamber 116 easily enters the field of view of the lens 103. In addition, the vegetable compartment 16 and the like can be photographed by the camera unit 10 even in a state where a drawer type door such as a vegetable compartment door is opened (refer to fig. 5). In this way, the lenses 103 are provided at positions where the respective storage chambers are viewed from above and below.

Fig. 7 is a front view of the refrigerator 1 in a state in which the left door 12 and the right door 13 are opened.

As described above, the inner panel 122 of the left door 12 is provided with the plurality of door frames 123 for accommodating food and the like (the same applies to the right door 13). When the left door 12 and the right door 13 are opened, the food or the like in the refrigerating chamber 116 and the door frames 123 and 133 is overlooked into the field of view of the lens 103 of the camera unit 10.

The rotary partition 124 shown in fig. 7 is a partition body for suppressing leakage of cool air from a gap between the left door 12 and the right door 13. In the example of fig. 7, a rotation spacer 124 is provided at an end portion of the left door 12 opposite to the axis of the hinge 121 (see fig. 8), and rotates in a predetermined manner in accordance with the opening and closing of the left door 12.

In addition, a photographing button 125 pressed by a user when the refrigerator compartment 116 is manually photographed using the camera unit 10 is provided at the left door 12, and a photographing button 135 is provided at the right door 13. In the example of fig. 7, an imaging button 125 (the same applies to the right door 13) is provided at a lower portion of the surface of the left door 12 opposite to the hinge 121 (see fig. 8) of the left door 12. In other terms, in a state where the left door 12 and the right door 13 are closed (see fig. 5), the imaging button 125 is provided on a surface of one of the left door 12 and the right door 13 facing the other (in the vicinity of the rotating spacer 124 in fig. 7).

The photographing performed when the user presses at least one of these photographing buttons 125, 135 is referred to as "manual photographing". The imaging performed by the arithmetic unit 112 based on the elapsed time from the timing when the left door 12 and the right door 13 start to open is referred to as "automatic imaging".

Fig. 8 is a plan view of the refrigerator 1 in a state in which the left door 12 and the right door 13 are opened.

In the example of fig. 8, the main body 101 of the camera unit 10 is disposed slightly to the left with respect to the center of the refrigerator 1 in the left-right direction. When described in more detail, the main body 101 of the camera unit 10 is disposed immediately above the joint between the left door 12 and the right door 13 in the closed state (see also fig. 5 and 6). By disposing the main body 101 in this manner, for example, when the left door 12 and the right door 13 are opened, food or the like of the door frames 123, 133 (see fig. 7) is easily taken into the field of view of the camera unit 10.

In addition, when the refrigerator 1 is used, the user is often aware of the seams (boundary portions) of the left door 12 and the right door 13. Therefore, by disposing the main body 101 directly above the joint between the left door 12 and the right door 13, it is possible to suppress the sense of incongruity of the user as compared with the case where the position of the main body 101 in the lateral direction is deviated from the joint.

Fig. 9 is a perspective view of the camera unit 10 from obliquely below.

The main body 101 of the camera unit 10 shown in fig. 9 includes a case 31b and a camera LED100 (Light Emitting Diode: illumination unit) in addition to the lens 103 (see also fig. 6). In the example of fig. 6, the housing of the main body 101 has a substantially rectangular parallelepiped shape elongated in the front-rear direction. A circular hole (not shown) is provided in the lower surface near the front end of the housing of the main body 101, and the lens 103 is exposed through the hole. Further, a camera LED100 is provided on the rear side (back side) of the lens 103.

The camera LED100 is a light source that irradiates light to the refrigerator compartment 116 and the like, so that the camera unit 10 can capture images of the refrigerator compartment 116 (see fig. 7) and the shelves 123 and 133 (see fig. 7) with appropriate brightness. Further, the camera LED100 (illumination section) is provided outside the cabinet (see fig. 6) of the refrigerator 1. As will be described later, in the present embodiment, the camera unit 10 captures an image of the refrigerating compartment 116 and the like while the in-box lamp of the refrigerator 1 is turned off and the camera LED100 is turned on (see fig. 7).

As shown in fig. 9, by providing the lens 103 on the front side of the camera LED100, reflected light from the refrigerator 116 and the like (see fig. 5) is easily incident on the lens 31a in a state where the left door 12 and the right door 13 are opened. Therefore, the food stored in the refrigerator 116 or the like can be appropriately photographed. In addition, the camera LED100 is preferably located further toward the front side than the front end (opening of the case) of the case (see fig. 6) of the refrigerator 1. Thus, light emitted from the camera LED100 is easily incident on the refrigerator 116 or the like, and a clear imaging result is easily obtained.

The refrigerator 1 is a storage box including a refrigerator compartment 116 as a storage area for storing articles, and a camera unit 10 for capturing images of an article range including the refrigerator compartment 116.

Fig. 10 is a sequence chart of the image processing of the first embodiment.

When the user opens the door of the refrigerator 1, the operation section 112 detects that the door is opened by the door sensor 117. Thus, the computing unit 112 photographs the refrigerator 116 in which the food is stored with the camera unit 10 (step S10). The arithmetic unit 112 uploads the captured camera image of the refrigerator compartment 116 to the server 2a (step S11). The camera image is an image before processing that enables a user to view the image.

The CPU21 of the server 2a functions as a pre-processing image acquisition unit that acquires a pre-processing image captured by the camera unit 10. The server 2a transmits the camera image before processing to the storage server 6a (step S12).

When the event is notified to the server 2b (step S13), the storage server 6a saves the camera image before processing (step S14). The storage server 6a functions as a first storage unit that stores the pre-processing image.

Then, when the storage server 6a transmits the camera image before processing to the server 2b (step S15), the storage server deletes the camera image before processing stored in the storage server itself (step S16). That is, the CPU21 of the storage server 6a functions as a deletion unit that deletes the camera image before processing.

The server 2b corrects the distortion of the camera image before processing due to the lens (step S17), and processes the camera image into an image for frontal observation by trapezoidal correction (step S18). Then, the server 2b requests the storage server 6b for the existing processed image (step S19), and receives the processed image from the storage server 6b (step S20).

When the existing processed image and the trapezoid corrected image are combined (step S21), the server 2b transmits the combined processed image to the storage server 6b and stores the combined processed image (step S22). The CPU21 of the server 2b functions as a post-processing image generating unit that generates a post-processing image by performing processing to delete at least a part of the range outside the range of the article or reduce the amount of information on the pre-processing image.

The storage server 6b holds the processed image received from the server 2b (step S23). The storage server 6b functions as a second storage unit that stores the processed image.

Then, the storage server 6b transmits the processed image to the portable terminal 3 in response to a request from the portable terminal 3 (step S24). Thus, the customer can view the image of the refrigerator compartment 116 through the mobile terminal 3.

The storage server 6a stores the pre-processing image, and the storage server 6b stores the post-processing image after the composition, whereby the respective images can be easily managed. The mobile terminal 3 can easily acquire the synthesized processed image simply by accessing the storage server 6 b.

The portable terminal 3 does not access the storage server 6a, and therefore does not know the IP address of the storage server 6a. Thus, since the access method from the mobile terminal 3 to the storage server 6a is not known, it is possible to suppress the customer and the living environment of the customer captured in the image before processing from being unintentionally seen by others.

The storage server 6b deletes the processed image when a predetermined period of time elapses after the processed image is stored. That is, the storage server 6a deletes the pre-processing image stored by itself before the storage server 6b deletes the post-processing image. This can suppress a customer or a living environment of the customer, which is captured in an image before processing, from being seen by others unintentionally.

Fig. 11 shows an image before processing taken by the camera unit 10 of the refrigerator 1. The refrigerating compartment 116 is photographed at the lower half of the image. The inside of the left door 12 is photographed at the left side of the image. The inside of the right door 13 is photographed at the right side of the image. Further, the living environment of the customer is photographed in the upper half of the image. This portion is an unnecessary portion when viewed from the processed image.

Fig. 12 shows an image before processing after the unnecessary portion is smeared.

The system performs masking processing on unwanted portions of the fisheye image. The masking process performed by the present system is a processing process of deleting at least a part of the range out of the range of the article. Thus, when viewing an image obtained by photographing the inside of the refrigerator of the customer, it is possible to suppress the customer and the living environment of the customer from being seen by others unintentionally.

The masking process of the unnecessary portion is performed by the server 2b, but may be performed by the server 2a or the refrigerator 1.

Fig. 13 shows an example of the processed image.

The server 2b corrects lens distortion of an image before processing captured by the camera unit 10 of the refrigerator 1, and performs trapezoidal correction to process the image into an image for frontal observation. At this time, when a face is detected in the image, the server 2b performs mosaic processing on the area in which the face is captured.

The mosaic processing executed by the system is processing for reducing the information amount of at least a partial range out of the range of the object. Thus, when viewing an image obtained by photographing the inside of the refrigerator of the customer, it is possible to suppress the customer from being seen by others unintentionally. The face detection process and the mosaic process are performed by the server 2b, but these processes may be performed on the image before processing by the server 2a or the refrigerator 1.

Fig. 14 to 20 show a series of screens for confirming attention to processing of camera images to a customer at the time of pairing the portable terminal 3 with the refrigerator 1. These screens are displayed by the CPU31 of the portable terminal 3 executing a refrigerator management program installed in the portable terminal 3.

Fig. 14 shows a new/second hand selection screen 41 displayed on the mobile terminal 3.

The selection screen 41 includes a new purchase button 411, a different button 412, and a return button 419.

The new purchase button 411 is a button for a new purchase user. When this button is clicked, the portable terminal 3 performs user registration after pairing with the refrigerator 1.

The other button 412 is a button facing a person who purchased the refrigerator 1 for a second hand. When this button is clicked, the mobile terminal 3 shifts to the warning screen 42 of fig. 15.

The back button 419 is a button for shifting to the screen immediately before shifting to the selection screen 41.

Fig. 15 shows a warning screen 42.

The warning screen 42 is a screen for warning the attention point when the refrigerator 1 is used.

The confirm button 421 is a button for ending the confirmation of the warning screen 42. When the confirm button 421 is clicked, the portable terminal 3 performs user registration after pairing with the refrigerator 1.

The back button 429 is a button for shifting to the screen immediately before shifting to the warning screen 42.

By the user registration, the identification number (individual number) of the refrigerator 1 is associated with the portable terminal 3. The identification number (individual number) of the refrigerator 1 is not limited, and the individual number of the camera unit 10 may be associated with the mobile terminal 3 to perform user registration.

Then, after the user registration, if there is an already registered user, the process shifts to the registration continuation user selection screen 43 of fig. 16. After the user registration, if there is no already registered user, the flow shifts to the transfer-time attention screen 45 of fig. 18.

Fig. 16 shows a registration continuation user selection screen 43.

The registration continuation user selection screen 43 is a screen for selecting a user who continues to register with the refrigerator 1. And check boxes 431 to 433 are check boxes for selecting users who continue registration.

The decision button 435 is a button for deciding on the selection after selecting the user to continue registration. By clicking the decision button 435, the portable terminal 3 shifts to the confirmation dialog 44 of fig. 17. The back button 439 is a button for shifting to a screen immediately preceding the shift to the registration continuation user selection screen 43.

Fig. 17 shows a confirmation dialog box 44.

The confirmation dialog 44 is a dialog for confirming registration cancellation of a user who does not continue registration.

The "yes" button 441 is a button for eliminating registration of a user that is not selected.

The no button 442 is a button that does not cancel registration of the unselected user. When the "yes" button 441 or the "no" button 442 is clicked, the transfer-time attention screen 45 of fig. 14 is shifted.

Fig. 18 shows a notice-at-transfer screen 45.

The notice at transfer screen 45 displays notice at transfer of the refrigerator 1. The confirm button 451 is a button indicating that the attention is confirmed.

The return button 459 is a button for shifting to a screen immediately preceding the shift to the attention screen 45 at the time of the shift.

Fig. 19 shows a camera use attention screen 46.

The camera use attention screen 46 is used to remind a customer to take a picture by the camera of the refrigerator 1, i.e., the camera unit 10. When the screen is slid and reaches the lower end, a confirmation button for shifting to the next image utilization attention screen 47 is displayed.

The back button 469 is a button for shifting to the previous screen shifted to the camera use attention screen 46.

Fig. 20 shows an image utilization attention screen 47.

The image utilization attention screen 47 is a screen for obtaining a customer's attention for utilization of an image captured by the camera unit 10, which is a camera of the refrigerator 1.

The consent button 471 is a button for the customer to consent to use the image. When the consent button 471 is clicked, a series of settings is ended and shifted to the home screen 50.

The back button 479 is a button for shifting to the previous screen shifted to the image utilization attention screen 47.

Fig. 21 shows a home screen 50.

The home screen 50 is a screen displayed after the refrigerator management program is started.

The refrigerator content inspection button 501 is a button for viewing the contents of the refrigerator 1 of the customer. When the content check button 501 of the refrigerator is clicked, the portable terminal 3 shifts to the content screen 51 of the refrigerator of fig. 17.

The usage manager button 502 is a button for explaining the usage of the refrigerator 1.

Fig. 22 shows a content screen 51 of the refrigerator.

The refrigerator content screen 51 is a screen for viewing a front view image of the refrigerator 1 in the refrigerator compartment 116.

The indoor selection button 512 is a button for selecting which of the refrigerating compartment 116, the inside of the left door 12, and the inside of the right door 13 is displayed.

The back button 519 is a button for shifting to the home screen 50 immediately before shifting to the content screen 51 of the refrigerator.

Second embodiment

In the second embodiment, the lens correction process, the trapezoidal correction process, and the like are executed not by the server 2b but by the operation unit 112 of the refrigerator 1. Since the camera image before processing can be not transmitted to the network, it is possible to suppress a customer and a living environment of the customer from being seen by others unintentionally.

When the user opens the door of the refrigerator 1, the operation section 112 detects that the door is opened by the door sensor 117. Thus, the computing unit 112 photographs the refrigerator 116 in which the food is stored with the camera unit 10 (step S30). The camera image is a pre-processing image that enables a user to view the image.

The arithmetic unit 112 saves the camera image before processing (step S31). The computing unit 112 corrects distortion of the camera image before processing due to the lens (step S32), and processes the camera image into an image for frontal observation by trapezoidal correction (step S33). Then, the arithmetic unit 112 deletes the camera image before processing (step S34). The arithmetic unit 112 requests the storage server 6b for the existing processed image (step S35), and receives the processed image from the storage server 6b (step S36).

When the existing processed image and the trapezoid corrected image are combined (step S37), the computing unit 112 transmits the combined processed image to the server 2b (step S38). The server 2b transmits the processed image to the storage server 6b to save (step S39). The storage server 6b holds the processed image received from the server 2b (step S40). Then, the storage server 6b transmits the processed image to the mobile terminal 3 in response to a request from the mobile terminal 3 (step S41). Thus, the customer can view the image of the refrigerator compartment 116 through the mobile terminal 3.

The storage server 6b deletes the processed image when a predetermined period of time has elapsed after the processed image is stored. That is, the storage server 6a deletes the pre-processing image stored by itself before the storage server 6b deletes the post-processing image. This can suppress the customer and the living environment of the customer, which are captured in the image before processing, from being seen by others unintentionally.

The present invention is not limited to the above-described embodiments, and includes various modifications. The above-described embodiments are described in detail for the purpose of easily explaining the present invention, and are not limited to the embodiments having all the configurations described. Some of the structures of one embodiment may be replaced with structures of other embodiments, and structures of other embodiments may be added to the structures of one embodiment. In addition, some of the structures of the embodiments may be added, deleted, or replaced with other structures.

For example, part or all of the above-described structures, functions, processing units, and the like may be realized by hardware such as an integrated circuit. The above-described structures, functions, and the like may also be implemented in software by interpreting and executing programs for realizing the respective functions by a processor. Information such as programs, tables, and files for realizing the respective functions can be placed in a memory, a hard disk, a recording device such as an SSD (Solid State Drive: solid state drive), or a recording medium such as a flash memory card or a DVD (Digital Versatile Disk: digital versatile disc).

In each embodiment, the control lines and the information lines are those required for explanation, and not necessarily all of them are shown in the product. In practice, it is also considered that almost all structures are connected to each other.

As modifications of the present invention, there are, for example, modifications as described in (a) to (d) below.

(a) The mosaic process for suppressing the customers and the living environments of the customers from being unintentionally seen by others may be performed by any one of the refrigerator 1, the server 2a, and the server 2b.

(b) The covering process for suppressing the customers and the living environments of the customers from being unintentionally seen by others may be performed by any one of the refrigerator 1, the server 2a, and the server 2b.

(c) The mosaic processing and the masking processing may be performed either or both.

(d) The present invention is not limited to the reading of the refrigerator compartment, and can be applied to the reading of the storage area of any storage.

Description of the reference numerals

1 refrigerator (storage)

100 camera LED

10 camera unit

112 arithmetic unit

113ROM

114RAM

115 communication unit

116 refrigerating chamber (storage area)

117 door sensor

12 left door

124 rotating partition

13 right door

14 ice making chamber

15 upper layer freezing chamber

16 vegetable room

17 lower layer freezing chamber

2a server (image acquisition unit before processing, deletion unit)

2b server (post-processing image generating unit)

21CPU

22ROM

23RAM

24 storage part

25 communication unit

3 Portable terminal

31CPU

32ROM

33RAM

34 storage part

35 communication unit

36 touch screen display

41 selection of a screen

411 new purchase button

412 other buttons

419 return button

42 warning screen

429 return button

43 registration continuation user selection screen

431 check box

432 check box

433 check box

435 determination button

439 Return button

44 confirmation dialog box

441 "yes" button

442 no button

45 transfer time reminding picture

451 confirm button

459 return button

46 camera usage attention screen

469 return button

47 image utilization attention screen

479 return button

50 homepage picture

501 content check button of refrigerator

502 methods of use housekeeping button

51 content Screen of refrigerator

512 indoor selection button

6a storage server (first storage unit)

6b storage server (second storage unit)

81-process-before-process image acquisition unit

82 first memory cell

83 deletion unit

84 post-processing image generation unit

85 second memory cell

86 registration unit

87 registered user information prompting unit

88 registered user rights deletion unit.

Claims

1. An image management system, comprising:

a storage box having a storage area for storing articles and a photographing unit for photographing an article range including the storage area;

a pre-processing image acquisition unit that acquires a pre-processing image captured by the imaging unit;

a first storage unit that stores the pre-processing image;

a post-processing image generation unit that performs processing to delete at least a part of the range outside the range of the article or reduce the amount of information on the pre-processing image to generate a post-processing image;

a second storage unit that stores the processed image; and

and a deletion unit that deletes the pre-processing image after the post-processing image is generated.

2. The image management system of claim 1, wherein,

the photographing part is installed at a position where it is possible to photograph a living space range including a living space outside the storage box,

the deletion unit deletes the pre-processing image at least before deleting the post-processing image.

3. The image management system of claim 1, wherein,

the post-processing image generation means deletes the pre-processing image including a predetermined range or reduces the information amount to generate a post-processing image.

4. The image management system of claim 1, wherein,

the post-processing image generation means deletes or reduces the amount of information including a human body part when the human body part is captured in any one of the pre-processing images, thereby generating a post-processing image.

5. The image management system of claim 1, wherein,

the image management system further includes:

a registration unit that registers a user who can view the processed image in association with the individual number of the storage or the imaging unit; and

and a registered user information presenting unit that presents information related to the registered user to the user when the user associated with the individual number is newly registered.

6. The image management system of claim 5, wherein,

the image management system further includes a registered user authority deletion unit that, when a user associated with the individual number is newly registered, causes the user to select a registered user to delete the viewing authority of the selected registered user.

7. The image management system of claim 1, wherein,

the IP address assigned by the first storage unit is different from the IP address assigned by the second storage unit.

8. An image management server is characterized by comprising:

a pre-processing image acquisition unit that acquires a pre-processing image obtained by capturing, by a capturing unit, an article range including a storage area in which articles are stored;

a first storage unit that stores the pre-processing image;

a second storage unit that stores the processed image; and

and a deletion unit that deletes the pre-processing image at least before deleting the post-processing image.

9. An image management method, characterized in that,

the 1 or more arithmetic units execute the following steps:

a step of acquiring a pre-processing image obtained by photographing an article range including an article storage area of a storage box by a photographing unit;

a step of storing the pre-processing image;

a step of performing processing to delete at least a part of the range outside the range of the article or reduce the information amount on the pre-processing image to generate a post-processing image;

a step of storing the processed image; and

and deleting the pre-processing image at least before deleting the post-processing image.

10. A method of producing an image management system,

the image management system is provided with:

a first storage unit that stores the pre-processing image;

a second storage unit that stores the processed image; and

a deletion unit that deletes the pre-processing image after the post-processing image is generated,

it is characterized in that the method comprises the steps of,

the image management system is produced by providing a control program for controlling the photographing section to the storage or a terminal used by a user.