KR102505705B1 - Image analysis server, object counting method using the same and object counting system - Google Patents

Abstract

The present invention relates to an image analysis server, an object counting method and an object counting system using the image analysis server.
According to one aspect of the present invention, the step of receiving an image including one or more objects by the user terminal; forming a plurality of boxes for each of the objects by an image analysis server, and deleting the remaining boxes leaving only the boxes corresponding to the object among the plurality of boxes; and counting the number of remaining boxes by the image analysis server and transmitting the number corresponding to the box to the user terminal.

Description

Image analysis server, object counting method using the image analysis server, and object counting system {Image analysis server, object counting method using the same and object counting system}

The present invention relates to an image analysis server, an object counting method and an object counting system using the image analysis server.

BACKGROUND As we enter an aging society, the demand for patients visiting hospitals is increasing, and accordingly, the types and number of drugs to be administered are also increasing.

However, when administering pills to patients or conducting inventory at small pharmacies or hospitals, it is inconvenient to manually count the number of pills. In addition, when a person manually counts pills, many mistakes occur in administering less than or more than a predetermined number of pills.

In order to solve this problem, large-scale pharmacies and hospitals introduce and use pill-counting devices, but these devices are expensive and thus difficult to purchase in small pharmacies and hospitals.

Patent Document: Korean Publication No. 10-2016-0057187 (published on May 23, 2016)

Embodiments of the present invention are proposed to solve the above problems, and an image analysis server and an image analysis server capable of simply counting the number of objects (eg, pills) without introducing complex and expensive equipment. It is intended to provide an object counting method and an object counting system using

In addition, it is intended to provide an image analysis server capable of accurately counting the number of closely attached objects (eg, pills), an object counting method using the image analysis server, and an object counting system.

According to one embodiment of the present invention, the step of receiving an image including one or more objects by the user terminal; forming a plurality of boxes for each of the objects by an image analysis server, and deleting the remaining boxes leaving only the boxes corresponding to the object among the plurality of boxes; and counting the number of remaining boxes by the image analysis server and transmitting the number corresponding to the box to the user terminal.

In addition, the step of forming a plurality of boxes for each of the objects by the image analysis server, leaving only the boxes corresponding to the number of boxes among the plurality of boxes and deleting the remaining boxes, the object recognition by the box setting module An object counting method using an image analysis server including forming a plurality of boxes for each of the objects by executing a deep learning model may be provided.

In addition, after the step of forming a plurality of boxes for each object, an object counting method using an image analysis server in which an algorithm for removing some of the plurality of boxes formed in each object by the first box removal module is executed. this can be provided.

In addition, after an algorithm for removing some of the plurality of boxes formed in each object is executed by the first box removal module, leaving only one box in one object and deleting the remaining boxes by the second box removal module. An object counting method using an image analysis server in which is executed may be provided.

In addition, the step of leaving only one box by one object by the second box removal module and setting the remaining boxes as a reference box among the remaining boxes by a reference box setting unit; setting an aggregation box that is a set of boxes overlapping the reference box by an aggregation box setting unit; setting a space remaining in the reference box as a comparison space by removing a space overlapping the aggregation box among spaces occupied by the reference box by a comparison space setting unit; When the ratio of the comparison space and the space occupied by the standard box is greater than the approximate coefficient by the box removal unit by the coarse coefficient comparison, the box set as the standard box is left, and the ratio of the space occupied by the comparison space and the standard box An object counting method using an image analysis server may be provided, including the step of removing a box set as the reference box when the approximate coefficient is smaller than the approximate coefficient.

In addition, an object counting method using an image analysis server in which the object recognition deep learning model executed by the box setting module is RetinaNet may be provided.

In addition, an object counting method using an image analysis server may be provided in which an algorithm for removing some of a plurality of boxes formed in each object by the first box removal module is non-maximum suppression (NMS).

In addition, the pill count is stored in the database according to the size and shape of the object, and the pill count determination module performs object counting using an image analysis server matching the pill count stored in the database according to the size and shape of the object shown in the image. A method may be provided.

According to another embodiment of the present invention, an image including one or more objects is received from a user terminal, a plurality of boxes are formed for each of the objects, and only a number of boxes corresponding to the objects are selected from among the plurality of boxes. An image analysis server may be provided that deletes remaining boxes, counts the number of remaining boxes, and transmits the number corresponding to the boxes to the user terminal.

In addition, the image analysis server may include a box setting module for executing an object recognition deep learning model to form a plurality of boxes for each of the objects; A first box removal module capable of executing an algorithm for removing some of a plurality of boxes formed in each object; and a second box removal module for leaving only one box for one object and deleting the other boxes.

In addition, the second box removal module, a reference box setting unit for setting an arbitrary box among the remaining boxes as a reference box; an aggregation box setting unit that sets an aggregation box that is a set of boxes overlapping the reference box; a comparison space setting unit configured to set a space remaining in the reference box as a comparison space by removing a space overlapping the aggregation box from among spaces occupied by the reference box; and when the ratio of the comparison space to the space occupied by the reference box is greater than the Orbit coefficient, leaving the box set as the reference box, and when the ratio of the comparison space to the space occupied by the reference box is less than the Orbit coefficient, the criterion An image analysis server including a box removal unit by pill coefficient comparison that removes a box set as a box may be provided.

In addition, the image analysis server may include a database storing pill coefficients according to the size and shape of the object; and a pill count determining module matching the pill count stored in the database according to the size and shape of the object appearing in the image.

According to another embodiment of the present invention, a user terminal capable of receiving an image including one or more objects; and the above-described image analysis server capable of forming a plurality of boxes for each of the objects, deleting the remaining boxes except for the boxes corresponding to the number of boxes among the plurality of boxes, and counting the remaining boxes. An object counting system may be provided.

An image analysis server according to embodiments of the present invention, an object counting method using the image analysis server, and an object counting system can simply count the number of objects (eg, pills) without introducing complicated and expensive equipment. It works.

In addition, there is an advantage in that the number of closely attached objects (eg, pills) can be accurately counted.

1 is a diagram schematically illustrating an object counting system according to an embodiment of the present invention.
Figure 2 is a diagram schematically showing the configuration of the image analysis server of Figure 1;
Figure 3 is a diagram schematically showing the sub-configuration of the second box removal module of the image analysis server of Figure 2;
4 is a flowchart schematically illustrating an object counting method using an image analysis server executed by the object counting system of FIG. 1;
FIG. 5 is a flowchart illustrating in detail step S2 among steps S1 to S3 of FIG. 4 .
FIG. 6 is a flowchart illustrating in more detail step S36 among steps S32 to S36 of FIG. 5 .
7 is a diagram conceptually illustrating receiving an object through the user terminal shown in FIG. 1 .
FIG. 8 conceptually shows that a plurality of boxes are formed for one pill by executing RetinaNet, an object recognition deep learning model, by the image analysis server of FIG. 1 .
FIG. 9 conceptually shows a box formed in one pill (object) by executing NMS (non maximum suppression), which is an algorithm for removing boxes by the image analysis server of FIG. 1 .
FIG. 10 is a conceptual diagram to aid understanding of step S36 shown in FIG. 6 .
11 is a diagram schematically illustrating a flowchart of transmitting a plurality of images to an image analysis server and counting one or more objects included in each of the plurality of images using the object counting system of FIG. 1;
12 is a diagram illustrating a screen displayed on the user terminal of FIG. 1 in case of a single analysis mode and a multi-analysis mode.
FIG. 13 is a diagram showing the number and type of objects included in each of a plurality of images analyzed by the image analysis server on the screen of the user terminal of FIG. 1 in case of multi-analysis mode.
FIG. 14 is a diagram schematically illustrating a multi-analysis auxiliary device on which the user terminal of FIG. 1 can be seated.
FIG. 15 is a diagram schematically illustrating the multi-analysis auxiliary device and moving belt of FIG. 14 for easily performing the multi-analysis mode.

1 is a diagram schematically illustrating an object counting system 1 according to an embodiment of the present invention.

Referring to FIG. 1 , an object counting system 1 may include an image analysis server 10 , a user terminal 20 and a manager terminal 30 .

Here, the image analysis server 10, the user terminal 20, and the manager terminal 30 are provided as independent devices and communicate data through the communication network 40, or the image analysis server 10 and the manager terminal 30 ) is composed of one physical device and may be provided for direct data communication.

In the present embodiment, the image analysis server 10, the user terminal 20, and the manager terminal 30 are provided as separate and independent devices as an example.

The object counting system 1 of this embodiment can be understood as a system capable of accurately counting the number of objects included in an image.

Specifically, when the user photographs an object through the user terminal 20, the image including the photographed object is transmitted to the image analysis server 10, and the image is converted into an image through a preset algorithm by the image analysis server 10. The number of displayed objects may be counted.

In this embodiment, the object is described as an example of a pill having a certain shape. If the object photographed by the user terminal 20 is a pill, the object counting system 1 of this embodiment may be understood as a system for counting the number of pills usable in pharmacies and hospitals.

However, the spirit of the present invention is not limited thereto, and the object may include any object having a certain shape.

The image analysis server 10 may be understood as a server that receives image data from the user terminal 20 and processes data necessary for counting the number of objects appearing in the image.

Objects included in one image may be objects of the same type having the same size and shape. That is, the image analysis server 10 may count the same object included in one image.

However, the spirit of the present invention is not limited thereto, and the objects included in one image may be different types of objects having different sizes and shapes. In this case, the image analysis server 10 is included in one image. You can also count different types of objects included.

The user terminal 20 may photograph an object placed on an object plate, which will be described later, and display it as an image.

In addition, the user terminal 20 is a device capable of communicating with the image analysis server 10, and may be a mobile terminal or a fixed terminal implemented as a computing device.

For example, the user terminal 20 may include a smart phone, a laptop computer, a tablet PC, a wearable device, a computer, and the like including a camera capable of photographing an object. However, the user terminal 20 is not limited to this example and may be provided as a separate camera.

The manager terminal 30 may be understood as a device capable of updating functions provided to the user terminal 20 through the image analysis server 10 or inputting a certain command. For example, the manager terminal 30 may include a smart phone, a laptop computer, a tablet PC, a wearable device, and a computer capable of communicating with the image analysis server 10 .

2 is a diagram schematically showing the configuration of the image analysis server 10 of FIG. 1, and FIG. 3 is a diagram schematically showing the sub-configuration of the second box removal module 330 of the image analysis server 10 of FIG. am.

Referring to FIGS. 2 and 3 , the image analysis server 10 may include a memory 200 , a processor 300 and a communication module 400 .

The processor 300 may be configured to process commands of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to the processor 300 from the memory 200 or the communication module 400 . In addition, commands may be provided to the processor 300 through a communication channel between respective components constituting the image analysis server 10 .

The processor 300 forms a plurality of boxes for an object, and inputs/outputs data necessary for deleting the remaining boxes leaving only the number of boxes corresponding to the object among the plurality of boxes, data processing, data management, communication network ( 40) can be used to perform various functions such as communication. Specific components of the processor 300 for executing this will be described later.

In addition, components of the processor 300 may include an artificial neural network pre-learned through deep learning. For example, at least one of the components of the processor 300 may be an artificial neural network realizing RetinaNet, which will be described in detail later.

The memory 200 is a computer-readable recording medium and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive.

The processor 300 may load program codes stored in the memory 200 and use them to count objects or determine the types of objects. These program codes may be loaded from a recording medium (for example, a DVD, memory card, etc.) readable by a separate computer, or transferred from another device through the communication module 400 and stored in the memory 200 .

In addition, the memory 200 will be provided with a database 210 capable of storing data necessary for forming a plurality of boxes for an object and deleting the remaining boxes leaving only the boxes corresponding to the object among the plurality of boxes. can

The communication module 400 may provide a function for the user terminal 20 and the image analysis server 10 or the manager terminal 30 and the image analysis server 10 to communicate with each other through the communication network 40 .

The image analysis server 10 includes a box setting module 310 as a physical component, a first box removal module 320, a second box removal module 330, a pill counting module 340, a counting module 350, and a type determination module 360. In addition, the second box removal module 330 may include a reference box setting unit 331, an aggregation box setting unit 332, a comparison space setting unit 333, and a box removal unit 334 by comparing pill counts. , and a detailed description of this will be given later.

FIG. 4 is a flowchart schematically illustrating an object counting method using an image analysis server executed by the object counting system 1 of FIG. 1 , and FIG. 5 is a more detailed view of step S2 among steps S1 to S3 of FIG. 4 . 6 is a flowchart showing step S36 in more detail among steps S32 to S36 of FIG. 5, and FIG. 7 is a diagram conceptually showing that an object is received by the user terminal 20 shown in FIG. , FIG. 8 conceptually shows that a plurality of boxes are formed for one pill by executing RetinaNet, an object recognition deep learning model, by the image analysis server 10 of FIG. 1, and FIG. It is a diagram conceptually showing a box formed in one pill (object) by executing non-maximum suppression (NMS), which is an algorithm for removing boxes by the server 10.

Referring to FIGS. 4 and 9 , the object counting method using an image analysis server includes the steps of receiving an image including one or more objects by a user terminal 20 (S1); Forming a plurality of boxes for each of the objects by the image analysis server 10 and deleting the remaining boxes leaving only the number of boxes corresponding to the object among the plurality of boxes (S2); Counting the remaining boxes by the image analysis server 10 and transmitting the number corresponding to the boxes to the user terminal 20 (S3).

Objects included in one image that can be counted by the image analysis server 10 may include objects of the same type having the same size and shape, or a plurality of types of objects having different sizes and shapes.

In this embodiment, an object included in one image counted by the image analysis server 10 is an object of the same type, for example.

In addition, hereinafter, the above-described process will be described in more detail by taking that the object is a pill as an example.

First, the step (S1) of receiving an image including one or more objects (eg, pills) by the user terminal 20 will be described in detail as follows.

The user may place the same type of pill having the same size and shape on the object plate 50 (see (a) of FIG. 7) and take an image of the pill through the user terminal 20 (see (a) of FIG. 7). see b)).

At this time, the pills should be placed on the object plate 50 so that they do not overlap.

However, the concept of the present invention is not limited thereto, and the object counting system 1 may include a function of warning that pills are overlapped by the multi-analysis aid 60 or the image analysis server 10, which will be described later. . A detailed description of this will be given later.

The object plate 50 may be a flat plate on which pills can be placed, and may be provided in a color contrasting with the pill or a color different from that of the pill. For example, if the pill is provided in white, the object plate 50 may be provided in black.

After that, the image including the pill photographed by the user terminal 20 may be transmitted to the image analysis server 10 .

In the present embodiment, a case where a user holds the user terminal 20 to capture an image is described as an example, but the spirit of the present invention is not limited thereto, and the user terminal 20 is attached to the multi-analysis assistant 60 described below. It is also possible to take an image while placing it (refer to FIG. 14), and a detailed description thereof will be described later.

Next, a plurality of boxes are formed for each of the objects (eg, pills) by the image analysis server 10, and only the boxes corresponding to the object are left among the plurality of boxes, and the remaining boxes are deleted. The detailed description of the step (S2) is as follows.

The image analysis server 10 may receive images including a plurality of pills of the same type from the user terminal 20 .

Thereafter, a plurality of boxes may be formed for one object by the box setting module 310 of the image analysis server 10 (S32).

For example, the box setting module 310 may be provided as an artificial neural network that executes RetinaNet, which is an object recognition deep learning model. When RetinaNet is running, multiple boxes can be formed for each pill. However, the object recognition deep learning model that can be executed by the box setting module 310 is not limited to RetinaNet, and the box setting module 310 may include executing one or more CenterNet among YOLO.

In the case of using RetinaNet, the number of object samples is very small compared to the number of background samples in learning the neural network in the method of detecting an object using a box, and problems caused by this can be solved.

Specifically, RetinaNet may be an integrated network composed of a backbone network and two task-specific subnetworks. The backbone network plays a role in calculating convolutional feature maps for all input images. The first subnet is a step of performing object classification in a convolutional way from the results of the backbone, and the second subnet can perform a role of convolutionally estimating a bounding box.

FIG. 8 conceptually shows that a plurality of boxes B are formed for one pill (object) by executing RetinaNet, which is an object recognition deep learning model, by the box setting module 310 .

When RetinaNet is executed by the box setting module 310, when pills are closely attached to each other, a plurality of boxes are formed for one pill, so the number of pills and the number of boxes are not the same. Therefore, in order to accurately count the number of pills even when the pills are closely attached to each other, after RetinaNet is executed, some of the boxes formed by RetinaNet must be removed.

Specifically, after RetinaNet is executed by the box setting module 310, an algorithm for removing some of a plurality of boxes formed in one object by the first box removal module 320 of the image analysis server 10 is executed. It can (S34).

For example, an algorithm executed by the first box removal module 320 may be non maximum suppression (NMS). Here, non-maximum suppression (NMS) may be understood as a non-maximum suppression algorithm that retains a maximum value and removes a non-maximum value when comparing a current pixel with neighboring pixels.

FIG. 9 conceptually shows a box formed in one pill (object) by executing non-maximum suppression (NMS), which is an algorithm for removing boxes by the first box removal module 320.

Even after non-maximum suppression (NMS) is executed, the number of pills and the number of boxes may be different if the pills are very closely attached.

For example, referring to FIG. 9 , it can be seen that five boxes (B1, B2, B3, B4, B5) are formed in three pills that are very closely attached. In this case, a step of deleting the remaining boxes while leaving only one box in one object may be executed in the second box removal module 330 (S36). Here, the second box removal module 330 may include a reference box setting unit 331, an aggregation box setting unit 332, a comparison space setting unit 333, and a box removal unit 334 by comparing pill coefficients. And, with this configuration, step S36 can be executed as follows (see FIG. 6).

Specifically, step S36 includes setting an arbitrary box among remaining boxes as a reference box by the reference box setting unit 331 (S361); Setting an aggregation box, which is a set of boxes overlapping the reference box, by the aggregation box setting unit 332 (S362); setting a space remaining in the reference box as a comparison space by removing a space overlapping the aggregation box among spaces occupied by the reference box by a comparison space setting unit 333 (S363); When the ratio of the comparison space and the space occupied by the standard box is greater than the approximate coefficient by the box removal unit 334 by the coarse coefficient comparison, the box set as the standard box is left, and the space occupied by the comparison space and the standard box If the ratio of is smaller than the round-robin coefficient, a step (S364) of removing the box set as the reference box may be included (see FIG. 5).

FIG. 10 is a conceptual diagram to aid understanding of step S36 shown in FIG. 6 .

Referring to FIGS. 1 to 10, step S36 is described as an example as follows.

When step S34 is executed by the first box removing module 320, more boxes than pills are formed for closely attached pills (for example, 5 boxes (B1 to B5) by 3 pills) can be formed).

In this case, the first box (B1), which is an arbitrary box among the five remaining boxes (B1 to B5), is set as the reference box, and the second box (B2), which is overlapping with the first box (B1), and the fourth box (B1). A box B4 and a fifth box B5 are set as aggregation boxes.

Thereafter, among the spaces occupied by the first box B1, spaces overlapping the aggregation boxes B2, B4, and B5 are removed, and the remaining space is set as the comparison space C.

After that, since the ratio of the space occupied by the comparison space (C) and the first box (B1), which is the reference box, is greater than the circular coefficient (the space occupied by the comparison space (C) / reference box (B1) > the circular coefficient), The first box B1 set as the reference box may remain.

Here, the pill coefficient represents a space in which an object (pill) can exist, and may be set differently according to the size and shape of the object (pill), and the pill coefficient is greater than 0 and less than 1 (for example, the pill coefficient is 0.85 ) can be set to the value of

Such a pill coefficient may be set by the pill coefficient determination module 340 of the image analysis server 10 .

Specifically, the pill count according to the size and shape of the object (pill) may be stored in the database 210, and an image including the object (pill) is transmitted from the user terminal 20 to the image analysis server 10. In this case, the pill coefficient determination module 340 matches the pill coefficient stored in the database 210 according to the size and shape of the object (pill), and sets the pill coefficient differently according to the type of object. Conceptually, as the size of the pill increases, the pill coefficient may increase between 0 and 1.

Similarly, when the fourth box B4 is set as the reference box, the ratio of the comparison space to the space occupied by the reference box fourth box B4 is smaller than the approximate coefficient, so the fourth box B4 set as the reference box can be removed.

In this way, in the case of going through steps S361 to S364, even if there are closely attached objects, the number of objects and the number of boxes can be the same.

Thereafter, a step S3 of counting remaining boxes by the image analysis server 10 and transmitting the number corresponding to the boxes to the user terminal 20 may be executed.

Specifically, the counting module 350 of the image analysis server 10 counts the remaining boxes and transmits them to the user terminal 20, and the user terminal 20 displays the counted number or voices the user through a speaker. can be forwarded to

In addition, the types of objects analyzed by the image analysis server 10 and the number of counted objects may be matched and stored in the database 210, and the user may view the types of objects and counted objects through the user terminal 20. You can also query the history of the number of .

Through this process, the user only needs to take images of tens to tens of thousands of pills and transmit them to the image analysis server 10, and the exact number of pills can be counted and notified to the user. can reduce the amount of time spent on inventory.

In addition, the above-described processor may be installed in the user terminal 20 in the form of an application or provided as a web page, and the user downloads the application or connects to the web page and uploads an image, and the pill included in the image The number of may be automatically transmitted to the user.

Hereinafter, the box setting module 310, the first box removal module 320, the second box removal module 330, the pill count determination module 340, and the counting module, which are sub-components of the above-described image analysis server 10, are described. (350) will be described in more detail.

As described above, the box setting module 310 may execute an object recognition deep learning model to form a plurality of boxes for each of the objects.

The first box removal module 320 may execute an algorithm for removing some of a plurality of boxes formed in each object.

The second box removal module 330 may leave only one box for one object and delete the remaining boxes.

Specifically, the second box removal module 330 may include a reference box setting unit 331, an aggregation box setting unit 332, a comparison space setting unit 333, and a box removal unit 334 by comparing pill counts. can,

The reference box setting unit 331 may set an arbitrary box among the remaining boxes as the reference box.

The set box setting unit 332 may set an set box, which is a set of boxes overlapping a reference box.

The comparison space setting unit 333 may set the space remaining in the reference box as the comparison space by removing a space that overlaps with the aggregation box from among spaces occupied by the reference box.

If the ratio of the space occupied by the comparison space and the reference box is greater than the pill coefficient, the box removal unit 334 by comparing the approximate coefficient leaves the box set as the reference box, and the ratio of the comparison space to the space occupied by the reference box is the approximate coefficient. If it is smaller than , the box set as the reference box can be removed.

The pill count determination module 340 may match the pill count stored in the database 210 according to the size and shape of the object appearing in the image.

The counting module 350 may count the number of boxes corresponding to the object and transmit the count to the user terminal 20 .

11 is a diagram schematically illustrating a flowchart of transmitting a plurality of images to the image analysis server 10 and counting one or more objects included in each of the plurality of images using the object counting system 1 of FIG. 1, 12 is a diagram showing a screen displayed on the user terminal 20 of FIG. 1 in case of single analysis mode and multi analysis mode, and FIG. 13 is a view showing a plurality of images analyzed by the image analysis server 10 in case of multi analysis mode. It is a diagram showing the number and type of objects included in each object on the screen of the user terminal 20 of FIG. 1 .

In the object counting system 1 of the above-described embodiment, one image is transmitted to the image analysis server 10 through the user terminal 20, and a plurality of objects included in one image are analyzed by the image analysis server 10. Although it has been described as an example, hereinafter, an object counting system 1 for an embodiment of transmitting a plurality of images to the image analysis server 10 and analyzing a plurality of objects included in each of the plurality of images will be described. .

Prior to the description of the method of counting objects included in a plurality of images using the image analysis server, the screen of the user terminal 20 will be described with reference to FIGS. 12 and 13 as follows.

The screen of the user terminal 20 includes an image enlargement unit 111, a single analysis button 112, a multi analysis button 113, an image input button 114, a multi analysis window 115, and a total number display unit 119. can include

An image being photographed or photographed by the user terminal 20 may be displayed on the image enlarger 111 .

A plurality of images captured by the user terminal 20 may be displayed on the multi-analysis window 115 , and the number of objects for each image analyzed by the image analysis server 10 may be displayed.

In addition, the multi-analysis window 115 includes an image selection window 115a for selecting each image and a number display unit 115b for displaying the number of images analyzed by the image analysis server 10. can be provided. In addition, the multi-analysis window 115 may be provided with a delete button 116 capable of deleting each image.

The type display unit 118 may display the type of object included in the image selected by the image selection window 115a.

The total number display unit 119 may display the sum of objects included in all of the plurality of images displayed on the multi-analysis window 115 .

11 to 13, the method of counting objects included in a plurality of images using an image analysis server is a single analysis mode in which one image can be input by the user terminal 20 or a plurality of images are input. Selecting a multi-analysis mode that can be used (S10); When the multi-analysis mode is selected, a plurality of images including one or more objects are input by the user terminal 20 and the plurality of input images are transmitted to the image analysis server 10 (S20); Counting the number of objects included in each of the plurality of images by the image analysis server 10 (S30); A step S40 of displaying the number of objects included in each of the plurality of images by the user terminal 20 may be included.

First, a step (S10) in which a single analysis mode in which one image can be input or a multi-analysis mode in which a plurality of images can be input is selected by the user terminal 20 will be described in detail.

A user may select a single analysis mode or a multi-analysis mode through the user terminal 20 .

Specifically, the user can execute the single analysis mode by touching or clicking the single analysis button 112 displayed on the screen of the user terminal 20, and execute the multi-analysis mode by touching or clicking the multi-analysis button 113. can make it

Here, when the single analysis mode is selected, it can be understood that only one image is captured through the user terminal 20, one image is transmitted to the image analysis server 10, and only one image is analyzed. .

In addition, when the multi-analysis mode is selected, it can be understood that a plurality of images are captured by the user terminal 20, the plurality of images are transmitted to the image analysis server 10, and all of the plurality of images are analyzed.

In addition, when the multi-analysis mode is selected, the user terminal 20 may be provided with an input window (not shown) for selecting the number of images to be captured. In this case, as many images as the number of images to be captured by the user can create

For example, if 5 types of pills need to be provided to patient A, the user can input 5 in the input window, and if 5 images are input, the 5 images will be transmitted to the image analysis server 10. can

Next, when the multi-analysis mode is selected, a plurality of images including one or more objects are input by the user terminal 20 and the plurality of input images are transmitted to the image analysis server 10 (S20). ) is described in detail as follows.

When the user selects the multi-analysis mode, the multi-analysis window 115 is activated on the screen of the user terminal 20, and a plurality of captured images may be displayed on the multi-analysis window 115.

A user can edit a plurality of images displayed on the multi-analysis window 115 . For example, the user may touch or click the delete button 116 of the multi-analysis window 115 to delete an image that is not to be analyzed.

When a plurality of images including one or more pills are input (captured) by the user terminal 20 , the user may input the type of pills displayed in the images through the user terminal 20 . However, the concept of the present invention is not limited thereto, and the type of pill may be automatically identified by the multi-analysis aid 60 and/or the image analysis server 10, which will be described later. A detailed description of this will be given later.

A plurality of images input in this way may be transmitted to the image analysis server 10 .

Next, a step (S30) of counting the number of objects included in each of a plurality of images by the image analysis server 10 will be described.

Specifically, in step S30, a plurality of boxes are formed for each object included in each of the plurality of images by the image analysis server 10, and the number of boxes corresponding to the object is formed among the plurality of boxes formed in each image. deleting the remaining boxes leaving only the boxes; The method may include counting the number of boxes remaining in each of the plurality of images by the image analysis server 10 and transmitting the number corresponding to the number of boxes remaining in each of the plurality of images to the user terminal 20 .

Here, the counting method of objects included in each image is the same as the above-described S2 and S3, and a detailed description thereof will be substituted for the above-described S2 and S3.

Next, a step (S40) of displaying the number of objects included in each of a plurality of images by the user terminal 20 will be described.

Specifically, in step S40, a plurality of images are displayed on the multi-analysis window 115 of the user terminal 20: an object included in each of the plurality of images in the multi-analysis window 115 of the user terminal 20 The step of displaying the number of; displaying the type of object included in each of the plurality of images on the type display unit 118 of the user terminal 20; A step of displaying the sum of objects included in all of the plurality of images on the total number display unit 119 of the user terminal 20 (see FIG. 13).

For example, four images are displayed on the multi-analysis window 115, and the number of pills is displayed on one side (eg, the bottom) of each image.

In addition, a type display unit 118 is provided on one side of the multi-analysis window 115, and the type of the selected image (eg, Nexium tablet) may be displayed on the type display unit 118. At this time, the selected image among the plurality of images displayed on the multi-analysis window 115 may be displayed in a different color from the non-selected images.

Meanwhile, the object counting system 1 of this embodiment may further include a multi-analysis auxiliary device 60 and a moving belt 70 for inputting a plurality of images in the multi-analysis mode of step S10 described above.

FIG. 14 is a diagram schematically showing the multi-analysis auxiliary device 60 on which the user terminal 20 of FIG. 1 can be seated, and FIG. 15 is the multi-analysis auxiliary device of FIG. 14 for easily performing the multi-analysis mode ( 60) and a schematic view of the moving belt 70.

Referring to FIGS. 14 and 15 , the object counting system 1 of this embodiment may further include a multi-analysis auxiliary device 60 and a moving belt 70 for easily performing a multi-analysis mode.

The multi-analysis auxiliary device 60 can be understood as a device on which the user terminal 20 can be seated, and the moving belt 70 can be understood as a device capable of moving a plurality of object plates 50 .

When the multi-analysis auxiliary device 60 and the moving belt 70 shown in FIGS. 14 and 15 are provided, a plurality of images including one or more objects are input by the user terminal 20 described above in S20. The steps can be easily realized.

Specifically, the step of inputting a plurality of images including one or more objects by the user terminal 20 is a step in which the user terminal 20 is seated on the terminal seat 67 of the multi-analysis auxiliary device 60. ; seating the plurality of object plates 50 on which objects are placed on the moving belt 70; Sequentially positioning the plurality of object plates 50 at the bottom of the user terminal 20 according to the movement of the moving belt 70; And each of the object plates 50 stays at the bottom of the user terminal 20 for a certain period of time and moves, so that the plurality of object plates 50 move to the bottom of the user terminal 20, and the user terminal 20 moves to the bottom of the user terminal 20. A step of generating a plurality of images by photographing an object placed on the object plate 50 may be included.

In addition, in the above-described embodiment, an embodiment in which only the number of objects of the same type is counted has been described, but in the case of using the object plate 50 including the type identification tag 52, the object counting system 1 uses different types of objects. It is also possible to determine the object of

Specifically, a type identification tag 52 provided in one or more of letters, barcodes, and certain symbols may be provided on one side of the object plate 50 . The type of object (pill) can be determined by the type identification tag 52 .

For example, the user can place different types of pills on the object plate 50 according to the type identification tag 52 attached to the object plate 50, and the user terminal 20 determines the type of the object plate 50. By photographing the identification mark 52, an image including the type identification mark 52 and an object is generated, or an image including an object and an image including the type identification mark 52 are generated, respectively, and the image analysis server In (10), the type and number of objects can be determined by matching and analyzing the object with the type recognition table 52. At this time, the processor 300 of the image analysis server 10 may further include a type determination module 360 capable of determining the type recognition table 52 .

In this case, after the step (S30) of counting the number of objects included in each of the plurality of images by the above-described image analysis server 10, the object type determination module 360 of the image analysis server 10 determining the type of the object by matching the type recognition table 52 with the type identification mark 52; and a step of displaying the number and type of objects included in each of the plurality of images by the user terminal 20 may be executed.

Specifically, data on the type of object according to the type identification table 52 is stored in the database 210, and the type determination module 360 receives data on the type of object stored in the database 210 and determines the type of object. type can be determined.

For example, if the type identification mark 52 is provided as a symbol 1234 and the database 210 stores the type of object corresponding to the symbol 1234 as a Nexium tablet, the user writes a Nexium tablet on the object plate 50 marked with the symbol 1234. By setting the definition, the type of object can be easily recognized by the image analysis server 10 without any cumbersome work.

Hereinafter, a physical device capable of determining the type of the above-described object will be described in more detail.

An object counting system 1 according to an embodiment of the present invention provides a space in which objects can be placed, and includes an object plate 50 including a type identification tag 52 provided in one or more of letters, barcodes, and certain symbols. ; A user terminal 20 capable of capturing an object plate 50 and generating an image including an image including one or more objects placed on the object plate 50 and an image including the type identification mark 52; and an image analysis server 10 capable of determining the number and type of objects included in the image. Here, the object and the type identification mark 52 may be photographed as one image or may be photographed as separate images.

The object plate 50 includes a flat seating portion 55 on which an object can be placed; and a type identification tag 52 disposed outside the seating portion 55 and provided with one or more of letters, barcodes, and certain symbols.

In addition, the object counting system 1 may further include a multi-analysis auxiliary device 60 including a terminal seating portion 67 spaced apart from the object plate 50 by a preset distance and on which the user terminal 20 may be placed. there is.

The multi-analysis aid 60 includes a lower end 62 to which the object plate 50 moves; an upper surface portion 66 including a terminal seating portion 67 on which the user terminal 20 can be placed; And it may include a side portion 64 connecting the lower portion 62 and the upper surface portion 66. Here, the height of the side portion 64 may be understood as a distance between the object plate 50 and the user terminal 20, and the side portion 64 may be provided to be adjustable in height.

In the case of using such a multi-analysis aid 60, an image of an object placed on the object plate 50 can be created by placing the user terminal 20 on the terminal seating part 67 and photographing the object. can be filmed

In addition, the multi-analysis auxiliary device 60 may include a sensor 69 capable of determining overlapping of objects placed on the object plate 50 .

For example, the sensor 69 is provided on the side part 64 of the multi-analysis aid 60, the object plate 50 passes through the front of the sensor 69, and the sensor 69 is the object plate 50 As it is moved, the height of the object placed on the object plate 50 may be scanned. Here, the height of the object may be understood as a length measured in a vertical direction from the seating portion 55 of the object plate 50 .

That is, the image captured by the user terminal 20 may be understood as capturing one side (top surface) of the object, and the sensor 69 attached to the multi-analysis aid 60 may capture the other side (side surface) of the object. ) can be understood as scanning.

As the object plate 50 passes the front of the sensor 69, the sensor 69 can scan all the objects placed on the object plate 50, and a certain range among the objects placed on the object plate 50. When an object exceeding is scanned, the user may be notified.

At this time, the multi-analysis auxiliary device 60 is provided with a speaker (not shown) connected to the sensor 69 to inform the user through a warning sound or to transmit a signal from the sensor 69 to the user terminal 20 so that the user A warning sound or warning display may be given to the user through the terminal 20 .

In this case, the user can check the objects placed on the object plate 50 and place them again without overlapping the objects.

In addition, the object counting system 1 further includes a moving belt 70 on which a plurality of object plates 50 are seated and which can move the plurality of object plates 50 to the bottom of the user terminal 20. can do.

Here, the moving belt 70 may be provided to form a closed curve. In this case, by placing a plurality of object plates 50 on the moving belt 70 forming a closed curve, the number of pills may be counted using the plurality of object plates 50 .

Also, when a plurality of object plates 50 are provided, the color of the seating portions 55 of the plurality of object plates 50 may be provided differently.

For example, the object plate 50 on which a red-type object is placed may be provided in a green-type color, and the color of the seating portion 55 of the object plate 50 on which a white-type object is placed is provided in a black-type color. It can be. In this case, the image analysis server 10 can recognize the object more easily by distinguishing the object and the background color.

Above, the image analysis server 10 according to an embodiment of the present invention, the object counting system 1 including the same, the object counting method using the image analysis server, and the counting method of objects included in a plurality of images using the image analysis server Although described as a specific embodiment, this is only an example, and the present invention is not limited thereto, and should be interpreted as having the widest scope according to the basic idea disclosed herein. A person skilled in the art may implement an embodiment that is not indicated by combining or substituting the disclosed embodiments, but this also does not deviate from the scope of the present invention. In addition, those skilled in the art can easily change or modify the disclosed embodiments based on this specification, and it is clear that such changes or modifications also fall within the scope of the present invention.

1: object counting system 10: image analysis server
20: user terminal 30: manager terminal
50: object plate 60: multi analysis aid
70: moving belt

Claims (13)

receiving an image including one or more objects by a user terminal;
forming a plurality of boxes for each of the objects by an image analysis server, and deleting the remaining boxes leaving only the boxes corresponding to the object among the plurality of boxes; and
Counting the number of remaining boxes by the image analysis server and transmitting the number corresponding to the boxes to the user terminal;
The step of deleting the remaining boxes leaving only the boxes corresponding to the number of the boxes among the plurality of boxes,
Executing an algorithm for removing some of a plurality of boxes formed in each object by a first box removal module; and
Leaving only one box in one object and deleting the remaining boxes by a second box removal module
Object counting method using image analysis server. According to claim 1,
Forming a plurality of boxes for each of the objects by the image analysis server, and deleting the remaining boxes leaving only the boxes corresponding to the object among the plurality of boxes,
Executing an object recognition deep learning model by a box setting module to form a plurality of boxes for each of the objects
Object counting method using image analysis server. delete delete According to claim 1,
The step of leaving only one box in one object and deleting the remaining boxes by the second box removal module,
setting an arbitrary box among remaining boxes as a reference box by a reference box setting unit;
setting an aggregation box that is a set of boxes overlapping the reference box by an aggregation box setting unit;
setting a space remaining in the reference box as a comparison space by removing a space overlapping the aggregation box among spaces occupied by the reference box by a comparison space setting unit;
When the ratio of the comparison space and the space occupied by the standard box is greater than the approximate coefficient by the box removal unit by the coarse coefficient comparison, the box set as the standard box is left, and the ratio of the space occupied by the comparison space and the standard box and removing the box set as the reference box when this is less than the circular coefficient.
Object counting method using image analysis server. According to claim 2,
The object recognition deep learning model executed by the box setting module is RetinaNet.
Object counting method using image analysis server. According to claim 1,
An algorithm for removing some of the plurality of boxes formed in each object by the first box removal module is NMS (non maximum suppression)
Object counting method using image analysis server. According to claim 5,
The pill count is stored in a database according to the size and shape of the object,
The pill count determination module matches the pill count stored in the database according to the size and shape of the object shown in the image.
Object counting method using image analysis server. As an image analysis server, the image analysis server,
Receiving an image including one or more objects from a user terminal, forming a plurality of boxes for each of the objects, leaving only the boxes corresponding to the object among the plurality of boxes and deleting the remaining boxes, Counting the number of boxes and transmitting the number corresponding to the boxes to the user terminal;
The image analysis server,
a box setting module that executes an object recognition deep learning model to form a plurality of boxes for each of the objects;
A first box removal module capable of executing an algorithm for removing some of a plurality of boxes formed in each object; and
A second box removal module for leaving only one box for one object and deleting the remaining boxes
Image analysis server. delete According to claim 9,
The second box removal module,
a reference box setting unit for setting an arbitrary box among remaining boxes as a reference box;
an aggregation box setting unit that sets an aggregation box that is a set of boxes overlapping the reference box;
a comparison space setting unit configured to set a space remaining in the reference box as a comparison space by removing a space overlapping the aggregation box from among spaces occupied by the reference box; and
If the ratio of the comparison space and the space occupied by the reference box is greater than the Orbit coefficient, the box set as the reference box is left, and if the ratio of the comparison space and the space occupied by the reference box is less than the Orbit coefficient, the reference box Including a box removal unit by pill coefficient comparison that removes the box set to
Image analysis server. According to claim 9,
The image analysis server,
a database storing pill counts according to the size and shape of the object; and
Further comprising a pill count determination module matching the pill count stored in the database according to the size and shape of the object shown in the image.
Image analysis server. a user terminal capable of receiving an image including one or more objects; and
Claims 9, 11, and 11, wherein a plurality of boxes are formed for each of the objects, only the boxes corresponding to the object are left among the plurality of boxes, and the remaining boxes are deleted, and the remaining boxes are counted. Including the image analysis server according to any one of claims 12
object counting system.

Images