RU2730112C1 - System and method of identifying objects in composite object - Google Patents

Abstract

FIELD: image processing means.SUBSTANCE: invention relates to image processing and relates to a system for identifying objects in a composite object. System comprises graphical user interface, memory, image capturing device and data processing device. Data processing device includes a video data acquisition module, an image analysis module, a segmentation module, an identification module and an output module. Identification of objects in a composite object comprises steps of obtaining video data from an image capturing device in real time, performing analysis of video data in order to detect a composite object in the frame, performing segmentation of the obtained image using an artificial neural network, identifying an object using another artificial neural network on each of the obtained individual images of the object, outputting the obtained identification result to the screen.EFFECT: technical result is higher accuracy and speed of identifying objects.39 cl, 2 dwg

Description

The present invention relates to the field of using artificial neural networks in computer vision, and more specifically to systems and methods for processing video data received from video cameras for automatic identification of various objects.

LEVEL OF TECHNOLOGY

In the context of this application, video systems mean software and hardware that uses computer vision methods for automated data collection based on the analysis of streaming video (video analysis). Such video systems rely on image processing algorithms, including algorithms for recognition, segmentation, classification and identification of images, which allow video analysis without direct human intervention. In addition, modern video systems allow you to automatically analyze video data from cameras and compare this data with the data available in the database.

Almost any warehouses and retail outlets are currently equipped with such video systems (mainly to ensure security, prevent theft and control goods and employees). A warehouse is an important structural unit that influences the efficiency of all business processes of an enterprise or retail outlet. The increase in the volume of goods and work made it impossible for its normal activities on the basis of simple warehouse accounting, in connection with which enterprises are looking for ways to automate warehouse processes. For example, some modern warehouses are fully automated and do not require the presence of a person (the goods move through the automated conveyor system to the desired location and, accordingly, are also automatically shipped). Automated systems can also be used to search and track goods, as well as to control and account for goods in a warehouse or in any other room (see, for example, patent RU 2698157 C1).

At the moment, such general technologies of automated warehouse management are widespread, such as: 1) labeling of goods or goods, which allows you to identify the goods and track the route of its movement; 2) address storage, which guarantees order in the warehouse, since all goods are assigned a specific location address; 3) two-dimensional bar-coding, necessary to optimize warehouse logistics and allowing you to instantly find out all information about the cargo using special software; 4) 3D scanning, which is more typical for industrial warehouses; and 5) RFID - a method based on which information about a cargo or product is written off from special RFID tags.

For example, from the prior art, the solution disclosed in US patent 8310363 B2, G08B 1/08, publ. 11/13/2012, which discloses various options for automated monitoring of inanimate objects that can move into or out of the room. Monitoring is carried out by means of a system of sensors designed to be placed on each at least one inanimate object that can move relative to objects and is located in the said interior part of the room, bounded by a frame, and the specified system of sensors is configured to receive data at least about one object for their subsequent processing. An event monitoring sensor associated with said sensor system monitors whether an event occurs associated with the movement of at least one object relative to said frame, wherein said sensor system receives data about at least one object only when said monitoring sensor event detects the occurrence of an event. The positioning system is designed to monitor the location of the protected premises. Said sensor system and said positioning system are connected by a communication system that transmits data regarding at least one object, said sensor system, and the location of an object provided by said positioning system to a remote device whenever said sensor system receives data at least one object as a result of detecting the occurrence of a motion event by the auxiliary event monitoring sensor, after which the remote device can take appropriate action.

The prior art also knows the solution disclosed in the application of China CN 102147888 A, G06K 17/00, publ. 08/10/2011, which describes an intelligent warehouse management system that includes a data processing device, RFID storage and RFID tray. RFID storage and RFID tray are respectively equipped with RFID electronic tags. The RFID tray is wirelessly connected to the data processing equipment. The system is different in that it also includes a manual warehouse manager terminal, a terminal on a vehicle, a terminal on a forklift, and a gate terminal installed at the warehouse door. This solution also describes a method applied to said intelligent warehouse management system, which implements intelligent warehouse management using RFID electronic tags.

To implement the solutions described above, each object in a warehouse or any other secured area must be equipped with either an appropriate sensor or an RFID tag that is read by various RFID readers to track objects. These types of solutions are complex and expensive to implement. In addition, there is a high probability of error in their implementation.

The prior art also knows the solution disclosed in the application US 2013/0088591 A1, G06K 9/78, publ. 04/11/2013, which describes an object identification and tracking system configured to associate an object identifier with an object's position in a warehouse, and the system contains several pairs of video cameras and a data processing facility associated with pairs of video cameras. The data processing facility is configured to: determine the object identifier for the object in question; determining the first position of the specified object based on images from the specified pairs of video cameras; and associating the object identifier with the specified first position. The known solution, although it uses video data and image processing methods, differs significantly from the proposed solution.

The main difference / advantage of the proposed solution from the prior art solutions is the use of existing standard video surveillance and image processing tools to identify all objects / goods in the warehouse. Our solution is mainly aimed at simplifying, speeding up and improving the accuracy of the identification process. In addition, when an order or delivery is formed, then most often the goods are placed on special pallets or trays. Our proposed solution provides for the recognition and identification of all separate objects as part of one composite object. That is, no separate identification of each object is required (as happens in technologies with labels and barcodes). This approach also speeds up the identification process.

In addition, artificial neural networks are increasingly used in modern video systems for image recognition and identification. An artificial neural network (ANN) is a mathematical model, as well as its hardware and / or software implementation, built on the principle of the organization and functioning of biological neural networks (networks of nerve cells in living organisms). One of the main advantages of ANNs is the possibility of their training, during which the ANN is able to independently identify complex dependencies between input and output data. It is the use of one or even several ANNs for image processing, as well as the use of standard video surveillance and video data processing tools that makes the proposed solution easier to implement in absolutely any room and more accurate in terms of identifying objects (including composite objects) in comparison with known from the prior art solutions.

DISCLOSURE OF THE INVENTION

The claimed technical solution is aimed at eliminating the disadvantages inherent in the prior art and at developing already known solutions.

The technical result of the claimed group of inventions is to improve the accuracy and speed of object identification through the use of at least one artificial neural network.

This technical result is achieved by the fact that the system for identifying objects in a composite object contains: a graphical user interface (GUI) containing data input / output devices configured to ensure user interaction with the system; a memory configured to store video data and a database that includes at least a sample of object reference images; at least one image capture device configured to obtain video data from the monitoring area; and at least one data processing device, comprising: a video data acquisition unit configured to receive video data from at least one image capturing device in real time; an image analysis module configured to analyze video data to detect at least one composite object in a frame, after which the resulting image is transmitted to the segmentation module; a segmentation module configured to segment the obtained image of the composite object into separate images of the objects that are part of the composite object, said segmentation being performed using an artificial neural network (ANN); an identification module, configured to identify objects using at least one other artificial neural network on each of the obtained individual images of objects; an output module configured to output the obtained identification result.

The specified technical result is also achieved by means of a method for identifying objects in a composite object performed by a computer system containing a graphical user interface, at least one data processing device and a memory storing video data and a database that includes at least a sample of reference images objects, the method comprising the steps of: receiving video data from at least one image capturing device in real time, and said image capturing device receives video data from the monitoring area; analyzing the video data in order to detect at least one composite object in the frame and obtain an image of the said composite object; performing segmentation of the obtained image of the composite object into separate images of the objects that are part of the composite object, and said segmentation is carried out using an artificial neural network (ANN); identify the object using at least one other artificial neural network on each of the obtained separate images of the object; display the obtained identification result on the screen.

In one particular version of the claimed solution, the composite objects include at least: pallets, pallets, trays, and the objects that make up the composite object include at least: cargo, goods, box.

In another particular embodiment of the claimed solution, the control zones include at least one of the following: storage room, car body.

In yet another particular embodiment of the claimed solution, segmentation is performed by color and / or shape and / or texture.

In another particular version of the claimed solution, identification is carried out by comparing each recognized image of an object with at least one reference image of objects contained in the database.

In another particular version of the claimed solution, all objects in the database are divided into object classes.

And in another particular version of the claimed solution, a separate ANN is provided for each class of objects, which is used for identification.

In another particular embodiment of the claimed solution, at least one data processing device further comprises a classification module configured to classify individual object images obtained after segmentation into object classes, and a separate artificial neural network is provided for said classification.

In another particular embodiment of the claimed solution, the system is additionally configured to automatically replenish the sample of reference images of each object for training at least one artificial neural network, and the replenishment of the sample of reference images of objects and training at least one artificial neural network are constant processes, since the set of objects and their appearance changes over time.

In another particular embodiment of the claimed solution, the sample of reference images of each object contains the N last loaded images for this object, where N is a positive integer preset by the user.

In yet another particular embodiment of the claimed solution, at least one data processing device further comprises an accounting and control module configured to count in a given control area both compound objects and identified objects that are part of a compound object to account for objects in a user-specified system time in each user-defined control zone.

In another particular version of the claimed solution, the accounting and control module is additionally configured to count objects that have left the control area and arrived in the control area.

In another particular version of the claimed solution, the accounting and control module is additionally configured to compare the number of identified objects that have departed from one control zone with the number of identified objects that have arrived in at least one other control zone, and the compared control zones are set by the system user and when a discrepancy is detected of the mentioned number, the output module automatically performs the actions predefined by the user of the system.

In another particular embodiment of the claimed solution, the actions predefined by the user of the system include at least one or a combination of: triggering an alarm, SMS notification of the system user, notification of the user by e-mail, audio notification of the user.

In another particular version of the claimed solution, the accounting and control module, in the event of a discrepancy between the mentioned number of identified objects in different control zones, additionally determines at least one time interval during which a violation potentially occurred, after which the output module automatically exports video data of this interval time and sending it to a predefined user of the system for analysis.

In another particular version of the claimed solution, the output module is additionally configured for automatic recording of processed video data into the archive and / or export of video data, and recording and export can be carried out for all video data in a time interval specified by the user of the system or only for those video data in which the facts of departure and the facts of the arrival of objects in each control area, to ensure the possibility of analysis based on archived data.

In another particular version of the claimed solution, the accounting and control module is additionally configured with the ability to generate a report based on the results of identification, counting and comparison of the number of identified objects, and the report can be generated for each control area separately or for a bundle of control areas, with the mentioned link control zones are either preset by the user, or set by the system user in real time.

In another particular embodiment of the claimed solution, the output module is additionally configured to output the received at least one report on the screen or to send the received at least one report to a predefined user of the system.

And in yet another particular embodiment of the claimed solution, video data analysis in order to detect at least one composite object in a frame is performed continuously or in a time range defined by the system user or by command from the system user.

In addition to the above, this technical result is also achieved due to a computer-readable storage medium containing instructions executed by a computer processor for implementing methods for identifying objects in a composite object.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram of a system for identifying objects in a composite object.

FIG. 2 is a block diagram of a method for identifying objects in a composite object.

CARRYING OUT THE INVENTION

Below will be a description of exemplary embodiments of the claimed group of inventions. However, the claimed group of inventions is not limited only to these embodiments. It will be obvious to those skilled in the art that other implementations may fall within the scope of the claimed group of inventions described in the claims.

The claimed technical solution in its various embodiments can be implemented in the form of computer systems and methods implemented by various computer means, as well as in the form of a computer-readable storage medium storing instructions executed by a computer processor.

FIG. 1 is a block diagram of a system for identifying objects in a composite object. This system in its complete set includes: memory (10), configured to store the database (DB) and video data; graphical user interface (20) containing data input / output means; at least one image capturing device (30, ..., 3n); and at least one data processing device (40, ..., 4m), comprising: a video data acquisition module (50), an image analysis module (60), a segmentation module (70), a classification module (75), an identification module (80) , accounting and control module (85) and output module (90). It should be noted that modules (75) and (85) may not be included in the basic configuration of the system.

In this context, computer systems are understood as any computing systems built on the basis of software and hardware interconnected technical means.

An image capturing device in the context of this application means a video camera.

The data processing device can be a processor, microprocessor, graphics processor, computer (electronic computer), PLC (programmable logic controller) or integrated circuit configured to execute certain commands (instructions, programs) for data processing. The processor can be multi-core for parallel data processing.

A graphical user interface (GUI) is a system of means for user interaction with a computing device based on the presentation of all system objects and functions available to the user in the form of graphical screen components (windows, icons, menus, buttons, lists, etc.). In this case, the user has random access, through input / output devices, to all visible display objects - interface blocks that are displayed on the display. Moreover, the data input / output device can be, but is not limited to, for example, a manipulator mouse, keyboard, touchpad, stylus, joystick, trackpad, etc.

The memory device configured to store data can be, but is not limited to, hard disk drives (HDD), flash memory, server, ROM (read only memory), solid state drives (SSD), optical storage devices, etc. ...

In the context of this application, the memory stores video data from image capture devices as well as a database (DB) that includes at least a sample of reference object images.

It should be noted that the described system may include any other devices known in the art, for example, such as various types of sensors, input / output devices, display devices, readers of labels or bar codes, etc.

Next, an example of the operation of the above system for identifying objects in a composite object will be described in detail.

All the stages of the system operation described below are also applicable to the implementation of the proposed method for identifying objects in a composite object, which will be discussed in more detail below.

Let's consider how this system works. Suppose this system, as well as the corresponding software, are installed in a warehouse of a large store. For delivery and transportation of goods to the warehouse from the supplier or to the address of buyers, trucks (for large-sized goods) or cars (for small loads and goods) are used.

The storage room, as well as the body of each car, are equipped with image capture devices (30,…, 3n). An image capturing device, in this case a video camera, is located in such a way as to continuously receive video data from a certain monitoring area in real time. Control areas in the context of this application include at least one of the following: a storage room, a truck body, a car body, or any other room used to store or transport various goods (objects).

It should be noted that the described identification system may include several video cameras in each separate monitoring area to obtain more video data and improve the accuracy of the results of their processing. Image capturing devices continuously receive video data in real time, which are recorded in the archive of the video surveillance system (to provide the possibility of further analysis based on archived data), and also transmitted to the data processing device.

Further, the main work is performed by at least one data processing device, for example, such as a computer graphics processor. Said data processing device (40, ..., 4m) includes separate software or hardware modules / blocks, each of which is configured to perform a specific task. In the described solution, as shown in FIG. 1, the data processing device contains modules such as: a video data acquisition module (50), an image analysis module (60), a segmentation module (70), a classification module (75), an identification module (80), an accounting and control module (85) and output module (90). Next, the operation of each module will be described in detail.

The video data acquisition module (50) continuously receives all video data from at least one image capturing device in real time. Then all the obtained video data is analyzed by the image analysis module (60) to identify / detect frames displaying / characterizing at least one composite object, after which the resulting image is transmitted to the segmentation module (70).

In the context of this application, composite objects include at least: pallets, pallets, trays (in other words, everything where several objects can be placed), and objects that make up a composite object include at least but not limited to: cargo , commodity, box, product and so on. In this case, as indicated in a particular embodiment of the system, the analysis of video data is performed continuously or in a time range specified by the system user or by a signal / command from the system user. That is, for a warehouse with a time of shipment and receipt of goods, for example, from 8:00 to 12:00, it is relevant to record and analyze video data only during this period of time to save memory and computing resources of the system.

It should be noted that all image capture devices are located in the monitoring areas in such a way as to completely cover the entire room / monitoring area. That is, in order to obtain a complete picture from the control area, the cameras' view areas may slightly overlap / overlap. Thus, the image analysis module can easily detect all composite or individual objects. After a frame with a composite object is detected, the obtained at least one image / frame of said composite object is automatically transferred to the segmentation module (70).

The segmentation module (70), in turn, is configured to segment the obtained image of the composite object into separate images of the objects included in the composite object. The mentioned segmentation is performed using its artificial neural network. It should be noted that the segmentation is performed by color and / or shape and / or texture. The system user can specify any kind of segmentation, or segmentation can be performed sequentially by each of the listed methods.

After dividing the image of a composite object into separate images of its constituent objects, these separate images are then transferred to the identification module (80). The mentioned module performs object identification using at least one other artificial neural network on each of the obtained individual images of objects. The identification is carried out by comparing each recognized object image with at least one object image contained in the database. If, during the identification process, the recognized image of the object sufficiently coincides with at least one image from the database, the system immediately stops the identification process. This approach avoids wasting the available computing resources of the system and speeds up the comparison process.

And the principle of identification is as follows: an artificial neural network receives a separate image of an object, after which it outputs a certain vector of numbers - an image descriptor. The database stores a sample of reference images of all objects, including a descriptor corresponding to each image. The ANN uses these descriptors to compare images. Moreover, the ANN is trained so that the smaller the angle between these vectors of numbers in space, the greater the likelihood of image coincidence. The cosine of the angle between the vectors of numbers (vectors from the database and the resulting vector of the object image) is used as a metric for comparison. Accordingly, the closer the cosine of the angle between vectors to one, the more likely it is that the object is the same in the compared pair of images. When configuring the system, the user can specify a range of values, at which the system will make a decision about the coincidence of objects. In this case, the artificial neural network sequentially compares each obtained separate image of an object with all images of objects available in the database until it gets a sufficient match.

It should be noted that in order to improve the accuracy of the system and improve and accelerate the identification of objects, before direct identification in some implementations of the proposed solution, the classification of individual images of objects can be carried out. For this, at least one data processing device provides an additional classification module (75), which uses a separate / its own ANN for classification. The above-mentioned classification module is configured with the ability to classify the individual object images obtained after segmentation into object classes. Examples of such classes may include, but are not limited to: electronics, household appliances, pet products, baby products, household products, clothing, automotive products, garden products, construction, sports, food, health, and the like. In addition, objects in each class can be further divided into more specific subclasses (for example, food can be divided into such subclasses as: tea, coffee, alcohol, confectionery, bakery products, groceries, canned goods, drinks, dairy products, and so on. , up to the division into such subclasses as: box, bag, bottle, perishable product, etc.). At the same time, all object files in the database are also divided into similar classes (available in a specific warehouse where the identification system in question is used). The data about each object in the database includes at least: name, main characteristics (such as size, shape, color palette) and a sample of reference images of the object. Moreover, each image in the sample of reference images of objects includes a descriptor characterizing the vector of numbers of this image. It should be noted that for each class (or even subclass) of objects, the identification module has its own separate ANN used for object identification (to improve the accuracy and speed of identification).

With regard to the sample of reference images of objects, the system under consideration for identification is configured to automatically replenish said sample of reference images of each object for training at least one used artificial neural network. In this case, the replenishment of the sample of reference images of objects and the training of at least one artificial neural network are constant processes, since the set of objects and their appearance change over time. It should be noted that the processes of replenishing the sample of images, training the artificial neural network and the immediate stages of image processing can be performed in parallel by at least one data processing device. In this case, each new image of an object is added to the sample of reference images of the corresponding object only after the identification process of this image is completed (to avoid errors).

Training of each artificial neural network in the context of the proposed solution is performed on the basis of the updated database. The user / operator of the system can set a certain time at which the training of the artificial neural network will be performed. For example, once a day. In this case, the mentioned training can be performed, for example, by a data processing device or by a cloud service or any other computing device.

It should be noted again that the sample of images of each object contains many images of this object (different angles and views). The system user can, when setting up its operation, set a specific number of images that should be contained in the sample (to maintain the relevance of the data in the database). Thus, the sample of reference images for each object contains the N most recently loaded images for that object, where N is a positive integer pre-set by the user. Suppose the user specified N = 20. After identifying a specific object (for example, a printer of a certain brand and model has been identified), the data processing device analyzes a sample of images of this particular printer. If the number of images in the image sample for the recognized object is 20, then the data processing device deletes the oldest (oldest) image and stores the new, just received image of the object into the sample. If the sample of images contains, for example, only 5 images of the object, then the new resulting image of the object is simply added to the selection (without removing the older images). Thus, it is possible to maintain the relevance of information about the objects available in the warehouse. This is necessary because the set of objects / goods and their appearance change over time.

After identifying all objects in different implementations of the system, different actions are performed in accordance with the requirements at different enterprises or retail outlets. In the basic implementation, the results of object identification are transmitted to the output module (90), which is configured to display the obtained identification results on the screen. The results obtained can be used either by the user of the system, or can be loaded into any other data processing system for analysis and further processing of the results obtained.

However, most often they want to use this kind of identification systems for complete accounting and control of goods in the warehouse. For this purpose, a separate accounting and control module is provided in the data processing device of the claimed system (85). This module is configured to count both compound objects and identified individual objects that are part of a compound object in a user-defined control zone. It should be noted that any interaction of the user with the system is carried out through the use of data input / output means contained in the graphical user interface (GUI) of the claimed identification system. The mentioned counting of objects is necessary to account for objects at a time specified by the user of the system in each user-specified control zone. At the same time, the accounting and control module is additionally configured to count objects that have left the control zone and arrived in the control zone. Thus, you can easily control how many objects (goods) were taken from the warehouse, how many arrived in the truck for transportation, how many were brought to the warehouse, and so on.

In addition, the accounting and control module (85) is further configured to match the number of identified objects that have departed from one control area with the number of identified identified objects that have arrived in at least one other control area. In this case, the compared control zones are set by the user of the system (by means of the GUI tools), and when a discrepancy of the mentioned number is detected, the output module (90) automatically performs actions predefined by the user of the system. For example, consider a situation when a set of goods was taken from a warehouse for delivery to a customer. In this case, the first control zone is the warehouse, the second control zone is the truck, which is supposed to deliver the goods to the buyer. If the identification system determines that 9 items of goods have left the warehouse at a certain specified time, and 9 items of goods have also been loaded into the truck, then the system does not take any action. However, if 9 goods have left the warehouse, and only 7 goods have arrived in the truck, then the system performs actions predefined by the user of the system. Moreover, in the context of this application, the actions predefined by the user of the system include at least one or a combination of: triggering an alarm, sms notification of the system user, notification of the user by e-mail, audio notification of the user, and the like. Thus, it is possible to detect the error in a timely manner and eliminate it even on the territory of the warehouse, without interfering with the process of the end buyer of a set of goods. It should be noted that this was considered only the simplest case.

However, more often than not, a very large number of goods leave the warehouse every day, which are transported to different customers in different cars. In this case, the number of objects leaving the warehouse (from zone 1) can be compared with the total number of objects that arrived at the trucks preset by the user of the system (zone 2 + zone 3 + zone 4 +…).

In addition, in some particular embodiments, the accounting and control module (85), in the event of a discrepancy between the mentioned number of identified objects in different control zones, additionally performs the determination of at least one time interval during which a violation potentially occurred. After determining the said time interval, the output module automatically exports video data of this time interval and sends it to a predetermined user of the system for analysis. That is, the user of the system can view the video data only for the time interval defined by the system and it is easy to understand where the error or violation occurred (without spending a lot of time).

As mentioned earlier, the output module (90) in its basic implementation is configured to output the identification results, however, the said module is also additionally configured to automatically record the processed video data into the archive and / or to export the said video data. Moreover, recording and export can be carried out for all video data in a time interval specified by the user of the system or only for those video data in which the facts of departure and the facts of arrival of objects in each control zone are recorded. This is necessary to provide the ability to quickly and accurately analyze video data from archived data.

In addition, as for the accounting and control module (85), it is additionally configured with the ability to generate a report based on the obtained identification results, counting results and the results of comparing the number of identified objects. It should be noted that the above report can be generated for each control zone separately or for a bundle of control zones, which is either preset by the user or set by the system user in real time (using the GUI tools). For example, for a bunch of such control zones as several trucks that pick up goods from the warehouse during the day.

In this case, the output module (90) is additionally configured to output the received at least one report to the screen. Moreover, these reports are generated either by a signal from the user, or at a time preset by the user (for example, once a day, at 21:00, after the end of shipments from the warehouse). In addition, reports can be automatically sent to predefined users of the system (for example, by sms or e-mail) or saved to the system memory. But when generating at least one report on a signal / command from a system user, this report can be immediately displayed by the output module (90).

Next, an example of a specific implementation of a method for identifying objects in a composite object will be described. FIG. 2 shows a block diagram of one of the options for implementing the method for identifying objects in a composite object.

The specified method is performed by a computer system already described above, comprising a graphical user interface, at least one data processing device and a memory storing video data and a database that includes at least a sample of reference images of objects.

The inventive method in the basic version contains the stages at which:

(100) receiving video data from at least one image capturing device in real time, and the said image capturing device receives video data from the monitoring area;

(200) analyzing the video data in order to detect at least one composite object in the frame and obtain an image of said composite object;

(300) performing segmentation of the obtained image of the composite object into separate images of the objects that are part of the composite object, and said segmentation is carried out using an artificial neural network (ANN);

(400) identify the object using at least one other artificial neural network on each of the obtained separate images of the object;

(500) display the obtained identification result on the screen.

It should be noted once again that this method can be implemented using the previously described computing system and, therefore, can be extended and refined by all the same particular embodiments that have already been described above for implementing a system for identifying objects in a composite object.

In addition, the embodiments of the present group of inventions can be implemented using software, hardware, software logic, or a combination thereof. In this embodiment, program logic, software, or a set of instructions is stored on one or more of various conventional computer-readable storage media.

As used herein, a "computer-readable medium" can be any medium or medium that can contain, store, transmit, distribute, or transport instructions (commands) for use (execution) by a computing system, such as, for example, a computer. In this case, the storage medium can be a non-volatile computer-readable storage medium.

If necessary, at least some of the various operations discussed in the description of this solution can be performed in a different order from the presented one and / or simultaneously with each other.

Although this technical solution has been described in detail for the purpose of illustrating the presently most necessary and preferred embodiments, it should be understood that the invention is not limited to the disclosed embodiments and, moreover, is intended to modify and various other combinations of features from the described embodiments. For example, it should be understood that the present invention contemplates that to the extent possible, one or more features of any embodiment may be combined with another one or more features of any other embodiment.

Claims (56)

1. A system for identifying objects in a compound object, containing: a graphical user interface (GUI) containing data input / output devices configured to provide user interaction with the system; a memory configured to store video data and a database that includes at least a sample of object reference images; at least one image capture device configured to obtain video data from the monitoring area; and at least one data processing device comprising: a video acquisition module configured to acquire video data from at least one image capturing device in real time; an image analysis module configured to analyze video data to detect at least one composite object in a frame, after which the resulting image is transmitted to the segmentation module; a segmentation module configured to segment the obtained image of the composite object into separate images of the objects that are part of the composite object, said segmentation being performed using an artificial neural network (ANN); an identification module, configured to identify objects using at least one other artificial neural network on each of the obtained individual images of objects; an output module configured to output the obtained identification result. 2. The system according to claim. 1, in which the composite objects include at least: pallets, pallets, trays, and the objects included in the composite object include at least: cargo, goods, box. 3. The system of claim. 1, in which the control zones include at least one of the following: storage room, car body. 4. The system according to any one of paragraphs. 1-3, in which the segmentation is performed by color and / or shape and / or texture. 5. The system of claim. 4, in which the identification is carried out by comparing each recognized image of an object with at least one reference image of objects contained in the database. 6. System according to any one of paragraphs. 1-3, in which all objects in the database are divided into object classes. 7. The system according to claim 6, in which for each class of objects there is a separate ANN used for identification. 8. The system of claim. 7, in which at least one data processing device further comprises a classification module configured to classify the separate object images obtained after segmentation into object classes, and a separate artificial neural network is provided for said classification. 9. System according to any one of paragraphs. 1, 8, further configured to automatically replenish the sample of reference images of each object for training at least one artificial neural network, moreover, the replenishment of the sample of reference images of objects and the training of at least one artificial neural network are constant processes, since the set of objects and their appearance change over time. 10. The system of claim 9, wherein the sample of reference images of each object contains the N most recently loaded images for that object, where N is a positive integer predefined by the user. 11. System according to any one of paragraphs. 1-3, 8, in which at least one data processing device further comprises an accounting and control module configured to count in a given control area both compound objects and identified objects that are part of a compound object to account for objects in a user-defined system time in each user-defined control zone. 12. The system according to claim 11, in which the accounting and control module is additionally configured to count objects that have left the control area and have arrived in the control area. 13. The system according to claim 12, in which the accounting and control module is additionally configured to compare the number of identified objects that have departed from one control zone with the number of identified objects that have arrived in at least one other control zone, and the compared control zones are set by the system user and upon detection inconsistencies of the mentioned number, the output module automatically performs the actions predefined by the user of the system. 14. The system according to claim 13, in which the actions predefined by the user of the system include at least one or a combination of: triggering an alarm, sms notification of the system user, notification of the user by e-mail, audio notification of the user. 15. The system according to claim 13, in which the accounting and control module, in the event of a discrepancy between the said number of identified objects in different control zones, additionally determines at least one time interval during which a violation potentially occurred, after which the output module automatically performs export video data of this time interval and sending it to a predefined user of the system for analysis. 16. System according to any one of paragraphs. 1, 13, in which the output module is additionally configured for automatic recording of processed video data to the archive and / or export of video data, and recording and export can be carried out for all video data in a time interval specified by the system user or only for those video data in which the facts of departure and the facts of the arrival of objects in each control area, to ensure the possibility of analysis based on archived data. 17. System according to any one of paragraphs. 11-15, in which the accounting and control module is additionally configured with the ability to generate a report based on the results of identification, counting and comparison of the number of identified objects, and the report can be generated for each control area separately or for a bunch of control zones, while the aforementioned bunch of zones control is either preset by the user, or set by the user of the system in real time. 18. The system of claim 17, wherein the output module is further configured to display the received at least one report on a screen or to send the received at least one report to a predetermined system user. 19. System according to any one of paragraphs. 1-6, in which the analysis of video data for the purpose of detecting at least one composite object in a frame is performed continuously or in a time range defined by the system user or on a command from the system user. 20. A method for identifying objects in a composite object, performed by a computer system containing a graphical user interface, at least one data processing device and a memory storing video data and a database that includes at least a sample of reference images of objects, the method comprising the steps , where: receive video data from at least one image capturing device in real time, and said image capturing device receives video data from the monitoring area; analyzing the video data in order to detect at least one composite object in the frame and obtain an image of the said composite object; performing segmentation of the obtained image of the composite object into separate images of the objects that are part of the composite object, and said segmentation is carried out using an artificial neural network (ANN); identify the object using at least one other artificial neural network on each of the obtained separate images of the object; display the obtained identification result on the screen. 21. The method according to claim. 20, in which the composite objects include at least: pallets, pallets, trays, and the objects included in the composite object include at least: cargo, goods, box. 22. The method according to claim 20, wherein the control zones include at least one of the following: a storage room, a car body. 23. The method according to any of paragraphs. 20-22, in which segmentation is performed by color and / or shape and / or texture. 24. The method according to claim 23, wherein identification is performed by comparing each recognized object image with at least one reference object image contained in the database. 25. The method according to any one of paragraphs. 20-22, in which all objects in the database are divided into object classes. 26. The method according to claim 25, in which a separate ANN is provided for each class of objects, which is used for identification. 27. The method according to claim. 26, additionally made with the possibility of classifying the separate images of objects obtained after segmentation into classes of objects, and a separate artificial neural network is provided for said classification. 28. The method according to any one of paragraphs. 20, 27, in which additionally the replenishment of the sample of reference images of each object is automatically performed for training at least one artificial neural network, moreover, the replenishment of the sample of reference images of objects and the training of at least one artificial neural network are constant processes, since the set of objects and their appearance change over time. 29. The method of claim 28, wherein the sample of the reference images of each object contains the N most recently loaded images for that object, where N is a positive integer predefined by the user. 30. The method according to any one of paragraphs. 20-22, 27, which, after the identification stage, additionally contains an accounting and control stage, at which both composite objects and identified objects that are part of a composite object are counted in a given control zone to account for objects at a time specified by the system user in each user-defined control zone. 31. The method according to claim 30, in which at the stage of accounting and control, the counting of objects that have left the control area and have arrived in the control area is additionally performed. 32. The method according to claim 31, further comprising the comparison of the number of identified objects that have departed from one control zone with the number of identified objects that have arrived in at least one other control zone, and the compared control zones are set by the user of the system and when a discrepancy of said number is detected by the system automatically user-defined actions are performed. 33. The method according to claim 32, wherein the actions predefined by the system user include at least one or a combination of: triggering an alarm, sms notification of the system user, notification of the user by e-mail, audio notification of the user. 34. The method according to claim 32, in which, if a discrepancy of said number of identified objects in different monitoring zones is detected, at least one time interval is additionally performed during which a violation has potentially occurred, after which the video data of this time interval and its sending to a predefined system user for analysis. 35. The method according to any of paragraphs. 20, 32, which additionally performs automatic recording of processed video data into the archive and / or export of video data, and recording and export can be carried out for all video data in a time interval specified by the user of the system or only for those video data in which the facts of departure and facts of arrival of objects are recorded to each control area, to ensure the possibility of analysis based on archived data. 36. The method according to any of paragraphs. 31-35, in which a report is additionally generated based on the results of identification, counting and comparison of the number of identified objects, and the report can be generated for each control area separately or for a bundle of control zones, while the aforementioned bundle of control zones is either preset by the user, or set by the user of the system in real time. 37. The method according to claim 36, further comprising displaying the received at least one report on the screen or sending the received at least one report to a predetermined system user. 38. The method according to any of paragraphs. 20-25, in which the analysis of video data in order to detect at least one composite object in the frame is performed continuously or in a time range defined by the system user or on a signal from the system user. 39. A computer-readable storage medium containing instructions executed by a computer processor for implementing methods for identifying objects in a composite object according to any one of claims. 20-38.

Images